Dataset Tools #

A “small” collection of Python and PowerShell scripts that dataset curators might find handy. The ⚡ in the title means it is a PowerShell script and 🐍 means Python and a 🦀 means Rust of course!

🦀 extract-metadata

Processes safetensors files, extracts their metadata, converts it into a JSON object, and writes the JSON to a new file. It can process individual files or all safetensors files in a directory.

🦀 format-json

Formats JSON files from single-line to multi-line format using serde_json.

🦀 remove-extra-file-extensions

This Rust script renames text files in a specified directory by removing any extra image extensions (.jpeg, .png, or .jpg) from their names.

🐍 Check for Duplicate Words Between Captions and Tags

This script traverses through a directory, searches for text files, processes each file to extract tags and captions, and highlights occurrences of tags within captions using random colors, displaying the results in a visually rich format in the terminal.

🐍 Check for Large Images

This script checks the resolution of all images in a specified directory and its subdirectories. If the resolution of an image exceeds a certain limit, the path of the image is written to an output file. The script uses multiprocessing to speed up the process.

🐍 Check for Transparency

This script recursively traverses a specified directory, identifying image files with extension .png. For each identified image, it checks if it contains transparency by examining its mode with PIL.

🐍 Convert RGBA to RGB in PNGs

This script automates the process of converting .png images from RGBA to RGB format in a specified directory, utilizing multiprocessing to enhance efficiency.

🐍 Count Images in Folder

This script counts the total number of JPEG and PNG images in a specified directory.

🐍 Create Empty Captions for Images

This Python script creates an empty text file with the same name as each image file (.jpg, .png, or .jpeg) present in a specified directory. The script checks if the directory exists, and then iterates through all the image files in the directory.

🐍 e621 JSON to Caption

This Python script is designed to process JSON files found within a specified directory and its subdirectories. Each JSON file is expected to contain data related to image posts sourced from e621.net or e6ai.net. The script parses these JSON files, extracts relevant information such as image URL, ratings, and tags, and generates caption files (.txt) based on this data.

🐍 FurryTagger

Loads eva02-vit-large-448-8046, applies it to a set of images in a specified directory, and write the model’s output tags to a text file for each image.

🐍 Newlines to Commas

Recursively modify the content of .txt files in the specified directory and its subdirectories by replacing newlines with commas and spaces.

🐍 Replace Transparency with Black

This Python script processes all .png images in a specified directory by adding a black layer to each, utilizing multiprocessing to handle the images in parallel for efficiency.

🐍 Search for Tag

This script is used to search for the word “anthrofied” in all .txt files within a specified directory and its subdirectories. It uses multiprocessing to speed up the search by checking multiple files simultaneously.

⚡ Format-JSONFiles

Formats JSON files from single-line to multi-line format using the jq command-line JSON processor.

⚡ Format-JSONFilesToSingleLine

Formats JSON files to a single-line format using the jq utility.

⚡ Get-Seed

Retrieves the ss_seed value from the metadata of a .safetensors file.

⚡ Inspect-Lora

Takes a file path as input and uses Python to read the metadata from a .safetensors file. It then pretty-prints the metadata contents to the console and saves it next to the LoRA.