Installation and Setup #

To get started with LoRA training, you need to set up a proper environment with the necessary tools and dependencies. This guide will walk you through the process.

What is LoRA? #

LoRA (Low-Rank Adaptation) is a technique designed to facilitate the fine-tuning of large-scale language and diffusion models efficiently. Instead of overhauling the entire set of model parameters —which can number in the billions— LoRA introduces small, trainable “low-rank” matrices that adapt the model’s behavior. This innovative approach was detailed in the paper “LoRA: Low-Rank Adaptation of Large Language Models” by researchers at Microsoft.

Setting up sd-scripts #

First, download kohya_ss’ sd-scripts, which contains the tools you’ll need for LoRA training.

Windows Installation #

Follow the instructions in the GitHub repository for Windows-specific setup.

Miniconda Installation (Windows/Linux) #

For Miniconda users on Windows or Linux, follow these steps:

# Installing sd-scripts
git clone https://github.com/kohya-ss/sd-scripts
cd sd-scripts

# Creating the conda environment and installing requirements
conda create -n sdscripts python=3.10.14
conda activate sdscripts
conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
python -m pip install --use-pep517 --upgrade -r requirements.txt
python -m pip install --use-pep517 lycoris_lora
accelerate config

When running accelerate config, you’ll be asked several questions. The typical configuration looks like:

• This machine
• No distributed training
• No
• No
• No
• All
• fp16

Additional Packages #

You might also want to install additional packages like xformers or bitsandbytes for improved performance:

# Installing xformers
python -m pip install --use-pep517 xformers

# Installing bitsandbytes for Windows
python -m pip install --use-pep517 bitsandbytes --index-url=https://jllllll.github.io/bitsandbytes-windows-webui

Preparing a Model #

For training, you’ll need a base model. You can use models in various formats:

Using Checkpoint Files #

You can use .safetensors or .ckpt files directly for training.

Downloading Specific Models #

Pony Diffusion SDXL #

Pony Diffusion V6 XL is a popular SDXL model fine-tuned for creating anthropomorphic and cartoon-style art.

1. Download from CivitAI:

   • Visit Pony Diffusion V6 XL on CivitAI
   • Download the “Pruned Model fp16” version (approximately 6.46 GB)
   • You should also download the recommended VAE (approximately 319 MB)

2. Important Usage Notes:

   • Always use clip skip 2 for this model
   • The model uses quality boosters in prompts: score_9, score_8_up, score_7_up, score_6_up, score_5_up, score_4_up
   • Special tags: source_pony, source_furry, source_cartoon, source_anime
   • Rating tags: rating_safe, rating_questionable, rating_explicit

NoobAI’s V-Prediction Models #

NoobAI V-Prediction models are SDXL models trained with v-prediction, which can produce higher quality results for specific styles.

1. Download Options:

   • Laxhar/noobai-XL-Vpred-1.0 on Hugging Face

2. Using with Diffusers: For models hosted on Hugging Face, you can download them using the Hugging Face CLI:

   # Install huggingface_hub if not already installed
   pip install huggingface_hub
   
   # Download NoobAI model
   huggingface-cli download Laxhar/noobai-XL-Vpred-1.0 --local-dir ./models/noobai-xl
   

Using Diffusers Format #

Many models are available in the Hugging Face Diffusers format, which is more modular and easier to work with for certain applications.

1. Download a Diffusers Model:

   # Example: Download Stable Diffusion XL base model
   huggingface-cli download stabilityai/stable-diffusion-xl-base-1.0 --local-dir ./models/sdxl-base
   

Using Models with sd-scripts #

Using Checkpoint Files in Training Configs #

When training with sd-scripts, you specify the model path in your training configuration:

# Example training command using a .safetensors checkpoint
accelerate launch --num_cpu_threads_per_process=2 "./train_network.py" \
  --pretrained_model_name_or_path="./models/ponyDiffusionV6XL.safetensors" \
  --v2 --v_parameterization \  # Use for v-prediction models
  --network_module="networks.lora" \
  --network_dim=32 \
  --network_alpha=16 \
  --train_batch_size=1 \
  --learning_rate=1e-4 \
  --max_train_epochs=5 \
  --output_dir="./output/my_lora" \
  --save_model_as="safetensors" \
  --dataset_config="./configs/dataset_config.toml" \
  --mixed_precision="fp16" \
  --clip_skip=2  # Important for Pony Diffusion

Training with Diffusers Format Models #

sd-scripts supports training with models in Diffusers format:

# Example training command using a Diffusers model
accelerate launch --num_cpu_threads_per_process=2 "./train_network.py" \
  --pretrained_model_name_or_path="./models/sdxl-base" \  # Path to local Diffusers folder
  --network_module="networks.lora" \
  --network_dim=32 \
  --network_alpha=16 \
  --train_batch_size=1 \
  --learning_rate=1e-4 \
  --max_train_epochs=5 \
  --output_dir="./output/my_lora" \
  --save_model_as="safetensors" \
  --dataset_config="./configs/dataset_config.toml" \
  --mixed_precision="fp16"

Alternatively, you can load Hugging Face models directly:

# Example loading directly from Hugging Face
accelerate launch --num_cpu_threads_per_process=2 "./train_network.py" \
  --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" \
  --network_module="networks.lora" \
  --network_dim=32 \
  --network_alpha=16 \
  --train_batch_size=1 \
  --learning_rate=1e-4 \
  --max_train_epochs=5 \
  --output_dir="./output/my_lora" \
  --save_model_as="safetensors" \
  --dataset_config="./configs/dataset_config.toml" \
  --mixed_precision="fp16" \
  --use_safetensors

Special Considerations for V-Prediction Models #

When training with v-prediction models like NoobAI:

1. Add the --v_parameterization flag to your training command
2. You may need lower learning rates (e.g., 5e-5 to 1e-4)

# Example for NoobAI v-prediction model
accelerate launch --num_cpu_threads_per_process=2 "./train_network.py" \
  --pretrained_model_name_or_path="Laxhar/noobai-XL-Vpred-1.0" \
  --v_parameterization \
  --zero_terminal_snr \
  --network_module="networks.lora" \
  --network_dim=32 \
  --network_alpha=16 \
  --train_batch_size=1 \
  --learning_rate=8e-5 \
  --max_train_epochs=5 \
  --output_dir="./output/my_noobai_lora" \
  --save_model_as="safetensors" \
  --dataset_config="./configs/dataset_config.toml" \
  --mixed_precision="fp16"

Next Steps #

Once your environment is set up and you have a base model ready, you can proceed to the Training Parameters section to learn how to configure and start your LoRA training.