Training Parameters
On this page
An anthro male protogen with white fur and blue eyes stands with his arms spread wide appearing to be in the midst of a transformation. He has a fluffy tail horns and a neck tuft and his body is covered in fur. He wears a screen face with a glowing blue smile and blue features. The protogen has a screen visor and is surrounded by a simple dark background with glowing blue accents. His eyes glow brightly and the protogen's robotic parts have blue glowing symbols. The overall scene is a digital painting with a mystical ambiance.

Training Parameters

This guide explains the key parameters for LoRA training and how to configure them for optimal results.

Sample Prompt File

A sample prompt file is used during training to generate sample images that help monitor your progress. Here’s an example for Pony:

# anthro female kindred
score_9, score_8_up, score_7_up, score_6_up, rating_explicit, source_furry, solo, female anthro kindred, mask, presenting, white pillow, bedroom, looking at viewer, detailed background, amazing_background, scenery porn, realistic, photo --n low quality, worst quality, blurred background, blurry, simple background --w 1024 --h 1024 --d 1 --l 6.0 --s 40
# anthro female wolf
score_9, score_8_up, score_7_up, score_6_up, rating_explicit, source_furry, solo, anthro female wolf, sexy pose, standing, gray fur, brown fur, canine pussy, black nose, blue eyes, pink areola, pink nipples, detailed background, amazing_background, realistic, photo --n low quality, worst quality, blurred background, blurry, simple background --w 1024 --h 1024 --d 1 --l 6.0 --s 40

Note: Sample prompts should not exceed 77 tokens. You can use tools like Count Tokens in Sample Prompts to check your prompts.

Launch Configuration

Launching with accelerate

For single GPU:

accelerate launch --num_cpu_threads_per_process=2 "./sdxl_train_network.py"

For multiple GPUs:

accelerate launch --num_processes=2 --multi_gpu --num_machines=1 --gpu_ids=0,1 --num_cpu_threads_per_process=2 "./sdxl_train_network.py"

Basic Parameters

Model Path 

--pretrained_model_name_or_path="/path/to/model/"

Specifies the directory containing the base model. Use a trailing slash for local diffusers models.

Output Directory 

--output_dir="/output_dir"

Where trained models will be saved.

Training Data 

--train_data_dir="/training_dir"

Directory containing your training dataset.

Resolution 

--resolution="1024,1024"

Set to match the model’s resolution (e.g., 1024x1024 for SDXL, 512x512 for SD 1.5).

Network Configuration

Network Dimension and Alpha 

--network_dim=8 --network_alpha=4
  • network_dim: Controls how many features your LoRA will train (rank)
  • network_alpha: Controls how many of the trained ranks can alter the base model

Network Module and Algorithm 

--network_module="lycoris.kohya"
--network_args "preset=full" "algo=locon" "conv_dim=256" "conv_alpha=4"
  • network_module: Specifies which network module to use
  • network_args: Configures specific parameters for the selected module

Presets

  • full: Train all layers in UNet and CLIP (default)
  • full-lin: Skip convolutional layers
  • attn-mlp: Train all transformer blocks
  • attn-only: Train only attention layers
  • unet-transformer-only: Train only UNet transformers
  • unet-convblock-only: Train only ResBlock, UpSample, DownSample

Algorithms

Common LyCORIS algorithms:

  • lora: Standard LoRA
  • loha: LoRA with Hadamard product
  • locon: LoRA for Convolution
  • lokr: LoRA with Kronecker product

Learning Rate and Optimization 

--lr_scheduler="cosine"
--learning_rate=0.0001 --unet_lr=0.0001 --text_encoder_lr=0.0001
--optimizer_type="AdamW"
  • lr_scheduler: Controls how learning rate changes (options: linear, cosine, cosine_with_restarts, polynomial, constant, constant_with_warmup, adafactor)
  • learning_rate, unet_lr, text_encoder_lr: Control training speed and stability
  • optimizer_type: Optimization algorithm (AdamW recommended)

Training Stability 

--scale_weight_norms=1.0
--max_grad_norm=1.0
  • scale_weight_norms: Constrains weights to prevent overtraining
  • max_grad_norm: Prevents exploding gradients (gradient clipping)

Precision Settings 

--mixed_precision="bf16"
--save_precision="fp16"
--no_half_vae
  • mixed_precision: Numerical precision during training (options: no, fp16, bf16)
  • save_precision: Precision of saved model weights
  • no_half_vae: Disables mixed precision for VAE to avoid NaN issues

Saving and Checkpoints 

--save_every_n_epochs=50
--output_name="mymodel"
  • save_every_n_epochs or save_every_n_steps: When to save checkpoints
  • save_last_n_epochs or save_last_n_steps: Only keep the most recent N saved files
  • output_name: Base name for saved files

Memory Optimization 

--cache_latents --cache_latents_to_disk
  • cache_latents: Pre-computes and caches latent representations to speed up training
  • cache_latents_to_disk: Stores cached latents on disk instead of memory

Training Duration 

--max_train_steps=400

or

--max_train_epochs=10

These parameters determine how long to train (steps or epochs).

Data Processing 

--shuffle_caption
--caption_extension=".txt"
--dataset_repeats=0
  • shuffle_caption: Randomizes caption order at each epoch
  • caption_extension: File extension for text captions
  • dataset_repeats: How many times to repeat dataset (0 for no repetition)

Attention Optimization 

--sdpa

or

--xformers

or

--mem_eff_attn

These flags enable different attention mechanisms to optimize memory usage.

Image Quality Enhancement 

--multires_noise_iterations=10 --multires_noise_discount=0.1
  • Adds noise at multiple resolutions during training
  • Helps generate more diverse images and works well with small datasets

Sample Generation 

--sample_prompts="/training_dir/sample-prompts.txt"
--sample_sampler="euler_a"
--sample_every_n_steps=100
  • Generates sample images during training to monitor progress
  • Available samplers: ddim, pndm, lms, euler, euler_a, heun, dpm_2, dpm_2_a, dpmsolver, dpmsolver++, dpmsingle, k_lms, k_euler, k_euler_a, k_dpm_2, k_dpm_2_a

Complete Example Command

Here’s a complete example command for LoRA training:

accelerate launch --num_cpu_threads_per_process=2 "./sdxl_train_network.py" \
    --lowram \
    --pretrained_model_name_or_path="/ponydiffusers/" \
    --train_data_dir="/training_dir" \
    --resolution="1024,1024" \
    --output_dir="/output_dir" \
    --enable_bucket \
    --min_bucket_reso=256 \
    --max_bucket_reso=1024 \
    --network_alpha=4 \
    --save_model_as="safetensors" \
    --network_module="lycoris.kohya" \
    --network_args \
               "preset=full" \
               "conv_dim=256" \
               "conv_alpha=4" \
               "use_tucker=False" \
               "use_scalar=False" \
               "rank_dropout_scale=False" \
               "algo=locon" \
               "train_norm=False" \
               "block_dims=8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8" \
               "block_alphas=0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625" \
    --network_dropout=0 \
    --lr_scheduler="cosine" \
    --learning_rate=0.0001 \
    --unet_lr=0.0001 \
    --text_encoder_lr=0.0001 \
    --network_dim=8 \
    --output_name="yifftoolkit" \
    --scale_weight_norms=1 \
    --no_half_vae \
    --save_every_n_epochs=50 \
    --mixed_precision="fp16" \
    --save_precision="fp16" \
    --caption_extension=".txt" \
    --cache_latents \
    --cache_latents_to_disk \
    --optimizer_type="AdamW" \
    --max_grad_norm=1 \
    --keep_tokens=1 \
    --max_data_loader_n_workers=8 \
    --bucket_reso_steps=32 \
    --multires_noise_iterations=10 \
    --multires_noise_discount=0.1 \
    --log_prefix=xl-locon \
    --gradient_accumulation_steps=6 \
    --gradient_checkpointing \
    --train_batch_size=8 \
    --dataset_repeats=0 \
    --max_train_steps=400 \
    --shuffle_caption \
    --sdpa \
    --sample_prompts=/training_dir/sample-prompts.txt \
    --sample_sampler="euler_a" \
    --sample_every_n_steps=100

Next Steps

After configuring your training parameters and completing your LoRA training, you may want to optimize your model size. Continue to the Model Shrinking guide to learn techniques for reducing model size without losing quality.

Related Posts

LoRA Training Guide

The LoRA Training Guide explains Low-Rank Adaptation (LoRA), a technique for fine-tuning large language and diffusion models efficiently by introducing small, trainable low-rank matrices instead of modifying all model parameters. This approach keeps the original model weights frozen and injects two additional matrices into each layer to learn necessary adjustments.

10 Minute LoRA Training for the Ultimate Degenerates

An offensive, unethical and biased guide on how to poop out awesome LoRAs with the bare minimal of effort and compute time.

Advanced Training Concepts

Understand the differences between steps and epochs, gradient accumulation, and other advanced concepts for optimizing LoRA training.