LoRA Training Guide #

Subsections #

How do the Captions Get Processed?
select_bucket

Installation Tips #

Firstly, download kohya_ss’ sd-scripts, you need to set up your environment either like this tells you for Windows, or if you are using Linux or Miniconda on Windows, you are probably smart enough to figure out the installation for it. I recommend always installing the latest PyTorch in the virtual environment you are going to use, which at the time of writing is 2.2.2. I hope future me has faster PyTorch!

Ok, just in case you aren’t smart enough how to install the sd-scripts under Miniconda for Windows I actually “guided” someone recently, just so I can tell you about it:

# Installing sd-scripts
git clone https://github.com/kohya-ss/sd-scripts
cd sd-scripts

# Creating the conda environment and installing requirements
conda create -n sdscripts python=3.10.14
conda activate sdscripts
conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
python -m pip install --use-pep517 --upgrade -r requirements.txt
python -m pip install --use-pep517 lycoris_lora
accelerate config

accelerate config will ask you a bunch of questions, you need to actually read each one and reply with the truth. In most cases the truth looks like this: This machine, No distributed training, no, no, no, all, fp16.

You might also want to install xformers or bitsandbytes.

# Installing xformers
# Use the same command just replace 'xformers' with any other package you may need.
python -m pip install --use-pep517 xformers

# Installing bitsandbytes for windows
python -m pip install --use-pep517 bitsandbytes --index-url=https://jllllll.github.io/bitsandbytes-windows-webui

Pony Training #

I’m not going to lie, it is a bit complicated to explain everything. But here is my best attempt going through some “basic” stuff and almost all lines in order.

Download Pony in Diffusers Format #

I’m using the diffusers version for training I converted, you can download it using git.

git clone https://huggingface.co/k4d3/ponydiffusers

Sample Prompt File #

A sample prompt file is used during training to sample images. A sample prompt for example might look like this for Pony:

# anthro female kindred
score_9, score_8_up, score_7_up, score_6_up, rating_explicit, source_furry, solo, female anthro kindred, mask, presenting, white pillow, bedroom, looking at viewer, detailed background, amazing_background, scenery porn, realistic, photo --n low quality, worst quality, blurred background, blurry, simple background --w 1024 --h 1024 --d 1 --l 6.0 --s 40
# anthro female wolf
score_9, score_8_up, score_7_up, score_6_up, rating_explicit, source_furry, solo, anthro female wolf, sexy pose, standing, gray fur, brown fur, canine pussy, black nose, blue eyes, pink areola, pink nipples, detailed background, amazing_background, realistic, photo --n low quality, worst quality, blurred background, blurry, simple background --w 1024 --h 1024 --d 1 --l 6.0 --s 40

Please note that sample prompts should not exceed 77 tokens, you can use Count Tokens in Sample Prompts from /dataset_tools to analyze your prompts.

If you are training with multiple GPUs, ensure that the total number of prompts is divisible by the number of GPUs without any remainder or a card will idle.

Training Commands #

`accelerate launch` #

For two GPUs:

accelerate launch --num_processes=2 --multi_gpu --num_machines=1 --gpu_ids=0,1 --num_cpu_threads_per_process=2  "./sdxl_train_network.py"

Single GPU:

accelerate launch --num_cpu_threads_per_process=2 "./sdxl_train_network.py"

And now lets break down a bunch of arguments we can pass to sd-scripts.

`--lowram` #

If you are running running out of system memory like I do with 2 GPUs and a really fat model that gets loaded into it per GPU, this option will help you save a bit of it and might get you out of OOM hell.

`--pretrained_model_name_or_path` #

The directory containing the checkpoint you just downloaded. I recommend closing the path if you are using a local diffusers model with a /. You can also specify a .safetensors or .ckpt if that is what you have!

    --pretrained_model_name_or_path="/ponydiffusers/"

`--output_dir` #

This is where all the saved epochs or steps will be saved, including the last one. If y

    --output_dir="/output_dir"

`--train_data_dir` #

The directory containing the dataset. We prepared this earlier together.

    --train_data_dir="/training_dir"

`--resolution` #

Always set this to match the model’s resolution, which in Pony’s case it is 1024x1024. If you can’t fit into the VRAM, you can decrease it to 512,512 as a last resort.

    --resolution="1024,1024"

`--enable_bucket` #

Creates different buckets by pre-categorizing images with different aspect ratios into different buckets. This technique helps to avoid issues like unnatural crops that are common when models are trained to produce square images. This allows the creation of batches where every item has the same size, but the image size of batches may differ.

`--bucket_no_upscale` #

Affects the resolution of images processed by the network by disabling any upscaling of images. When this option is set, the network will only downscale images to fit within the maximum area specified by self.max_area if the image’s $width \times height$ exceeds this value.

The select_bucket function, particularly the code snippet you’ve highlighted, is responsible for determining the appropriate bucket for an image based on its resolution and aspect ratio. Here’s a step-by-step explanation of what the code does:
Check if downscaling is needed: If the product of image_width and image_height is greater than self.max_area, the image is too large and must be downscaled while maintaining its aspect ratio. Calculate the resized dimensions: It calculates the width and height that the image should be resized to, such that the resized image’s area does not exceed self.max_area and the aspect ratio is preserved. Round the dimensions: The round_to_steps function is used to round the resized dimensions to the nearest multiple of self.reso_steps, which is a parameter that defines the step size for resolution buckets. Determine the aspect ratio error: The code compares the aspect ratio of the width and height after rounding to decide which dimension to prioritize in order to minimize the error in aspect ratio after resizing. Select the resized size: Based on the smaller aspect ratio error, it chooses the resized dimensions that best maintain the original aspect ratio of the image. In summary, the select_bucket function is ensuring that when downscaling is necessary, the image is resized to dimensions that are multiples of the resolution step size (self.reso_steps) and as close as possible to the original aspect ratio, without exceeding the maximum allowed area (self.max_area). Upscaling is not performed when –bucket_no_upscale is set.

`--min_bucket_reso` and `--max_bucket_reso` #

Specifies the minimum and maximum resolutions used by the buckets. These values are ignored if --bucket_no_upscale is set.

    --min_bucket_reso=256 --max_bucket_reso=1024

`--network_alpha` #

Specifies how many of the trained Network Ranks are allowed to alter the base model.

    --network_alpha=4

`--save_model_as` #

You can use this to specify either ckpt or safetensors for the file format.

    --save_model_as="safetensors"

`--network_module` #

Specifies which network module you are going to train.

    --network_module="lycoris.kohya"

`--network_args` #

The arguments passed down to the network.

    --network_args \
               "use_reentrant=False" \
               "preset=full" \
               "conv_dim=256" \
               "conv_alpha=4" \
               "use_tucker=False" \
               "use_scalar=False" \
               "rank_dropout_scale=False" \
               "algo=locon" \
               "train_norm=False" \
               "block_dims=8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8" \
               "block_alphas=0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625" \

Let’s break it down!

`preset` #

The Preset/config system added to LyCORIS for more fine-grained control.

full
- default preset, train all the layers in the UNet and CLIP.
full-lin
- full but skip convolutional layers.
attn-mlp
- “kohya preset”, train all the transformer block.
attn-only
- only attention layer will be trained, lot of papers only do training on attn layer.
unet-transformer-only
- as same as kohya_ss/sd_scripts with disabled TE, or, attn-mlp preset with train_unet_only enabled.
unet-convblock-only
- only ResBlock, UpSample, DownSample will be trained.

`conv_dim` and `conv_alpha` #

The convolution dimensions are related to the rank of the convolution in the model, adjusting this value can have a significant impact and lowering it affected the aesthetic differences between different LoRA samples. and an alpha value of 128 was used for training a specific character’s face while Kohaku recommended to set this to 1 for both LoCon and LoHa.

conv_block_dims = [conv_dim] * num_total_blocks
conv_block_alphas = [conv_alpha] * num_total_blocks

`module_dropout` and `dropout` and `rank_dropout` #

rank_dropout is a form of dropout, which is a regularization technique used in neural networks to prevent overfitting and improve generalization. However, unlike traditional dropout which randomly sets a proportion of inputs to zero, rank_dropout operates on the rank of the input tensor lx. First a binary mask is created with the same rank as lx with each element set to True with probability 1 - rank_dropout and False otherwise. Then the mask is applied to lx to randomly set some of its elements to zero. After applying the dropout, a scaling factor is applied to lx to compensate for the dropped out elements. This is done to ensure that the expected sum of lx remains the same before and after dropout. The scaling factor is 1.0 / (1.0 - self.rank_dropout).

It’s called “rank” dropout because it operates on the rank of the input tensor, rather than its individual elements. This can be particularly useful in tasks where the rank of the input is important.

If rank_dropout is set to 0, it means that no dropout is applied to the rank of the input tensor lx. All elements of the mask would be set to True and when the mask gets applied to lx all of it’s elements would be retained and when the scaling factor is applied after dropout it’s value would just equal self.scale because 1.0 / (1.0 - 0) is 1. Basically, setting this to 0 effectively disables the dropout mechanism but it will still do some meaningless calculations, and you can’t set it to None, so if you really want to disable dropouts simply don’t specify them! 😇

def forward(self, x):
    org_forwarded = self.org_forward(x)

    # module dropout
    if self.module_dropout is not None and self.training:
        if torch.rand(1) < self.module_dropout:
            return org_forwarded

    lx = self.lora_down(x)

    # normal dropout
    if self.dropout is not None and self.training:
        lx = torch.nn.functional.dropout(lx, p=self.dropout)

    # rank dropout
    if self.rank_dropout is not None and self.training:
        mask = torch.rand((lx.size(0), self.lora_dim), device=lx.device) > self.rank_dropout
        if len(lx.size()) == 3:
            mask = mask.unsqueeze(1)
        elif len(lx.size()) == 4:
            mask = mask.unsqueeze(-1).unsqueeze(-1)
        lx = lx * mask

        scale = self.scale * (1.0 / (1.0 - self.rank_dropout))
    else:
        scale = self.scale

    lx = self.lora_up(lx)

    return org_forwarded + lx * self.multiplier * scale

The network you are training needs to support it though! See PR#545 for more details.

`use_tucker` #

Can be used for all but (IA)^3 and native fine-tuning.

Tucker decomposition is a method in mathematics that decomposes a tensor into a set of matrices and one small core tensor reducing the computational complexity and memory requirements of the model. It is used in various LyCORIS modules on various blocks. In LoCon for example, if use_tucker is True and the kernel size k_size is not (1, 1), then the convolution operation is decomposed into three separate operations.

A 1x1 convolution that reduces the number of channels from in_dim to lora_dim.
A convolution with the original kernel size k_size, stride stride, and padding padding, but with a reduced number of channels lora_dim.
A 1x1 convolution that increases the number of channels back from lora_dim to out_dim.

If use_tucker is False or not set, or if the kernel size k_size is (1, 1), then a standard convolution operation is performed with the original kernel size, stride, and padding, and the number of channels is reduced from in_dim to lora_dim.

`use_scalar` #

An additional learned parameter that scales the contribution of the low-rank weights before they are added to the original weights. This scalar can control the extent to which the low-rank adaptation modifies the original weights. By training this scalar, the model can learn the optimal balance between preserving the original pre-trained weights and allowing for low-rank adaptation.

# Check if the 'use_scalar' flag is set to True
if use_scalar:
    # If True, initialize a learnable parameter 'scalar' with a starting value of 0.0.
    # This parameter will be optimized during the training process.
    self.scalar = nn.Parameter(torch.tensor(0.0))
else:
    # If the 'use_scalar' flag is False, set 'scalar' to a fixed value of 1.0.
    # This means the low-rank weights will be added to the original weights without scaling.
    self.scalar = torch.tensor(1.0)

The use_scalar flag allows the model to determine how much influence the low-rank weights should have on the final weights. If use_scalar is True, the model can learn the optimal value for self.scalar during training, which multiplies the low-rank weights before they are added to the original weights. This provides a way to balance between the original pre-trained weights and the new low-rank adaptations, potentially leading to better performance and more efficient training. The initial value of 0.0 for self.scalar suggests that the model starts with no contribution from the low-rank weights and learns the appropriate scale during training.

`rank_dropout_scale` #

A boolean flag that determines whether to scale the dropout mask to have an average value of 1 or not. This is particularly useful when you want to maintain the original scale of the tensor values after applying dropout, which can be important for the stability of the training process.

def forward(self, orig_weight, org_bias, new_weight, new_bias, *args, **kwargs):
    # Retrieve the device that the 'oft_blocks' tensor is on. This ensures that any new tensors created are on the same device.
    device = self.oft_blocks.device

    # Check if rank dropout is enabled and the model is in training mode.
    if self.rank_dropout and self.training:
        # Create a random tensor the same shape as 'oft_blocks', with values drawn from a uniform distribution.
        # Then create a dropout mask by checking if each value is less than 'self.rank_dropout' probability.
        drop = (torch.rand(self.oft_blocks, device=device) < self.rank_dropout).to(
            self.oft_blocks.dtype
        )

        # If 'rank_dropout_scale' is True, scale the dropout mask to have an average value of 1.
        # This helps maintain the scale of the tensor's values after dropout is applied.
        if self.rank_dropout_scale:
            drop /= drop.mean()
    else:
        # If rank dropout is not enabled or the model is not in training mode, set 'drop' to 1 (no dropout).
        drop = 1

`algo` #

The LyCORIS algorithm used, you can find a list of the implemented algorithms and an explanation of them, with a demo you can also dig into the research paper.

`train_norm` #

Controls whether to train normalization layers used by all algorithms except (IA)^3 or not.

`block_dims` #

Specify the rank of each block, it takes exactly 25 numbers, that is why this line is so long.

`block_alphas` #

Specifies the alpha of each block, this too also takes 25 numbers if you don’t specify it network_alpha will be used instead for the value.

That concludes the network_args.

`--network_dropout` #

This float controls the drop of neurons out of training every step, 0 or None is default behavior (no dropout), 1 would drop all neurons. Using weight_decompose=True will ignore network_dropout and only rank and module dropout will be applied.

    --network_dropout=0 \

`--lr_scheduler` #

A learning rate scheduler in PyTorch is a tool that adjusts the learning rate during the training process. It’s used to modulate the learning rate in response to how the model is performing, which can lead to increased performance and reduced training time.

Possible values: linear, cosine, cosine_with_restarts, polynomial, constant (default), constant_with_warmup, adafactor

Note, adafactor scheduler can only be used with the adafactor optimizer!

    --lr_scheduler="cosine" \

`--lr_scheduler_num_cycles` #

Number of restarts for cosine scheduler with restarts. It isn’t used by any other scheduler.

    --lr_scheduler_num_cycles=1 \

`--learning_rate` and `--unet_lr` and `--text_encoder_lr` #

The learning rate determines how much the weights of the network are updated in response to the estimated error each time the weights are updated. If the learning rate is too large, the weights may overshoot the optimal solution. If it’s too small, the weights may get stuck in a suboptimal solution.

For AdamW the optimal LR seems to be 0.0001 or 1e-4 if you want to impress your friends.

    --learning_rate=0.0001 --unet_lr=0.0001 --text_encoder_lr=0.0001

`--network_dim` #

The Network Rank (Dimension) is responsible for how many features your LoRA will be training. It is in a close relation with Network Alpha and the Unet + TE learning rates and of course the quality of your dataset. Personal experimentation with these values is strongly recommended.

    --network_dim=8

`--output_name` #

Specify the output name excluding the file extension.

WARNING: If for some reason this is ever left empty your last epoch won’t be saved!

    --output_name="last"

`--scale_weight_norms` #

Max-norm regularization is a technique that constrains the norm of the incoming weight vector at each hidden unit to be upper bounded by a fixed constant. It prevents the weights from growing too large and helps improve the performance of stochastic gradient descent training of deep neural nets.

Dropout affects the network architecture without changing the weights, while Max-Norm Regularization directly modifies the weights of the network. Both techniques are used to prevent overfitting and improve the generalization of the model. You can learn more about both in this research paper.

    --scale_weight_norms=1.0

`--max_grad_norm` #

Also known as Gradient Clipping, if you notice that gradients are exploding during training (loss becomes NaN or very large), consider adjusting the --max_grad_norm parameter, it operates on the gradients during the backpropagation process, while --scale_weight_norms operates on the weights of the neural network. This allows them to complement each other and provide a more robust approach to stabilizing the learning process and improving model performance.

    --max_grad_norm=1.0

`--no_half_vae` #

Disables mixed precision for the SDXL VAE and sets it to float32. Very useful if you don’t like NaNs.

`--save_every_n_epochs` and `--save_last_n_epochs` or `--save_every_n_steps` and `--save_last_n_steps` #

--save_every_n_steps and --save_every_n_epochs: A LoRA file will be created at each n-th step or epoch specified here.
--save_last_n_steps and --save_last_n_epochs: Discards every saved file except for the last n you specify here.

Learning will always end with what you specify in --max_train_epochs or --max_train_steps.

    --save_every_n_epochs=50

`--mixed_precision` #

⚠️

    --mixed_precision="fp16"

`--save_precision` #

⚠️

    --save_precision="fp16"

`--caption_extension` #

The file extension for caption files. Default is .caption. These caption files contain text descriptions that are associated with the training images. When you run the training script, it will look for files with this specified extension in the training data folder. The script uses the content of these files as captions to provide context for the images during the training process.

For example, if your images are named image1.jpg, image2.jpg, and so on, and you use the default .caption extension, the script will expect the caption files to be named image1.caption, image2.caption, etc. If you want to use a different extension, like .txt, you would set the caption_extension parameter to .txt, and the script would then look for image1.txt, image2.txt, and so on.

    --caption_extension=".txt"

`--cache_latents` and `--cache_latents_to_disk` #

⚠️

    --cache_latents --cache_latents_to_disk

`--optimizer_type` #

The default optimizer is AdamW and there are a bunch of them added every month or so, therefore I’m not listing them all, you can find the list if you really want, but AdamW is the best as of this writing so we use that!

    --optimizer_type="AdamW"

`--dataset_repeats` #

Repeats the dataset when training with captions, by default it is set to 1 so we’ll set this to 0 with:

    --dataset_repeats=0

`--max_train_steps` #

Specify the number of steps or epochs to train. If both --max_train_steps and --max_train_epochs are specified, the number of epochs takes precedence.

    --max_train_steps=400

`--shuffle_caption` #

Shuffles the captions set by --caption_separator, it is a comma , by default which will work perfectly for our case since our captions look like this:

rating_questionable, 5 fingers, anthro, bent over, big breasts, blue eyes, blue hair, breasts, butt, claws, curved horn, female, finger claws, fingers, fur, hair, huge breasts, looking at viewer, looking back, looking back at viewer, nipples, nude, pink body, pink hair, pink nipples, rear view, solo, tail, tail tuft, tuft, by lunarii, by x-leon-x, mythology, krystal (darkmaster781), dragon, scalie, wickerbeast, The image showcases a pink-scaled wickerbeast a furred dragon creature with blue eyes., She has large breasts and a thick tail., Her blue and pink horns are curved and pointy and she has a slight smiling expression on her face., Her scales are shiny and she has a blue and pink pattern on her body., Her hair is a mix of pink and blue., She is looking back at the viewer with a curious expression., She has a slight blush.,

As you can tell, I have separated the caption part not just the tags with a , to make sure everything gets shuffled. I’m at this point pretty certain this is beneficial especially when your caption file contains more than 77 tokens.

NOTE: --cache_text_encoder_outputs and --cache_text_encoder_outputs_to_disk can’t be used together with --shuffle_caption. Both of these aim to reduce VRAM usage, you will need to decide between these yourself!

`--sdpa` or `--xformers` or `--mem_eff_attn` #

Each of these options modifies the attention mechanism used in the model, which can have a significant impact on the model’s performance and memory usage. The choice between --xformers or --mem_eff_attn and --spda will depend on your GPU. You can benchmark it by repeating a training with them!

--xformers: This flag enables the use of XFormers in the model. XFormers is a library developed by Facebook Research that provides a collection of transformer models optimized for different hardware and use-cases. These models are designed to be highly efficient, flexible, and customizable. They offer various types of attention mechanisms and other features that can be beneficial in scenarios where you have limited GPU memory or need to handle large-scale data.
--mem_eff_attn: This flag enables the use of memory-efficient attention mechanisms in the model. The memory-efficient attention is designed to reduce the memory footprint during the training of transformer models, which can be particularly beneficial when working with large models or datasets.
--sdpa: This option enables the use of Scaled Dot-Product Attention (SDPA) within the model. SDPA is a fundamental component of transformer models that calculates the attention scores between queries and keys. It scales the dot products by the dimensionality of the keys to stabilize gradients during training. This mechanism is particularly useful for handling long sequences and can potentially improve the model’s ability to capture long-range dependencies.

    --sdpa

`--multires_noise_iterations` and `--multires_noise_discount` #

Multi-resolution noise is a new approach that adds noise at multiple resolutions to an image or latent image during the training of diffusion models. A model trained with this technique can generate visually striking images with a distinct aesthetic compared to the usual outputs of diffusion models.

A model trained with multi-resolution noise can generate a more diverse range of images than regular stable diffusion, including extremely light or dark images. These have historically been challenging to achieve without resorting to using a large number of sampling steps.

This technique is particularly beneficial when working with small datasets but you I don’t think you should ever not use it.

The --multires_noise_discount parameter controls the extent to which the noise amount at each resolution is weakened. A value of 0.1 is recommended. The --multires_noise_iterations parameter determines the number of iterations for adding multi-resolution noise, with a recommended range of 6 to 10.

Please note that --multires_noise_discount has no effect without --multires_noise_iterations.

Implementation Details #

The get_noise_noisy_latents_and_timesteps function samples noise that will be added to the latents. If args.noise_offset is true, it applies a noise offset. If args.multires_noise_iterations is true, it applies multi-resolution noise to the sampled noise.

The function then samples a random timestep for each image and adds noise to the latents according to the noise magnitude at each timestep. This is the forward diffusion process.

The pyramid_noise_like function generates noise with a pyramid structure. It starts with the original noise and adds upscaled noise at decreasing resolutions. The noise at each level is scaled by a discount factor raised to the power of the level. The noise is then scaled back to roughly unit variance. This function is used to implement the multi-resolution noise.

    --multires_noise_iterations=10 --multires_noise_discount=0.1

`--sample_prompts` and `--sample_sampler` and `--sample_every_n_steps` #

You have the option of generating images during training so you can check the progress, the argument let’s you pick between different samplers, by default it is on ddim, so you better change it!

You can also use --sample_every_n_epochs instead which will take precedence over steps. The k_ prefix means karras and the _a suffix means ancestral.

    --sample_prompts=/training_dir/sample-prompts.txt --sample_sampler="euler_a" --sample_every_n_steps=100

My recommendation for Pony is to use euler_a for toony and for realistic k_dpm_2.

Your sampler options include the following:

ddim, pndm, lms, euler, euler_a, heun, dpm_2, dpm_2_a, dpmsolver, dpmsolver++, dpmsingle, k_lms, k_euler, k_euler_a, k_dpm_2, k_dpm_2_a

So, the whole thing would look something like this:

accelerate launch --num_cpu_threads_per_process=2  "./sdxl_train_network.py" \
    --lowram \
    --pretrained_model_name_or_path="/ponydiffusers/" \
    --train_data_dir="/training_dir" \
    --resolution="1024,1024" \
    --output_dir="/output_dir" \
    --enable_bucket \
    --min_bucket_reso=256 \
    --max_bucket_reso=1024 \
    --network_alpha=4 \
    --save_model_as="safetensors" \
    --network_module="lycoris.kohya" \
    --network_args \
               "preset=full" \
               "conv_dim=256" \
               "conv_alpha=4" \
               "use_tucker=False" \
               "use_scalar=False" \
               "rank_dropout_scale=False" \
               "algo=locon" \
               "train_norm=False" \
               "block_dims=8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8" \
               "block_alphas=0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625" \
    --network_dropout=0 \
    --lr_scheduler="cosine" \
    --learning_rate=0.0001 \
    --unet_lr=0.0001 \
    --text_encoder_lr=0.0001 \
    --network_dim=8 \
    --output_name="yifftoolkit" \
    --scale_weight_norms=1 \
    --no_half_vae \
    --save_every_n_epochs=50 \
    --mixed_precision="fp16" \
    --save_precision="fp16" \
    --caption_extension=".txt" \
    --cache_latents \
    --cache_latents_to_disk \
    --optimizer_type="AdamW" \
    --max_grad_norm=1 \
    --keep_tokens=1 \
    --max_data_loader_n_workers=8 \
    --bucket_reso_steps=32 \
    --multires_noise_iterations=10 \
    --multires_noise_discount=0.1 \
    --log_prefix=xl-locon \
    --gradient_accumulation_steps=12 \
    --gradient_checkpointing \
    --train_batch_size=8 \
    --dataset_repeats=0 \
    --max_train_steps=400 \
    --shuffle_caption \
    --sdpa \
    --sample_prompts=/training_dir/sample-prompts.txt \
    --sample_sampler="euler_a" \
    --sample_every_n_steps=100

LoRA Training Guide #

Subsections #

Installation Tips #

Pony Training #

Download Pony in Diffusers Format #

Sample Prompt File #

Training Commands #

accelerate launch #

--lowram #

--pretrained_model_name_or_path #

--output_dir #

--train_data_dir #

--resolution #

--enable_bucket #

--bucket_no_upscale #

--min_bucket_reso and --max_bucket_reso #

--network_alpha #

--save_model_as #

--network_module #

--network_args #

preset #

conv_dim and conv_alpha #

module_dropout and dropout and rank_dropout #

use_tucker #

use_scalar #

rank_dropout_scale #

algo #

train_norm #

block_dims #

block_alphas #

--network_dropout #

--lr_scheduler #

--lr_scheduler_num_cycles #

--learning_rate and --unet_lr and --text_encoder_lr #

--network_dim #

--output_name #

--scale_weight_norms #

--max_grad_norm #

--no_half_vae #

--save_every_n_epochs and --save_last_n_epochs or --save_every_n_steps and --save_last_n_steps #

--mixed_precision #

--save_precision #

--caption_extension #

--cache_latents and --cache_latents_to_disk #

--optimizer_type #

--dataset_repeats #

--max_train_steps #

--shuffle_caption #

--sdpa or --xformers or --mem_eff_attn #

--multires_noise_iterations and --multires_noise_discount #