Regularization Images #

Regularization images are a crucial component of the DreamBooth training methodology that help prevent “language drift” in the model. This guide explains what they are and how to use them effectively.

What Are Regularization Images? #

Regularization images are examples of the base class (or generic concept) that your specific subject belongs to. They help the model maintain its understanding of the general concept while learning your specific instance.

For example:

• If training a “wickerbeast” concept, your regularization images would be general “animal” or “furry creature” images
• If training a specific person, your regularization images would be generic human images
• If training an art style, your regularization images would be generic artwork images

Why Regularization Images Are Necessary #

Without regularization images, the model often experiences “overfitting” and “language drift” where:

1. Overfitting: The model becomes too focused on reproducing your specific training images rather than generalizing
2. Language Drift: The model forgets what the general class looks like and associates the class word entirely with your specific instance

For example, if you train a model on “wickerbeast” images without regularization, the model might:

• Begin to generate wickerbeasts whenever you prompt for “animal” or “furry”
• Lose its ability to generate other animal types when the class word is used
• Apply wickerbeast-specific features to unrelated generations

How to Create Regularization Images #

Manual Collection Method #

1. Collect 50-100 images that represent the general class of your subject
2. Place them in a separate directory (e.g., reg_data/)
3. Create caption files with generic class terms

Using Text-to-Image Generation #

You can generate regularization images using Stable Diffusion:

python /path/to/sd-scripts/gen_img.py \
  --ckpt="/path/to/model.safetensors" \
  --outdir="reg_data" \
  --prompt="[generic class term], high quality, detailed" \
  --scale=7.5 \
  --n_iter=10 \
  --steps=30 \
  --W=512 --H=512

Using Dataset Class Extraction #

• Use class images from public datasets
• Filter for appropriate images that represent your generic class

Recommended Ratio and Settings #

For DreamBooth training in sd-scripts, a good rule of thumb is:

• Ratio: 10-20 regularization images for every unique subject/concept you’re training
• If training with 50 concept images, aim for 100-200 regularization images
• In the TOML configuration, mark the regularization subset with is_reg = true
• Set prior_loss_weight = 1.0 in your training command (this balances regularization influence)

Directory Structure for Regularization #

Basic Command-Line Structure #

training_dir/
├── train_data/               # Your concept images
│   ├── concept1.png
│   ├── concept1.txt          # Caption: "your concept, attribute1, attribute2"
│   └── ...
└── reg_data/                 # Regularization images
    ├── reg1.png
    ├── reg1.txt              # Caption: "generic class"
    └── ...

TOML Configuration Structure #

[[datasets]]
resolution = 512
batch_size = 4

  [[datasets.subsets]]
  image_dir = "datasets/concept"
  class_tokens = "your concept"
  num_repeats = 10

  [[datasets.subsets]]
  is_reg = true              # This marks it as a regularization dataset
  image_dir = "datasets/reg"
  class_tokens = "generic class"    # Generic class term
  num_repeats = 1            # Typically use fewer repeats for regularization

Special Considerations #

When preparing regularization images, consider these tips:

1. Avoid using images of your specific concept in your regularization set
2. Focus on general examples with varied appearances
3. Use broader class terms in your regularization captions
4. Include different poses and contexts to help the model generalize better

Command-Line Options for Regularization #

When using sd-scripts’ basic command-line approach, you can specify your regularization directory:

accelerate launch train_network.py \
  --pretrained_model_name_or_path="./models/stable-diffusion-v1-5" \
  --train_data_dir="./train_data" \           # Directory with your concept images
  --reg_data_dir="./reg_data" \               # Regularization images directory
  --prior_loss_weight=1.0 \                   # Weight for regularization loss
  # ...other training parameters

Examples for Different Subjects #

Character/Species (e.g., Wickerbeast) #

For a wickerbeast character:

• Training images: Your specific wickerbeast character
• Regularization images: Generic anthropomorphic animals (not wickerbeasts)
• Training captions: “wickerbeast, red fur, standing”
• Regularization captions: “animal” or “anthropomorphic animal”

Person/Portrait #

For a specific person:

• Training images: Photos of the specific person
• Regularization images: Generic human portraits
• Training captions: “person name, portrait”
• Regularization captions: “person” or “human portrait”

Art Style #

For an art style:

• Training images: Examples of the specific art style
• Regularization images: Generic artwork in various styles
• Training captions: “art style name, artwork”
• Regularization captions: “artwork” or “illustration”

By properly implementing regularization images in your training, you’ll achieve a better balance where your model learns to generate your specific concept while maintaining the ability to generate other types when requested.

Next Steps #

With your regularization images prepared, the next steps in your training journey are:

1. Automatically tag and caption your dataset to create descriptive metadata

2. Learn about training parameters to get the best results from your model

3. Set up training monitoring to track your model’s progress