Split Your Sigmas #

---

Introduction #

---

In the world of diffusion models, controlling the sampling process is key to achieving desired results. One powerful technique that’s often overlooked is splitting sigma values during sampling. This approach allows for more granular control over how noise is removed throughout the diffusion process, enabling you to apply different sampling strategies at different denoising stages.

This guide will introduce you to sigma splitting in ComfyUI, demonstrate how to implement it through a practical workflow, and explain why it can significantly improve your image generation results.

Understanding Sigmas in Diffusion Models #

---

Before diving into splitting sigmas, let’s briefly understand what sigma values represent in diffusion models:

• Sigma values correspond to noise levels in the diffusion process
• They typically follow a schedule that gradually decreases from high noise to low noise
• Early steps (high sigmas) establish basic composition and structure
• Later steps (low sigmas) refine details and textures

By splitting sigmas, we can apply different sampling configurations or even different models to different parts of this process, allowing for specialized handling of structure versus details.

The SplitSigmas Node #

---

ComfyUI offers several nodes for manipulating sigma values:

1. SplitSigmasDenoise: Splits sigma schedules into high and low portions based on a threshold
2. SplitAtSigma: Divides a sigma schedule at a specific step number
3. KarrasScheduler: Generates a sequence of sigma values following the Karras schedule

The key node we’ll focus on is SplitAtSigma, which allows us to precisely divide our sampling process at a specific step.

Setting Up a Sigma Splitting Workflow #

---

Let’s implement a workflow that uses sigma splitting to get better control over different stages of the generation process:

1. We’ll start with a standard model setup, including model loading and conditioning
2. We’ll configure separate samplers with different settings for structure and details
3. We’ll use the split sigmas to direct each part of the diffusion process to the appropriate sampler

Here’s how our example workflow is structured:

1. Model setup with perturbed attention and downscaling for improved base quality
2. A LoRA loaded to enhance the model’s capabilities
3. Two separate sampling paths:
   • First path handles early denoising steps (structure, composition)
   • Second path refines details with specialized settings

The Workflow #

---

The Workflow Explained #

---

Let’s break down the key components of our workflow:

1. Sigma Generation and Splitting #

$$ \text{KarrasScheduler} \rightarrow \text{SplitSigmasDenoise} \rightarrow \text{SplitAtSigma} $$

This sequence:

1. Generates sigma values using the Karras scheduler
2. Splits them into high and low categories
3. Further divides the low sigmas at a specific step (step 7 in our example)

This split gives us two separate sigma sequences to work with.

2. Main Sampling Paths #

We have two distinct sampling paths:

Path 1 (Early Steps):

$$ \text{CFGGuider (scale 1)} \rightarrow \text{SamplerCustomAdvanced} \rightarrow \text{Output Latent} $$

Path 2 (Later Steps):

$$ \text{CFGGuider (scale 7)} \rightarrow \text{DetailDaemonSampler} \rightarrow \text{SamplerCustomAdvanced} \rightarrow \text{Final Output} $$

Notice how we use different CFG scales for the different stages - a lower scale for early structure and a higher scale for later details. This allows the initial composition to form more freely while enforcing the prompt more strictly during detail generation.

3. Detail Enhancement #

For the second phase, we enhance detail generation with:

$$ \text{SamplerEulerAncestral} \rightarrow \text{DetailDaemonSamplerNode} $$

For details on DetailDaemon, please see our dedicated article on DetailDaemon.

Why Split Your Sigmas? #

---

Splitting sigmas offers several key advantages:

1. Targeted Control: Different CFG scales for structure vs. details
2. Better Detail Preservation: Specialized samplers for fine details
3. Reduced Artifacts: More careful handling of critical steps
4. Creativity Balance: Freedom in composition while maintaining prompt adherence

In practice, this often results in images with better composition and more refined details than a single-pass approach.

Implementation Example #

---

Let’s look at the complete workflow and explain the key nodes in more detail:

Model Preparation #

First, we enhance our base model with specialized techniques:

$$ \text{LoadCheckpoint} \rightarrow \text{PerturbedAttentionGuidance} \rightarrow \text{PatchModelAddDownscale} $$

This sequence:

1. Loads our base model (PonyDiffusionV6XL)
2. Applies Perturbed Attention Guidance (PAG) for improved structural coherence
3. Adds downscaling through Kohya DeepShrink for enhanced processing at higher resolutions

Perturbed Attention Guidance (PAG) #

For details on Perturbed Attention Guidance (PAG), please see our dedicated article on Perturbed Attention Guidance.

PatchModelAddDownscale (Kohya DeepShrink) #

For details on PatchModelAddDownscale (Kohya DeepShrink), please see our dedicated article on PatchModelAddDownscale.

LoRA Application #

We then apply a LoRA to enhance the model:

$$ \text{by\_woolrool-v2s400 (1.0)} $$

Noise and Latent Setup #

$$ \text{RandomNoise} + \text{EmptyLatentImage} (1536 \times 1536) $$

These provide the starting point for our generation process.

Splitting the Process #

The key sigma splitting components:

$$ \text{KarrasScheduler (60 steps)} \rightarrow \text{SplitSigmasDenoise (threshold 1)} \rightarrow \text{SplitAtSigma (step 7)} $$

This divides our 60-step process into distinct phases.

Dual Processing Paths #

The workflow implements two parallel processing paths:

1. Split Sigmas Path: Uses the first part of the split sigma schedule with a CFG scale of 1, focusing on setting up the basic structure.

2. Standard Path: Uses the full sigma schedule with a CFG scale of 7, producing a comparison image that adheres more strongly to the prompt throughout the entire process.

This dual approach allows you to see the difference between the split sigma technique and a standard approach side by side.

Results and Comparisons #

---

When comparing results with and without sigma splitting, you’ll notice:

1. Better Composition: The initial structure forms more naturally
2. Enhanced Details: Finer details appear sharper and more coherent
3. Reduced Unwanted Artifacts: Fewer distortions in complex areas
4. Improved Prompt Adherence: Better balance between creativity and following the prompt

Tips for Effective Sigma Splitting #

---

To get the most from this technique:

1. Experiment with Split Points: The ideal split point varies by model and image complexity
2. Adjust CFG Scales Carefully: Find the right balance between low CFG for structure and high CFG for details
3. Consider Different Samplers: Try various sampler combinations for each phase
4. Match with Scheduler: Karras schedulers work particularly well with splitting techniques

Conclusion #

---

Splitting sigmas is a powerful technique that gives you greater control over the diffusion process, allowing for specialized handling of different denoising stages. By implementing this approach in ComfyUI, you can achieve more refined outputs with better composition and sharper details.

The workflow provided in this guide serves as a starting point - feel free to experiment with different split points, samplers, and CFG scales to find the perfect combination for your specific use case.

Further Reading #

---

For those interested in a more academic approach to optimizing different stages of the diffusion process, the research by Zhang et al. (2024) introduces a “multi-stage framework” that similarly divides the diffusion process into distinct intervals, each handled by specialized decoders while sharing a universal encoder. Their work demonstrates significant improvements in both sampling quality and training efficiency across multiple diffusion model architectures.

Zhang, H., Lu, Y., Alkhouri, I., Ravishankar, S., Song, D., & Qu, Q. (2024). Improving Efficiency of Diffusion Models via Multi-Stage Framework and Tailored Multi-Decoder Architectures. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024). arXiv:2312.09181

Another complementary approach worth exploring is “Align Your Steps” (AYS) by Sabour et al. (2024), which focuses on optimizing the exact sequence of noise levels in the sampling schedule rather than using hand-crafted schedules. While sigma splitting divides the process into phases with different settings, AYS optimizes when to place each denoising step for maximum effectiveness. Their work reveals that dataset-specific optimized schedules can significantly improve output quality, especially in few-step synthesis regimes, without requiring any changes to model architecture or training process.

Sabour, A., Fidler, S., & Kreis, K. (2024). Align Your Steps: Optimizing Sampling Schedules in Diffusion Models. arXiv:2404.14507

---

Comparison #

---

---