Custom ComfyUI Workflow with the Krita AI Plugin #

Introduction #

Integrating ComfyUI with Krita’s AI Diffusion plugin creates a powerful digital art workflow. This combination allows you to generate and edit AI images directly within Krita, using custom ComfyUI workflows and your favorite diffusion models, including SDXL and Pony Diffusion. This guide walks you through the setup process and shows you how to create an effective workflow between these tools.

Installation #

Getting the Krita AI Plugin #

First visit the project’s releases page on GitHub, then download the latest zip file and make sure you click the first zip for it instead any of the Source code archives!

Installing Required ComfyUI Components #

To establish a connection between Krita and ComfyUI, you’ll need some specific custom nodes:

cd ~/ComfyUI/custom_nodes
git clone https://github.com/Acly/comfyui-tooling-nodes

If you’re planning to use the Translate Text node (helpful for multilingual prompts), install the following dependency:

pip install argostranslate

Connecting to ComfyUI #

After installing the plugin in Krita, you’ll need to connect it to your ComfyUI instance:

1. Open Krita and access the AI Diffusion docker panel
2. Click on the workspace selection button on the top left side of the Docker
3. Select “Custom” to configure your own workflow

This allows you to use your own ComfyUI workflows rather than the default ones provided by the plugin. Custom workflows give you access to all your favorite models, LoRAs, and generation techniques.

Essential Nodes For Working with Krita Plugins in ComfyUI #

The Krita Output Node #

The Krita Output node is essential for sending generated images back to Krita. Connect this node to the VAE Decode’s image output in your workflow.

This connection ensures that your generated diffusion images are sent back to your Krita canvas.

The Krita Canvas Node #

The Krita Canvas node handles communication between Krita’s selection and ComfyUI:

1. Connect the width and height outputs to nodes that require canvas dimensions (such as Empty Latent Image or CLIPTextEncodeSDXL)
2. Connect the seed input to the KSampler’s seed for consistent generation

This setup ensures your ComfyUI workflow respects the dimensions and selections from your Krita canvas.

Converting Widgets to Inputs #

A quick reminder that in order to convert a widget to an input all you need to do, is right click on the node and find the widget under Convert Widget to Input like this:

If you are using the new frontend you can also just drag the seed noodle directly over the widget. 🐺

Recommended Workflow Structure #

For optimal results with the Krita AI Plugin and ComfyUI, structure your workflow as follows:

1. Model Selection - Start with your preferred checkpoint model (SDXL, Pony Diffusion XL, etc.)
2. Prompt Engineering - Use the CLIP Text Encode nodes with your prompts
3. Canvas Integration - Connect the Krita Canvas node to provide dimensions
4. Generation Parameters - Set up your KSampler with appropriate settings
5. LoRA Integration - Add custom LoRA models if desired
6. Output Connection - Finish with the Krita Output node to send results back to Krita

This structure allows for a seamless workflow between Krita’s drawing tools and ComfyUI’s generation capabilities. You can find additional custom nodes to enhance your workflow in our ComfyUI Custom Scripts guide.

Advanced Tips #

Inpainting Workflow #

For effective inpainting in Krita with ComfyUI:

1. Create a selection or mask in Krita
2. Use the Krita Canvas node’s mask output
3. Connect it to an inpainting node in ComfyUI
4. Generate content only within your selected area

This technique is perfect for generating specific elements while preserving the rest of your artwork.

Working with Control Nets #

You can enhance your Krita workflow with ControlNet:

1. Create a sketch or line drawing in Krita
2. Use the Krita Layer node to send it to ComfyUI
3. Process it through a ControlNet preprocessor node
4. Connect the result to your main generation workflow

This allows for precise control over your AI-generated elements based on your Krita drawings.

Troubleshooting #

If you encounter issues with the Krita-ComfyUI connection:

• Ensure ComfyUI is running and accessible (default: http://127.0.0.1:8188)
• Check that the comfyui-tooling-nodes are properly installed
• Verify your workflow contains both Krita Canvas and Krita Output nodes
• Restart both Krita and ComfyUI if the connection fails

For more experimental features and advanced workflows, check our Experimental Stuff guide.

Conclusion #

Integrating the Krita AI Diffusion plugin with ComfyUI creates a powerful workflow that combines traditional digital art with AI generation. This setup gives you the best of both worlds:

1. Krita’s powerful drawing and painting tools
2. ComfyUI’s flexible node-based diffusion workflow
3. The ability to use custom models, LoRAs, and workflows
4. Seamless integration between drawing and AI generation

By following this guide, you can create a customized workflow that enhances your creative process, allowing for rapid iteration and experimentation with diffusion models directly within your favorite digital art software.