(OLD) ~~The Shrunk LoRA Guide~~

(OLD) The Shrunk LoRA Guide #

⚠️ NOTE: This document has been deprecated #

The shrunk version of all my LoRAs should be found now right under the original download links for each model here.

At some point I realized I get the best results by shrinking my LoRAs with -3.55 threshold using the Frobenius Norm recipe, so this page is now totally obsolete! Sadly, this process sometimes, like in the case of normal SDXL LoRAs (such as blue_frost) makes the shrinking process less.. shrinky.. So this page is still a little bit relevant, if you are looking for only really tiny LoRAs and don’t care about quality loss!

But Here it is Anyway #

The image is a humorous and imaginative scene depicting a man and a woman looking closely at a tiny wolf on a spoon. The man, wearing glasses and a sweatshirt with a large “T” on it, is holding a magnifying glass to better see the miniature wolf. The woman, peering closely beside him, also seems amazed and curious about the small wolf. The tiny wolf, adorned with a hotdog bun costume, adds a whimsical and playful element to the image.

NOTE: Some of these boys have XY plots with them, you can click on them to download the full sized image. The first row is always the original LoRA.

Method #1 #

amalia-v2e400 #

amicus-v1e200 #

arielsatyr-v1e400 #

arielsatyr-v2e400 #

auroth-v1e250 #

bdsm-v1e400 #

blaidd-v1e400 #

An XY plot.

blp-v1e400 #

blue_frost #

butterchalk-v3e400 #

by_clybius-v1e400 #

by_hibbary-v1e400 #

by_himari-v1e400 #

cecily_lin-v1e37 #

cervine_penis-v1e400 #

chillet-v3e10 #

chunie-v1e5 #

cooliehigh-v1e45 #

dagasi-v1e134 #

darkgem-v1e4 #

dharrel-v1e350 #

dharrel-v2e400 #

forefront-foot_perspective-v1e400 #

foxparks-v2e134 #

furry_sticker-v1e250 #

goronic-v1e1 #

greg_rutkowski-v1e400 #

hamgas-v1e400 #

An XY plot.

honovy-v1e4 #

hoodwink-v1e400 #

jayjay-v1e400 #

jinxit-v1e10 #

kame_3-v1e80 #

kenket-v1e4 #

lodestone-v1e400 #

louart-v1e10 #

lovander-v3e10 #

magic-v1e400 #

maliketh-v1e1 #

martlet-v1e200 #

non-euclidean_sex-v1e400 #

pup_mask-v1e400 #

ramona-v1e400 #

realistic-v4e400 #

realistic-v4e500 #

skecchiart-v1e134 #

skiltaire-v1e400 #

small_dom_big_sub-v2e400 #

space-v1e500 #

spaceengine-v1e500 #

spectrumshift-v1e400 #

squishy-v1e10 #

tibetan-v2e500 #

veemon-v1e400 #

whisperingfornothing-v1e58 #

wickerbeast-v1e500 #

wjs07-v1e200 #

wolfy-nail-v1e400 #

woolrool-v1e4 #

Method #2 - Frobenius Norm #

Click to reveal links

amalia-v2e400

amicus-v1e200

arielsatyr-v1e400

arielsatyr-v2e400

auroth-v1e250

bdsm-v1e400

blaidd-v1e400

An XY plot.

blp-v1e400

blue_frost

butterchalk-v3e400

by_clybius-v1e400

by_hibbary-v1e400

by_himari-v1e400

cecily_lin-v1e37

cervine_penis-v1e400

chillet-v3e10

chunie-v1e5

cooliehigh-v1e45

dagasi-v1e134

darkgem-v1e4

dharrel-v1e350

dharrel-v2e400

forefront-foot_perspective-v1e400

foxparks-v2e134

furry_sticker-v1e250

goronic-v1e1

greg_rutkowski-v1e400

hamgas-v1e400

An XY plot.

honovy-v1e4

hoodwink-v1e400

jayjay-v1e400

jinxit-v1e10

kame_3-v1e80

kenket-v1e4

lodestone-v1e400

lovander-v3e10

louart-v1e10

magic-v1e400

maliketh-v1e1

martlet-v1e200

non-euclidean_sex-v1e400

pup_mask-v1e400

ramona-v1e400

realistic-v4e400

realistic-v4e500

skecchiart-v1e134

skiltaire-v1e400

small_dom_big_sub-v2e400

space-v1e500

spaceengine-v1e500

spectrumshift-v1e400

squishy-v1e10

tibetan-v2e500

veemon-v1e400

whisperingfornothing-v1e58

wickerbeast-v1e500

wjs07

wolfy-nail-v1e400

woolrool-v1e4

Method #3 #

Click to reveal links

amalia-v2e400

amicus-v1e200

arielsatyr-v1e400

arielsatyr-v2e400

auroth-v1e250

bdsm-v1e400

blaidd-v1e400

An XY plot.

blp-v1e400

blue_frost

butterchalk-v3e400

by_clybius-v1e400

by_hibbary-v1e400

by_himari-v1e400

cecily_lin-v1e37

cervine_penis-v1e400

chillet-v3e10

chunie-v1e5

cooliehigh-v1e45

dagasi-v1e134

darkgem-v1e4

dharrel-v1e350

dharrel-v2e400

forefront-foot_perspective-v1e400

foxparks-v2e134

furry_sticker-v1e250

goronic-v1e1

greg_rutkowski-v1e400

hamgas-v1e400

An XY plot.

honovy-v1e4

hoodwink-v1e400

jayjay-v1e400

jinxit-v1e10

kame_3-v1e80

kenket-v1e4

lodestone-v1e400

louart-v1e10

lovander-v3e10

magic-v1e400

maliketh-v1e1

martlet-v1e200

non-euclidean_sex-v1e400

pup_mask-v1e400

ramona-v1e400

realistic-v4e400

realistic-v4e500

skecchiart-v1e134

skiltaire-v1e400

small_dom_big_sub-v2e400

space-v1e500

spaceengine-v1e500

spectrumshift-v1e400

squishy-v1e10

tibetan-v2e500

veemon-v1e400

whisperingfornothing-v1e58

wickerbeast-v1e500

wjs07-v1e200

wolfy-nail-v1e400

woolrool-v1e4

What the Heck is a Subspace Factor #

The subspace factor is a measure of how well the subspace spanned by the LoRA layer’s rank-update matrices (U and Vh) aligns with the subspace spanned by the weight matrix of the corresponding layer in the base model.

Specifically, the base_subspace_factors attribute computed in the DLoRA class represents the dot products between the columns of Vh and the rows of the base model weight matrix W_base, multiplied by the columns of U. These dot products measure the correlations between the subspaces spanned by the LoRA update and the base model weights.

A high subspace factor indicates that the LoRA layer is updating a subspace that is already present and important in the base model weights. Conversely, a low subspace factor suggests that the LoRA layer is introducing new directions that were not strongly represented in the base model.

The subspace_ratios attribute computes the ratio of the LoRA singular values to the absolute values of the subspace factors. This gives a sense of how much the LoRA layer is scaling the existing subspaces versus introducing new subspaces relative to the base model.

In summary, the subspace factor quantifies the alignment between the LoRA update and the base model weight subspaces, providing insight into how the LoRA layer is adapting the base model.

What is the Spectral Norm #

The spectral norm (also known as the operator norm or matrix norm) is a measure of the maximum singular value of a matrix. It is calculated as:

spectral_norm(W) = σ_max(W)

Where σ_max(W) is the largest singular value of the matrix W.

The base_spectral_norm attribute of the DLoRA class is computed as:

self.base_spectral_norm = pt.svd_lowrank(W_base, q=1, niter=niter)[1][0].item()

Here, W_base is the weight matrix of the corresponding layer in the base model, flattened into a 2D matrix. pt.svd_lowrank computes an approximation of the singular value decomposition, returning the largest singular value σ_max(W_base) with q=1.

The spectral norm provides a scale for the singular values of a matrix. In the case of LoRA, the sval_ratios attribute computes the ratio of the LoRA singular values to the base model’s spectral norm:

self.sval_ratios = self.S / self.base_spectral_norm

This ratio gives an indication of how much the LoRA layer is scaling the singular values of the base model weight matrix.

In summary, the spectral norm is the maximum singular value of a matrix, and it is used in the LoRA context to normalize and interpret the scale of the LoRA singular values relative to the base model weights.

Jun 30, 2024
Edit this page