DiffSteer

Overview

The usual ways of stopping a text-to-image model from producing something all have gaps. Keyword filters get bypassed by rephrasing. Post-hoc classifiers slap a censor bar on an image the model already generated. Fine-tuning methods like ESD[3] and UCE[4] do erase the concept, but they rewrite model weights, need retraining per concept, and risk collateral damage to unrelated generation.

DiffSteer asks a different question: can you find the concept inside the model and subtract it at inference time, with no weight changes at all? The place to look is the CLIP[8] text encoder that conditions Stable Diffusion[1]. If a concept has a direction in that encoder's activation space, removing the direction means the U-Net never receives the concept signal in the first place.

Prior work, Concept Steerers[9], showed this works with a k-sparse autoencoder trained on a single CLIP layer, building on sparse dictionary learning from the language-model interpretability literature[5][6]. My contribution is to extend it to CrossCoders[7]: sparse autoencoders with a shared encoder and separate per-layer decoders, trained across several layers at once, on the hypothesis that concept information is spread across encoder depth rather than concentrated in one layer.

Method

Sparse dictionaries over CLIP activations

Raw activations are dense and polysemantic, so you cannot cleanly ablate a concept by zeroing coordinates. Sparse dictionary learning re-expresses an activation as a sparse combination of an overcomplete learned basis, whose features are far closer to monosemantic.

Both models were trained on activations harvested by forward-passing I2P[2] benchmark prompts through the frozen CLIP text encoder of Stable Diffusion v1.4, covering 400M+ tokens from I2P and CC3M/LAION captions.

• K-Sparse Autoencoder: encodes a layer-9 token activation (d = 768) with a top-k constraint (k = 48) into a 6144-dimensional latent (8× expansion). Training minimises fraction of variance unexplained plus an auxiliary loss that revives dead latents.
• CrossCoder: concatenates layers 4, 8 and 9 into a single 3 × 768 = 2304-dim vector, encodes it through one shared encoder into a sparse code, then reconstructs each layer with its own decoder. An entropy term (α_spec = 0.01) pushes each latent to load on a single layer, which keeps the features interpretable.

Finding and subtracting the concept

For the SAE, the steering vector is the decoded dense encoding of a concept phrase ("naked, nude, bare, exposed, stripped, topless"), scaled by α = −0.4. For the CrossCoder, concept latents are found contrastively: the positive part of Δz = (z_c − z_∅): and the top-32 latents by Δz define a mask. Per-layer steering vectors are decoded from the masked code at α = −0.5.

At inference, forward hooks apply the steering vectors simultaneously at all three layers. Each hook adds the vector to the conditional embedding and subtracts it from the unconditional one, so the edit is amplified through classifier-free guidance[10] rather than diluted by it.

Configuration

Both trained on a TPU v4-8 for 400M tokens. Adam, lr 4×10⁻⁴, batch size 4096, no weight decay.

Results

Evaluated on 80 prompts spanning photorealistic photography, classical painting styles (Klimt, Monet, Schiele), digital art, and indirect phrasings, at a fixed seed with 20 DDIM steps and guidance scale 7.5. Nudity rate is the fraction of images with NudeNet detections (threshold 0.45) across five anatomical classes; CLIP score is image–prompt cosine similarity under CLIP ViT-B/32.

CrossCoder steering cuts the nudity rate by 50 percentage points from baseline, versus 43.8 for the single-layer SAE, a 55% relative reduction in residual nudity (11.2% → 5.0%). The cost is 1.09 CLIP points relative to SAE, which is the honest trade-off: removing a concept from the conditioning signal necessarily pulls the generation slightly away from the prompt.

Where single-layer steering falls short

A pattern the binary nudity rate does not capture: SAE steering frequently leaves peripheral skin regions exposed while successfully blocking the most visible anatomy. CrossCoder steering produces complete coverage far more consistently. This is what you would expect if nudity information is spread across encoder depth and a single-layer dictionary only captures part of it.

Takeaway

With no changes to model weights, cross-layer sparse steering reduces nudity generation from 55.0% to 5.0%. Beyond the safety result, this is evidence about representation: the fact that spanning three layers beats a single layer by more than 2× on residual nudity suggests the concept is spread across encoder depth, and that cross-layer sparse dictionaries capture that distribution more completely than single-layer alternatives do.

References

1. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. CVPR, 2022.
2. P. Schramowski, M. Brack, B. Deiseroth, K. Kersting. Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models. CVPR, 2023.
3. R. Gandikota, J. Materzynska, J. Fiotto-Kaufman, D. Bau. Erasing Concepts from Diffusion Models. ICCV, 2023.
4. R. Gandikota, H. Orgad, Y. Belinkov, J. Materzynska, D. Bau. Unified Concept Editing in Diffusion Models. WACV, 2024.
5. T. Bricken, A. Templeton, J. Batson, B. Chen, T. Conerly, et al. Towards Monosemanticity: Decomposing Language Models with Dictionary Learning. Transformer Circuits Thread, Anthropic, 2023.
6. H. Cunningham, A. Ewart, L. Riggs, R. Huben, L. Sharkey. Sparse Autoencoders Find Highly Interpretable Features in Language Models. ICLR, 2024.
7. J. Lindsey, A. Templeton, J. Marcus, T. Conerly, J. Batson, C. Olah. Sparse Crosscoders for Cross-Layer Features and Model Diffing. Transformer Circuits Thread, Anthropic, 2024.
8. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, et al. Learning Transferable Visual Models From Natural Language Supervision. ICML, 2021.
9. D. Kim, D. Ghadiyaram. Concept Steerers: Leveraging K-Sparse Autoencoders for Controllable Generations. arXiv:2501.19066, 2025.
10. J. Ho, T. Salimans. Classifier-Free Diffusion Guidance. NeurIPS Workshop on Deep Generative Models, 2021.