Asymmetric Bias in Text-to-Image Generation with Adversarial Attacks

[Download Paper] [Source Code]

Text-to-image models are widely used, so it matters when adversarial suffixes can rewrite what they draw. We study entity swapping: adding a suffix that tries to replace one entity in a prompt with another. Attack success is asymmetric — it is much easier to turn “a human dancing in the rain” into a robot than the reverse. Probing the model’s beliefs gives a useful signal of when an attack is likely to work, with success rates ranging from about 60% in some conditions to under 5% in others.