PixTention: Dynamic pixel-level adapter using attention maps

Abstract

Recent advances in image generation have popularized adapter-based fine-tuning, where Low-Rank Adaptation (LoRA) modules enable efficient personalization with minimal storage costs. However, current approaches often suffer from two key limitations: (1) manually selecting suitable LoRA adapters is time-consuming and requires expert knowledge, and (2) applying multiple adapters globally can introduce style interference and reduce image fidelity, especially for prompts with multiple distinct concepts. We propose PixTention, a framework that addresses these challenges via a novel three-stage process: Curator, Selector, and Integrator. The Curator uses a vision-language model to generate enriched semantic descriptions of LoRA adapters and clusters their embeddings based on shared visual themes, enabling efficient hierarchical retrieval. The Selector embeds user prompts and first selects the most relevant adapter clusters, then identifies top-K adapters within them via cosine similarity. The Integrator leverages cross-attention maps from diffusion models to assign each retrieved adapter to specific semantic regions in the output image, ensuring localized, prompt-aligned transformations without global style overwriting. Through experiments on COCO-Multi and a custom StyleCompose dataset, PixTention achieves higher CLIP scores, IoU and lower FID than baseline retrieval and reranking methods, demonstrating superior text-image alignment and image realism. Our results highlight the importance of semantic clustering, region-specific adapter composition, and cross-modal alignment in advancing controllable, high-fidelity image generation.