Data Augmentation Techniques Using Text-to-Image Diffusion Models for Enhanced Data Diversity

Abstract

Data augmentation is a widely used technique to enhance the performance of deep learning models. However, traditional augmentation methods, dependent solely on original data, often fall short in maintaining data diversity and generalization capabilities. In this paper, we propose a novel data augmentation approach leveraging pretrained text-to-image diffusion models to generate diverse and contextually rich images. Our approach integrates three advanced techniques: rich-text prompts, multi-object image generation, and inpainting. We demonstrate the effectiveness of these methods through extensive experiments on the Oxford-IIIT Pets and Caltech-101 datasets, where our diffusion-based augmentations significantly improved downstream classification accuracy and model generalization. No-tably, the inpainting technique excels in handling class imbalances by balancing the diversity and structural integrity of original data, while rich-text prompts and multi-object generation offer substantial gains by enhancing diversity and realism. Additionally, our methods show enhanced generalization to unseen data, proving their robustness and applicability to various deep learning tasks. © 2024 IEEE.