Theoretical Analysis of Contrastive Learning under Imbalanced Data: From Training Dynamics to a Pruning Solution

AI 总结

• Contrastive learning has emerged as a powerful framework for learning generalizable representations, yet its theoretical understanding rema…
• Such an imbalance can degrade representation quality and induce biased model behavior, yet a rigorous characterization of these effects is …
• In this work, we develop a theoretical framework to analyze the training dynamics of contrastive learning with Transformer-based encoders u…
• Our results reveal that neuron weights evolve through three distinct stages of training, with different dynamics for majority features, min…
• We further show that minority features reduce representational capacity, increase the need for more complex architectures, and hinder the s…
• Inspired by these neuron-level behaviors, we show that pruning restores performance degraded by imbalance and enhances feature separation, …

内容摘录

Contrastive learning has emerged as a powerful framework for learning generalizable representations, yet its theoretical understanding remains limited, particularly under imbalanced data distributions that are prevalent in real-world applications. Such an imbalance can degrade representation quality and induce biased model behavior, yet a rigorous characterization of these effects is lacking. In this work, we develop a theoretical framework to analyze the training dynamics of contrastive learning with Transformer-based encoders under imbalanced data. Our results reveal that neuron weights evolve through three distinct stages of training, with different dynamics for majority features, minority features, and noise. We further show that minority features reduce representational capacity, increase the need for more complex architectures, and hinder the separation of ground-truth features from noise. Inspired by these neuron-level behaviors, we show that pruning restores performance degraded by imbalance and enhances feature separation, offering both conceptual insights and practical guidance. Major theoretical findings are validated through numerical experiments.