Multimodal adversarial robustness toolkit. Novel cross-modal attack with transfer analysis. Published preprint.
Multimodal models like CLIP are increasingly used in production, but their adversarial robustness is poorly understood. How do perturbations in one modality affect the other?
Attack Pipeline: Image → FGSM/PGD perturbation → CLIP encoder → embedding space Text → Token manipulation → CLIP encoder → embedding space Transfer Analysis: Cross-modal: image perturbations → text retrieval Cross-model: CLIP attacks → other vision-language models
| Decision | Choice | Why |
|---|---|---|
| Attack methods | FGSM + PGD | Standard gradient-based attacks, extended to multimodal setting. |
| Target model | CLIP | Most deployed multimodal model. High impact research. |
| Tracking | Weights & Biases | Experiment tracking for hundreds of attack configurations. |