ADAPT-MAGNET: A Multimodal Multitask Architecture for Gender and Emotion Recognition

ADAPT-MAGNET (efficiently ADApted Audio-visual Parameter-efficient Tuning for MAGNET) is a multi-modal, multi-task architecture for joint emotion and gender recognition from audio and video. It evolves MAGNET — a prior state-of-the-art architecture based on lightweight, task-specific encoders trained from scratch — by replacing those encoders with large-scale pre-trained audio (Whisper Large v3) and visual (CLIP ViT-L/14-336) foundation models, the same backbones used in the NVIDIA AVLM model. These backbones are adapted to the downstream tasks with Parameter-Efficient Fine-Tuning (PEFT) — primarily LoRA, alongside a from-scratch implementation of Visual/Audio Prompt Tuning — instead of full fine-tuning, keeping the adaptation lightweight while leveraging web-scale pre-trained representations. Models are trained and evaluated on CREMA-D, where ADAPT-MAGNET surpasses the MAGNET baseline, reaching up to 91.3% accuracy on emotion recognition (+4.4% over MAGNET) and 100% accuracy on gender recognition.

Raw audio and video clips are independently pre-processed and passed to the modality-specific audio encoder $f_a(\cdot)$ and visual encoder $f_v(\cdot)$, both adapted via LoRA. The resulting embeddings are fused into a unified audio-visual representation, which feeds two task-specific branches — the Gender Head and the Emotion Head — producing the final predictions.

Table of contents

• Repository structure
• 1. Environment setup
• 2. Download the dataset
• 3. Prepare the data
• 4. Configuration files
• 5. Train
• 6. Test
• 7. Visualize results (TensorBoard)
• 8. Demo (optional)
• Notes and known limitations
• References

Repository structure

• conf/: YAML configuration files. Top-level files (e.g. multimodal_MT_freeze.yml) are passed to train.py/test.py via --conf; each one points to a "sub configuration" file under conf/emotion/, conf/gender/ or conf/multitask/ that holds the actual model/preprocessing/training hyperparameters.
• datasets/: CremaDataset, the torch.utils.data.Dataset used to load CREMA-D (audio + video + labels). The raw data itself is not versioned (see .gitignore) and must be downloaded (step 2).
• models/: audio/video encoders (audio_models.py, video_models.py), the single-task and multi-task heads (single_tasks.py, multitask.py), and the S3FD face detector (FaceDetector/) used for face cropping.
• utils/: config loading (yaml_loader.py), misc helpers (misc.py), audio/video pre-processing (preprocess.py), and dataset preparation utilities (dataset_analysis.py, including the train/val/test split).
• demo/: sample audio/video clips and a small pipeline (demo_model.py, demo_run.py, demo_show.py) to run inference on them with a trained checkpoint.
• envs/: exported Conda environments (Windows, CUDA 11.8) used on the original development machines, kept for reference/reproducibility.
• train.py / test.py: entry points for training and testing.

Usage guide

The full step-by-step guide has been split into one document per step under docs/:

1. Environment setup — Conda environment and dependencies.
2. Download the dataset — CREMA-D (required) and MIT Indoor (optional).
3. Prepare the data — remove corrupted clips, generate the train/val/test split, extract face bounding boxes.
4. Configuration files — the conf/ naming convention and main/sub configuration structure.
5. Train — python train.py --conf <config>.yml.
6. Test — python test.py --conf <config>.yml.
7. Visualize results (TensorBoard) — tensorboard --logdir logs.
8. Demo (optional) — run inference on sample clips with a trained checkpoint.

Notes and known limitations

• The train/val/test split described in step 3 is an actor-level split with configurable ratios and a fixed seed for reproducibility; it does not attempt to balance emotion/gender distribution across splits.
• k-fold cross-validation is supported by train.py/test.py (--kfold) but the fold CSVs must be created manually (see note in step 3).
• Green-screen background replacement (MIT Indoor) is implemented but disabled in every shipped configuration.

References

1. Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., & Verma, R. (2014). Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4), 377-390.
2. Quattoni, A., Torralba, A. MIT Indoor Scenes. CVPR, 2009.
3. De Simone, G., Greco, L., Saggese, A., & Vento, M. (2025). Integrating Visual and Audio Cues for Emotion and Gender Recognition: a Multi modal and Multi task approach. Information Fusion, 104071.