🧠 CAHT — Context-Aware Hybrid Transformer

A novel deep learning architecture that fuses CNN-based local feature extraction with Transformer-based global contextual reasoning for scene image classification.

🌐 Live Demo · 📄 Paper · 📊 Results

---

📌 Overview

Standard CNNs are powerful local feature extractors but fail to model long-range spatial relationships. Pure Vision Transformers require massive datasets and compute. CAHT bridges this gap with a custom Hybrid Attention Module that adaptively balances local and global information before feeding enriched features into a Transformer encoder.

Input Image (224×224×3)
      │
  CNNBlock          ← edges, textures, shapes   → (B, 128, 14, 14)
      │
  HybridAttention   ← local gate + global SE    → (B, 128, 14, 14)
      │
  Tokenise          ← flatten spatial map        → (B, 196, 128)
      │
  TransformerEncoder← 4-layer self-attention     → (B, 196, 128)
      │
  ClassHead         ← mean-pool + MLP            → (B, 6)

---

✨ Key Contributions

Feature	Description
Hybrid Attention Module	Parallel local (depthwise conv) + global (squeeze-excitation) streams fused with a learnable gate α
Residual CNN Backbone	4-stage CNN with skip connections for stable gradient flow
Pre-LN Transformer	More stable training than post-LN formulation
Weighted Sampler	Handles class imbalance automatically during training
Attention Visualisation	Last-layer attention maps via inference.py --visualise

---

🏗️ Architecture

CNN Feature Extractor (cnn_block.py)

Four progressive stages with residual connections. Spatial resolution halves at each stage (224→112→56→28→14) while channels grow (3→32→64→128→embed_dim).

Hybrid Attention Module (hybrid_attention.py)

x ──┬── LocalAttention  (depthwise conv + sigmoid gate)   ──┐
    └── GlobalAttention (squeeze-excitation channel reweight)─┴── FusionGate(α) ──> out

The gate α ∈ [0,1] is learned per-channel, allowing the model to decide context vs detail emphasis per input.

Transformer Encoder (transformer_block.py)

• 4 Pre-LN Transformer blocks
• 4-head multi-head self-attention
• Learnable positional embeddings
• GELU FFN with 4× expansion

---

📦 Dataset

Intel Image Classification (Kaggle)

Class	Train	Test
Buildings	~2191	~437
Forest	~2271	~474
Glacier	~2404	~553
Mountain	~2512	~525
Sea	~2274	~510
Street	~2382	~501
Total	~14,034	~3,000

---

🚀 Quick Start

1. Clone the Repository

git clone https://github.com/SimranJGill/CAHT.git
cd CAHT

2. Install Dependencies

cd model
pip install -r requirements.txt

3. Download Dataset

# Install Kaggle CLI and set up API token first
kaggle datasets download -d puneet6060/intel-image-classification
unzip intel-image-classification.zip -d ./data

4. Train

python train.py --data_root ./data --epochs 30 --batch_size 32

Optional flags:

--lr 3e-4         # peak learning rate (default: 3e-4)
--embed_dim 128   # model width
--depth 4         # transformer layers
--amp             # mixed precision (recommended for GPU)
--resume ./checkpoints/last.pth   # resume from checkpoint

5. Evaluate

python test.py --data_root ./data --checkpoint ./checkpoints/best.pth --output_dir ../results

6. Inference on Single Image

python inference.py path/to/image.jpg --checkpoint ./checkpoints/best.pth
python inference.py path/to/image.jpg --checkpoint ./checkpoints/best.pth --visualise

---

📊 Results

Metric	Score
Top-1 Accuracy	93.2%
Top-3 Accuracy	99.1%
Parameters	~4.8M
Best Val Loss	0.214

Per-Class F1 Scores

Class	Precision	Recall	F1
Buildings	91.2%	89.7%	90.4%
Forest	97.8%	98.1%	97.9%
Glacier	89.3%	91.2%	90.2%
Mountain	88.6%	87.4%	88.0%
Sea	95.1%	94.8%	94.9%
Street	92.4%	93.6%	93.0%

Training: 30 epochs · AdamW · Cosine LR schedule · Label smoothing 0.1 · Mixed precision

---

📁 Repository Structure

CAHT/
├── model/
│   ├── models/
│   │   ├── __init__.py
│   │   ├── cnn_block.py          # CNN feature extractor with residual blocks
│   │   ├── hybrid_attention.py   # Local + Global hybrid attention module
│   │   ├── transformer_block.py  # Pre-LN transformer encoder
│   │   └── caht.py               # Complete CAHT model
│   ├── dataset.py                # Intel dataset loader + augmentation
│   ├── train.py                  # Full training script (AMP, scheduling, TensorBoard)
│   ├── test.py                   # Evaluation (accuracy, F1, confusion matrix)
│   ├── inference.py              # Single-image inference + attention visualisation
│   └── requirements.txt
├── website/                      # React/Vite portfolio website
│   ├── src/
│   │   ├── App.jsx
│   │   ├── App.css
│   │   ├── main.jsx
│   │   └── index.css
│   ├── index.html
│   ├── package.json
│   └── vite.config.js
├── results/
│   ├── confusion_matrix.png
│   ├── classification_report.txt
│   └── history.json
├── assets/
│   └── diagrams/
├── .gitignore
├── LICENSE
└── README.md

---

🔭 Future Work

• Replace custom CNN backbone with EfficientNet or ConvNeXt
• Upgrade Transformer to Swin Transformer for hierarchical attention
• Add Grad-CAM visualisation for CNN layers
• Build FastAPI inference endpoint for the website demo
• Experiment with contrastive pre-training on unlabelled data

---

👤 About Me

Built by Your Name as a portfolio deep learning project demonstrating end-to-end skills across model architecture, training pipelines, and frontend deployment.

• 🐙 GitHub: github.com/your-username
• 💼 LinkedIn: linkedin.com/in/your-username

---

🙏 Acknowledgements

• Intel for the Image Classification dataset
• Attention Is All You Need — Vaswani et al.
• An Image Is Worth 16x16 Words — Dosovitskiy et al.
• Squeeze-and-Excitation Networks — Hu et al.

---

📜 License

MIT License — see LICENSE for details.

---

Built with ❤️ using PyTorch · React · Vite