🔗 Prinsip Contrastive Learning

Contrastive learning melatih model untuk membedakan:

• ✅ Positive pairs: (image, matching caption) - harus dekat
• ❌ Negative pairs: (image, non-matching caption) - harus jauh

📐 InfoNCE Loss

CLIP menggunakan InfoNCE (Noise Contrastive Estimation) loss:

🎨 Example: Batch dengan N=4 pairs

🎬 Contrastive Pairing Animation

🏗️ CLIP Architecture

CLIP terdiri dari dua encoder terpisah yang dilatih secara joint:

⚙️ Key Components

• 🖼️ Vision Transformer (ViT): Patch-based image processing
• 📝 Text Transformer: Masked self-attention untuk text
• 🎯 Projection heads: Map ke shared embedding space
• 📏 L2 normalization: Ensure embeddings pada unit sphere

🎬 Encoder Architecture Animation

💡 Embedding Space

📊 Training Batch

Training CLIP dilakukan dengan batch besar (e.g., N=32,768):

1. Batch of N (image, text) pairs dari dataset
2. Encode images → N image embeddings (I₁, I₂, ..., I_N)
3. Encode texts → N text embeddings (T₁, T₂, ..., T_N)
4. Compute N×N similarity matrix
5. Calculate symmetric loss

🔥 Similarity Matrix

Untuk batch N=4, similarity matrix S menunjukkan cosine similarity antara semua pairs:

📐 Symmetric Loss

🎬 Training Batch Animation

⚡ Training Details

• 📊 Dataset: 400M (image, text) pairs dari internet
• 🎯 Batch size: 32,768 (very large!)
• ⏱️ Training time: ~12 days pada 592 V100 GPUs
• 🔧 Optimizer: AdamW dengan cosine learning rate schedule

🎯 Apa itu Zero-Shot?

Zero-shot classification: Model bisa classify ke class yang belum pernah dilihat saat training!

📊 Example: Image Classification

Task: Classify image ke 3 classes

🎬 Zero-Shot Demo Animation

💡 Prompt Engineering

Prompt design sangat mempengaruhi akurasi! Tips:

• 🎯 Template: "a photo of a {class}" works well
• 📝 Ensemble: Use multiple prompts per class
• 🔍 Context: Add context, e.g. "a photo of a {class}, a type of pet"
• 🌐 Domain: Adjust untuk domain-specific (medical, satellite)

🚀 CLIP Applications

CLIP telah menjadi foundation model untuk berbagai aplikasi multimodal:

🌟 Notable Projects Using CLIP

• 🎨 DALL-E 2: Text-to-image generation dengan CLIP-guided diffusion
• 🖼️ Stable Diffusion: Open-source generation model using CLIP text encoder
• 🔍 OpenCLIP: Open reproduction of CLIP dengan larger datasets
• 🎬 Video understanding: Extend CLIP to video domain (CLIP4Clip)
• 🏥 Medical imaging: Zero-shot diagnosis dengan domain-specific prompts

💡 Why CLIP is Powerful

💻 CLIP Model Implementation

import torch
import torch.nn as nn
import torch.nn.functional as F

class CLIP(nn.Module):
    def __init__(self, image_encoder, text_encoder, embed_dim=512):
        super().__init__()
        self.image_encoder = image_encoder  # ViT or ResNet
        self.text_encoder = text_encoder    # Transformer
        
        # Projection heads
        self.image_proj = nn.Linear(image_encoder.output_dim, embed_dim)
        self.text_proj = nn.Linear(text_encoder.output_dim, embed_dim)
        
        # Learnable temperature
        self.logit_scale = nn.Parameter(torch.ones([]) * np.log(1/0.07))
    
    def encode_image(self, images):
        # images: (batch, 3, 224, 224)
        image_features = self.image_encoder(images)
        image_embeds = self.image_proj(image_features)
        image_embeds = F.normalize(image_embeds, dim=-1)
        return image_embeds
    
    def encode_text(self, text):
        # text: (batch, max_length) token ids
        text_features = self.text_encoder(text)
        text_embeds = self.text_proj(text_features)
        text_embeds = F.normalize(text_embeds, dim=-1)
        return text_embeds
    
    def forward(self, images, texts):
        image_embeds = self.encode_image(images)  # (N, embed_dim)
        text_embeds = self.encode_text(texts)      # (N, embed_dim)
        
        # Scaled cosine similarity
        logit_scale = self.logit_scale.exp()
        logits_per_image = logit_scale * image_embeds @ text_embeds.T  # (N, N)
        logits_per_text = logits_per_image.T
        
        return logits_per_image, logits_per_text

🎓 Training Loop

def train_clip(model, dataloader, optimizer, device):
    model.train()
    
    for images, texts in dataloader:
        images = images.to(device)
        texts = texts.to(device)
        
        # Forward pass
        logits_per_image, logits_per_text = model(images, texts)
        
        # Ground truth: diagonal matrix (positive pairs)
        batch_size = images.shape[0]
        labels = torch.arange(batch_size, device=device)
        
        # Symmetric loss
        loss_img = F.cross_entropy(logits_per_image, labels)
        loss_txt = F.cross_entropy(logits_per_text, labels)
        loss = (loss_img + loss_txt) / 2
        
        # Backward
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        return loss.item()

🎯 Zero-Shot Inference

def zero_shot_classify(model, image, class_names, device):
    """
    Classify image to one of class_names without training.
    """
    model.eval()
    
    # Prepare image
    image = preprocess(image).unsqueeze(0).to(device)
    
    # Generate text prompts
    prompts = [f"a photo of a {name}" for name in class_names]
    text_tokens = tokenize(prompts).to(device)
    
    with torch.no_grad():
        # Encode
        image_embed = model.encode_image(image)      # (1, 512)
        text_embeds = model.encode_text(text_tokens)  # (num_classes, 512)
        
        # Compute similarities
        logit_scale = model.logit_scale.exp()
        similarities = logit_scale * image_embed @ text_embeds.T  # (1, num_classes)
        
        # Softmax to get probabilities
        probs = F.softmax(similarities, dim=-1).squeeze(0)
        
        # Prediction
        pred_idx = probs.argmax().item()
        pred_class = class_names[pred_idx]
        pred_conf = probs[pred_idx].item()
        
    return pred_class, pred_conf, probs

# Example usage
class_names = ["dog", "cat", "car", "airplane"]
pred, conf, all_probs = zero_shot_classify(model, image, class_names, device)
print(f"Prediction: {pred} ({conf*100:.1f}% confidence)")

🔧 Using Pre-trained CLIP

import clip

# Load pre-trained CLIP
device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)

# Load and preprocess image
from PIL import Image
image = preprocess(Image.open("dog.jpg")).unsqueeze(0).to(device)

# Prepare text
text = clip.tokenize(["a dog", "a cat", "a car"]).to(device)

# Get predictions
with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    
    logits_per_image, logits_per_text = model(image, text)
    probs = logits_per_image.softmax(dim=-1).cpu().numpy()

print("Label probs:", probs)  # [[0.92, 0.06, 0.02]]

🚀 CLIP Variants

• 🌐 OpenCLIP: Open-source reproduction dengan larger datasets (LAION-2B)
• 📊 MetaCLIP: Curated training data for better quality
• 🎬 CLIP4Clip: Extend to video understanding
• 🔊 AudioCLIP: Add audio modality
• 🏥 MedCLIP: Domain-specific for medical imaging

🔧 Fine-tuning Strategies

Fine-tuning approaches:

• 🎯 Full fine-tuning: Update all parameters
• ⚡ Adapter layers: Add small trainable modules
• 🔒 Prompt tuning: Learn continuous prompts
• 📊 Linear probe: Only train classifier head

💡 Tips & Best Practices

• 📝 Prompt engineering: Experiment with different templates
• 🎯 Ensemble: Average predictions across multiple prompts
• 🔍 Image preprocessing: Follow CLIP's normalization
• ⚖️ Scaling: Larger models (ViT-L/14) perform better but slower
• 🌐 Multilingual: Use M-CLIP for non-English text

🔮 Future Directions

• 🎨 Generative models: Better integration dengan diffusion models
• 🎬 Video understanding: Temporal consistency
• 🧠 3D vision: Extend to 3D scenes
• 🌍 Multilingual & multicultural: Better global coverage
• ⚡ Efficiency: Smaller models for edge deployment

✅ Selamat!