Chapter 3: Multimodal AI Core Concepts

Exam focus

• Modalities: text, image, audio, video, 3D/spatial data
• Fusion strategies: early, late, intermediate
• Cross-modal attention
• Joint embedding space
• Representation alignment and modality bridging
• Image captioning, VQA, text-to-image, TTS, STT
• Image-text retrieval and multimodal search

Scope Bullet Explanations

• Modalities: Each modality has different structure, noise profile, and annotation cost.
• Fusion strategies: Where and how modality streams are combined in the pipeline.
• Cross-modal attention: Learns alignment between modality tokens/regions/segments.
• Joint embedding: Shared semantic vector space enabling cross-modal retrieval.
• Representation alignment: Reduce modality gap so semantically similar items are close.
• Cross-modal tasks: Practical application patterns likely to appear in scenario questions.

Chapter overview

Chapter 3 is the conceptual core of GENM. The exam frequently tests whether you can reason about multimodal architecture choices, task-specific fusion, and alignment quality implications.

Assumed foundational awareness

Expected background:

• embedding and similarity intuition,
• attention basics,
• retrieval concepts (top-k, relevance, rerank).

Learning objectives

• Differentiate modalities and their engineering constraints.
• Select suitable fusion strategy by task and data profile.
• Explain alignment/embedding mechanics behind cross-modal retrieval.
• Map common multimodal tasks to architecture components.

3.1 Modality characteristics and constraints

Text is discrete and symbolic. Images are spatial tensors. Audio is continuous waveform with time-frequency structure. Video adds temporal continuity. 3D/spatial data adds geometry and scene context.

Data quality and synchronization requirements increase sharply as modalities increase.

3.2 Fusion strategy selection

Early fusion

Combine features early. Useful for strong low-level correlation but can suffer from modality noise propagation.

Late fusion

Process modalities separately, combine decisions later. Improves robustness and modularity but may miss fine-grained cross-modal interactions.

Intermediate fusion

Blend modalities at selected hidden stages. Practical compromise for many applied systems.

3.3 Cross-modal attention and alignment

Cross-modal attention allows text tokens to attend to image/audio/video representations (or vice versa), improving grounded reasoning.

Representation alignment often uses contrastive objectives and paired data to shape shared semantic space.

3.4 Cross-modal task archetypes

• Image captioning: visual -> text generation.
• VQA: joint reasoning over image and question.
• Text-to-image: language-conditioned media generation.
• TTS/STT: text <-> audio transduction.
• Retrieval/search: query in one modality, results in another.

3.5 Multimodal search blueprint

A robust pattern:

1. ingest modality-specific assets,
2. standardize and align metadata,
3. generate embeddings,
4. retrieve candidates,
5. rerank with cross-modal scorer,
6. generate or return grounded result.

Common failure modes

• Ignoring modality alignment during preprocessing.
• Choosing early fusion when modality noise is high.
• Evaluating only one modality-specific metric.
• Skipping metadata/rights tracking in search pipelines.

Chapter summary

Multimodal design depends on choosing the right fusion point and alignment strategy for the task. Most production failures come from data and alignment weaknesses, not only model architecture.

Mini-lab: multimodal retrieval design

1. Define a query scenario (text->image or image->text).
2. Choose embedding and fusion strategy.
3. Add reranking and quality checks.
4. Define latency and relevance metrics.

Deliverable:

• end-to-end retrieval architecture note with risk controls.

Review questions

1. When is late fusion preferable to early fusion?
2. Why is representation alignment critical for cross-modal retrieval?
3. What does cross-modal attention provide that concatenation does not?
4. Why is multimodal data synchronization difficult in practice?
5. How do VQA and captioning differ architecturally?
6. What is one risk of shared embedding spaces without proper negative sampling?
7. How can intermediate fusion improve performance-cost balance?
8. Why should metadata be treated as first-class signal in multimodal search?
9. What is a common failure pattern in text-to-image system evaluation?
10. How do you decide if modality bridging is successful?

Key terms

Early fusion, late fusion, intermediate fusion, joint embedding space, representation alignment, cross-modal attention, multimodal search.

Exam traps

• Assuming all multimodal tasks need the same fusion strategy.
• Ignoring alignment quality while tuning model size.
• Treating retrieval as solved after embedding generation.