Multimodal Chain-of-Thought Prompting

Multimodal Chain-of-Thought (Multimodal-CoT) extends traditional chain-of-thought reasoning beyond text to incorporate multiple modalities, primarily text and images. The approach uses a two-stage framework that separates rationale generation from answer inference, allowing the model to leverage visual information during reasoning.¹⁾

Existing chain-of-thought studies have focused primarily on the language modality.²⁾ However, many real-world reasoning tasks involve visual information – diagrams, charts, photographs, and scientific figures. A textbook with no figures or tables severely limits knowledge acquisition. Multimodal-CoT addresses this gap by jointly modeling text and vision modalities during the reasoning process.

Multimodal-CoT operates through two distinct stages:

The model receives both language inputs (question, context, options) and vision inputs (associated images) to generate intermediate reasoning steps called rationales. These rationales combine information from both modalities, articulating visual insights in language form.

For example, given a physics question about magnets with an accompanying diagram, the rationale might state: “The north pole of one magnet is closest to the south pole of the other magnet. Poles that are different attract.”

The original language input is appended with the rationale generated in Stage 1. This augmented input, along with the original vision input, is used to infer the final answer. The rationale serves as a bridge between visual grounding and explicit textual reasoning.

Multimodal-CoT achieved state-of-the-art performance on the ScienceQA benchmark with a model under 1 billion parameters:³⁾

The method was also evaluated on the A-OKVQA benchmark, further demonstrating its effectiveness across multimodal reasoning tasks.

• Hallucination mitigation: Visual grounding helps prevent the model from generating reasoning steps that contradict the image.
• Enhanced convergence speed: The two-stage approach converges faster than single-stage multimodal reasoning.
• Parameter efficiency: Achieves state-of-the-art with sub-1B parameter models, making it practical for deployment.
• Modular design: The two stages can be independently optimized.

• Requires paired data: Training needs datasets with aligned text and image inputs.
• Two-stage complexity: The sequential pipeline adds inference latency compared to single-pass methods.
• Vision encoder dependency: Performance is bounded by the quality of the vision encoder used.
• Limited modality scope: Currently focused on text and images; does not extend to audio, video, or other modalities.

• Prompt Engineering
• chain_of_thought_prompting
• Zero-Shot Prompting
• Few-Shot Prompting
• Program-Aided Language Models (PAL)