Natural language understanding (NLU) and natural language generation (NLG) form the core linguistic capabilities that enable AI agents to interpret user intent and produce coherent, contextually appropriate responses. In modern LLM-based agents, these capabilities are unified within transformer architectures, though specialized techniques remain critical for high-accuracy domain-specific applications.

Intent recognition has evolved from classifier-based pipelines (Rasa NLU, Dialogflow) to end-to-end LLM approaches that jointly parse intent, extract entities, and generate responses.

Instruction Following is a defining capability of modern agents:

• InstructGPT demonstrated that RLHF fine-tuning dramatically improves instruction adherence¹⁾
• Claude, GPT-4, and Gemini achieve high-fidelity instruction following through constitutional AI training, RLHF, and direct preference optimization (DPO)
• IFEval is a benchmark specifically measuring verifiable instruction-following constraints (e.g., “respond in exactly 3 sentences”)²⁾
• Modern agents handle multi-step instructions, conditional logic, and format constraints with near-human accuracy

Intent Recognition in 2025 achieves 95-98% accuracy in production systems through:

• Few-shot classification with frontier LLMs
• Retrieval-Augmented Generation (RAG) for domain-specific intent disambiguation
• Context-aware systems that maintain intent across multi-turn conversations

Semantic parsing translates natural language into formal representations (SQL, API calls, logical forms). Key advances include:

• Text-to-SQL: Systems like DIN-SQL and DAIL-SQL leverage LLMs to achieve >85% execution accuracy on Spider benchmark
• Code Generation: Models serve as semantic parsers that translate intent into executable programs
• Tool Selection: Agents parse user requests into structured tool calls with parameters, a form of semantic parsing central to tool utilization

PaLM demonstrated breakthrough performance on BIG-Bench across 150+ tasks spanning semantic understanding, with subsequent models building on this foundation.³⁾

Grounding connects language to real-world referents and actions:

• Embodied Grounding: Systems like SayCan and PaLM-E ground language in physical affordances, enabling robots to understand “pick up the sponge” relative to their environment^{4) 5)}
• Visual Grounding: Models like GPT-4V, Gemini, and Claude 3.5 ground textual descriptions in visual inputs, enabling multimodal reasoning
• Situational Grounding: Agents ground instructions in their current execution context (file system state, browser content, conversation history)

Challenges persist in grounding language to physical causality, cultural context, and implicit world knowledge that humans take for granted.

Modern LLMs increasingly integrate multiple modalities:

• Vision-Language Models: GPT-4V, Gemini 1.5/2.5, and Claude 3.5 process images and text jointly, enabling tasks like chart analysis, UI understanding, and visual question answering
• Audio Understanding: Gemini 2.5 and GPT-4o process speech directly, enabling real-time multilingual translation and conversational AI
• Document Understanding: Models parse PDFs, screenshots, and handwritten text, combining OCR-level perception with semantic comprehension
• SeamlessM4T: Multimodal translation system supporting speech-to-speech, speech-to-text, and text-to-speech across 100+ languages⁶⁾

NLG in agents goes beyond simple text completion:

• Structured Generation: Producing JSON, code, or formatted outputs (see Structured Outputs)
• Retrieval-Augmented Generation (RAG): Grounding responses in retrieved documents to reduce hallucination
• Multi-turn Coherence: Maintaining topic, style, and factual consistency across long conversations
• Constrained Generation: Following format, length, tone, and content constraints specified by the user or system

Key benchmarks for evaluating NLU capabilities:

• MMLU: 57-subject knowledge evaluation; GPT-4 achieves ~87%, Gemini Ultra ~90%⁷⁾
• SuperGLUE: Sentence understanding tasks; effectively saturated by modern LLMs⁸⁾
• BIG-Bench: 200+ tasks testing diverse language capabilities⁹⁾
• HumanEval / MBPP: Code generation as a proxy for semantic understanding
• HELM: Holistic evaluation spanning accuracy, calibration, robustness, fairness, and efficiency¹⁰⁾

• Conversational Agents
• AI Code Generation
• AutoGen
• Natural Language Programming
• AI Agents for SQL and Database Interaction