LLM agents are reshaping personalized recommendation by replacing static retrieval heuristics with dynamic, reasoning-driven systems that browse, understand user intent, and adaptively rank items. AgentRecBench (NeurIPS 2025) provides the first comprehensive benchmark, while ARAG (Walmart, 2025) demonstrates production-scale agentic recommendation via multi-agent RAG.

AgentRecBench introduces an interactive textual simulator and modular agent framework for evaluating LLM-powered recommender systems. Accepted at NeurIPS 2025, it is the first benchmark designed specifically for agent-based recommendation.

Interactive Simulator: A textual environment with rich user/item metadata (profiles, reviews, interaction histories) that supports autonomous information retrieval. Agents navigate user-item networks to gather evidence before making recommendations.

Three Evaluation Scenarios:

• Classic: Standard recommendation performance on general tasks
• Evolving-Interest: Tests agent adaptation to changing user preferences over time
• Cold-Start: Evaluates generalization with limited interaction data

Modular Agent Framework with four core components:

Benchmarks over 10 methods including classical (handcrafted features), ranking-oriented agents (RecMind), simulation-oriented agents (Agent4Rec), and conversational approaches. Metrics focus on Top-N ranking accuracy with 20 candidates per test instance.

ARAG (Walmart Global Tech, 2025) integrates a multi-agent collaboration mechanism into the RAG pipeline for personalized recommendation. It addresses the failure of static RAG heuristics to capture nuanced user preferences in dynamic scenarios.

Four Specialized Agents:

• User Understanding Agent: Summarizes user preferences from both long-term behavioral history and current session context
• NLI (Natural Language Inference) Agent: Evaluates semantic alignment between candidate items retrieved by RAG and the inferred user intent
• Context Summary Agent: Consolidates and summarizes findings from the NLI agent into coherent preference signals
• Item Ranker Agent: Generates final ranked recommendations based on contextual fit across all agent outputs

The recommendation score for an item $i$ given user context $u$ is computed through the agent pipeline:

$$\text{score}(i | u) = f_{\text{rank}}\left( f_{\text{summary}}\left( f_{\text{NLI}}(i, f_{\text{user}}(u)) \right) \right)$$

where each $f$ represents a specialized agent's transformation of the information.

class ARAGPipeline:
    def __init__(self, retriever, user_agent, nli_agent, 
                 summary_agent, ranker_agent):
        self.retriever = retriever
        self.user_agent = user_agent
        self.nli_agent = nli_agent
        self.summary_agent = summary_agent
        self.ranker_agent = ranker_agent
 
    def recommend(self, user_id, session_context, k=5):
        user_profile = self.user_agent.summarize_preferences(
            long_term=self.retriever.get_user_history(user_id),
            session=session_context
        )
        candidates = self.retriever.retrieve_candidates(user_profile)
        nli_scores = []
        for item in candidates:
            alignment = self.nli_agent.evaluate_alignment(
                item_description=item.metadata,
                user_intent=user_profile.intent
            )
            nli_scores.append((item, alignment))
        context_summary = self.summary_agent.summarize(nli_scores)
        ranked_items = self.ranker_agent.rank(
            candidates=nli_scores,
            context=context_summary,
            top_k=k
        )
        return ranked_items

ARAG achieves significant improvements over standard RAG and recency-based baselines across three datasets:

Ablation studies confirm that each agent contributes meaningfully, with the NLI agent and User Understanding agent showing the largest individual contributions.

• Agentic > Static: AgentRecBench demonstrates that agent-based systems consistently outperform traditional recommendation approaches, especially in cold-start and evolving-interest scenarios
• Multi-Agent Decomposition: ARAG shows that decomposing recommendation into specialized sub-tasks (user understanding, semantic alignment, ranking) yields large gains over monolithic approaches
• Session Awareness: Both systems emphasize the importance of combining long-term user history with real-time session signals
• Reasoning Over Retrieval: The shift from “retrieve and rank” to “retrieve, reason, and rank” represents a fundamental change in recommendation architecture

• AgentRecBench: A Comprehensive Benchmark for LLM Agent-Based Recommender Systems (arXiv:2505.19623)
• ARAG: Agentic Retrieval Augmented Generation for Personalized Recommendation (arXiv:2506.21931)
• AgentRecBench Dataset on Hugging Face

• API Tool Generation: Doc2Agent and LRASGen
• Knowledge Graph World Models: AriGraph
• Story Generation Agents: StoryWriter