LLM-powered agents for video editing enable prompt-driven autonomous editing workflows, transforming natural language instructions into structured edit operations over long-form video content through hierarchical semantic indexing and agentic planning.

Video editing is a time-consuming creative process that requires both technical skill and narrative judgment. LLM agents bridge this gap by converting natural language directives into concrete editing actions. Key approaches include hierarchical semantic indexing for long-form video comprehension, agent-assisted editing with step-by-step planning, and story-driven autonomous editing pipelines that maintain narrative coherence across clips.

The framework introduced in the prompt-driven agentic editing paper uses a modular, cloud-native pipeline for long-form video comprehension and editing:

• Ingestion Module: Processes raw video into analyzable segments
• Hierarchical Semantic Indexing: Builds multi-level semantic representations (scene graphs, narrative arcs, temporal relationships)
• Agentic Planning Engine: LLM decomposes user prompts into editing sub-tasks
• Execution Pipeline: Applies edits (trimming, sequencing, transitions) based on the plan

The semantic index organizes video content at multiple granularities:

<latex>I_{semantic} = \{L_{frame}, L_{scene}, L_{narrative}\}</latex>

where $L_{frame}$ captures per-frame descriptions, $L_{scene}$ groups frames into semantic scenes, and $L_{narrative}$ models story-level arcs.

LAVE (LLM Agent-assisted Video Editing) implements a semi-autonomous workflow where the agent collaborates with the user:

Backend Processing: Video frames are sampled every second, captioned using VLMs (e.g., LLaVA), then processed by GPT-4 to generate titles, summaries, and unique clip IDs, converting visual content to text for LLM processing.

Agent Workflow States:

1. Plan State: LLM decomposes user prompts into actions (footage overview, idea brainstorming, semantic search, storyboarding)
2. Execute State: Agent performs approved actions sequentially, presenting results for user refinement

A user study with 8 participants (novices to experts) demonstrated LAVE produces satisfactory videos rated as easy to use and useful, enhancing creativity and the sense of co-creation.

For story-driven autonomous editing, the agent follows a narrative-aware pipeline:

<latex>E_{story} = \arg\max_{e \in \mathcal{E}} \alpha \cdot S_{coherence}(e) + \beta \cdot S_{pacing}(e) + \gamma \cdot S_{engagement}(e)</latex>

where the editing sequence $e$ is optimized over coherence, pacing, and engagement scores weighted by $\alpha$, $\beta$, $\gamma$.

from dataclasses import dataclass
 
@dataclass
class VideoClip:
    clip_id: str
    start_time: float
    end_time: float
    caption: str
    scene_label: str
    narrative_arc: str
 
class VideoEditAgent:
    def __init__(self, llm, vlm):
        self.llm = llm
        self.vlm = vlm
 
    def build_semantic_index(self, video_path: str,
                             sample_rate: float = 1.0) -> dict:
        frames = self.extract_frames(video_path, sample_rate)
        captions = [self.vlm.caption(f) for f in frames]
        scenes = self.llm.generate(
            f"Group these captions into semantic scenes:\n"
            f"{captions}\nReturn scene boundaries and labels."
        )
        narrative = self.llm.generate(
            f"Identify narrative arcs across scenes:\n{scenes}"
        )
        return {
            "frame_level": captions,
            "scene_level": scenes,
            "narrative_level": narrative
        }
 
    def plan_edit(self, user_prompt: str,
                  semantic_index: dict) -> list[dict]:
        plan = self.llm.generate(
            f"User wants: {user_prompt}\n"
            f"Available footage:\n{semantic_index}\n"
            f"Create a step-by-step editing plan with "
            f"clip selections and transitions."
        )
        return self.parse_edit_plan(plan)
 
    def execute_plan(self, plan: list[dict],
                     clips: list[VideoClip]) -> str:
        timeline = []
        for step in plan:
            selected = self.semantic_search(
                step["query"], clips
            )
            trimmed = self.trim_clip(selected, step)
            timeline.append(trimmed)
        return self.render_timeline(timeline)

• "Prompt-Driven Agentic Video Editing with Hierarchical Semantic Indexing" (2025)
• Wang et al. "LAVE: LLM-Powered Agent-Assisted Video Editing" (2024)

• Image Editing Agents
• Music Composition Agents
• Game Playing Agents