====== Video Editing Agents ======

LLM-powered agents for video editing enable prompt-driven autonomous editing workflows, transforming natural language instructions into structured edit operations over long-form video content through hierarchical semantic indexing and agentic planning.(([[https://arxiv.org/abs/2509.16811|"Prompt-Driven Agentic Video Editing with Hierarchical Semantic Indexing" (2025)]]))

===== Overview =====

Video editing is a time-consuming creative process that requires both technical skill and narrative judgment. LLM agents bridge this gap by converting natural language directives into concrete editing actions. Key approaches include hierarchical semantic indexing for long-form video comprehension, agent-assisted editing with step-by-step planning, and story-driven autonomous editing pipelines that maintain narrative coherence across clips.

===== Prompt-Driven Agentic Video Editing =====

The framework introduced in the prompt-driven agentic editing paper uses a modular, cloud-native pipeline for long-form video comprehension and editing:(([[https://arxiv.org/abs/2509.16811|"Prompt-Driven Agentic Video Editing with Hierarchical Semantic Indexing" (2025)]]))

  * **Ingestion Module**: Processes raw video into analyzable segments
  * **Hierarchical Semantic Indexing**: Builds multi-level semantic representations (scene graphs, narrative arcs, temporal relationships)
  * **Agentic Planning Engine**: LLM decomposes user prompts into editing sub-tasks
  * **Execution Pipeline**: Applies edits (trimming, sequencing, transitions) based on the plan

The semantic index organizes video content at multiple granularities:

<latex>I_{semantic} = \{L_{frame}, L_{scene}, L_{narrative}\}</latex>

where $L_{frame}$ captures per-frame descriptions, $L_{scene}$ groups frames into semantic scenes, and $L_{narrative}$ models story-level arcs.

===== LAVE: Agent-Assisted Video Editing =====

LAVE (LLM Agent-assisted Video Editing) implements a semi-autonomous workflow where the agent collaborates with the user:(([[https://arxiv.org/abs/2402.10294|Wang et al. "LAVE: LLM-Powered Agent-Assisted Video Editing" (2024)]]))

**Backend Processing**: Video frames are sampled every second, captioned using VLMs (e.g., LLaVA), then processed by GPT-4 to generate titles, summaries, and unique clip IDs, converting visual content to text for LLM processing.

**Agent Workflow States**:
  - **Plan State**: LLM decomposes user prompts into actions (footage overview, idea brainstorming, semantic search, storyboarding)
  - **Execute State**: Agent performs approved actions sequentially, presenting results for user refinement

A user study with 8 participants (novices to experts) demonstrated LAVE produces satisfactory videos rated as easy to use and useful, enhancing creativity and the sense of co-creation.(([[https://arxiv.org/abs/2402.10294|Wang et al. "LAVE: LLM-Powered Agent-Assisted Video Editing" (2024)]]))

===== Story-Driven Editing =====

For story-driven autonomous editing, the agent follows a narrative-aware pipeline:

<latex>E_{story} = \arg\max_{e \in \mathcal{E}} \alpha \cdot S_{coherence}(e) + \beta \cdot S_{pacing}(e) + \gamma \cdot S_{engagement}(e)</latex>

where the editing sequence $e$ is optimized over coherence, pacing, and engagement scores weighted by $\alpha$, $\beta$, $\gamma$.

===== Code Example =====

<code python>
from dataclasses import dataclass

@dataclass
class VideoClip:
    clip_id: str
    start_time: float
    end_time: float
    caption: str
    scene_label: str
    narrative_arc: str

class VideoEditAgent:
    def __init__(self, llm, vlm):
        self.llm = llm
        self.vlm = vlm

    def build_semantic_index(self, video_path: str,
                             sample_rate: float = 1.0) -> dict:
        frames = self.extract_frames(video_path, sample_rate)
        captions = [self.vlm.caption(f) for f in frames]
        scenes = self.llm.generate(
            f"Group these captions into semantic scenes:\n"
            f"{captions}\nReturn scene boundaries and labels."
        )
        narrative = self.llm.generate(
            f"Identify narrative arcs across scenes:\n{scenes}"
        )
        return {
            "frame_level": captions,
            "scene_level": scenes,
            "narrative_level": narrative
        }

    def plan_edit(self, user_prompt: str,
                  semantic_index: dict) -> list[dict]:
        plan = self.llm.generate(
            f"User wants: {user_prompt}\n"
            f"Available footage:\n{semantic_index}\n"
            f"Create a step-by-step editing plan with "
            f"clip selections and transitions."
        )
        return self.parse_edit_plan(plan)

    def execute_plan(self, plan: list[dict],
                     clips: list[VideoClip]) -> str:
        timeline = []
        for step in plan:
            selected = self.semantic_search(
                step["query"], clips
            )
            trimmed = self.trim_clip(selected, step)
            timeline.append(trimmed)
        return self.render_timeline(timeline)
</code>

===== Architecture =====

<mermaid>
graph TD
    A[Raw Video Input] --> B[Frame Extraction]
    B --> C[VLM Captioning - LLaVA]
    C --> D[Semantic Scene Grouping]
    D --> E[Narrative Arc Analysis]
    E --> F[Hierarchical Semantic Index]
    G[User Prompt] --> H[LLM Planning Agent]
    F --> H
    H --> I[Edit Plan]
    I --> J{User Approval?}
    J -->|Yes| K[Execution Agent]
    J -->|No| L[Plan Refinement]
    L --> H
    K --> M[Semantic Clip Search]
    K --> N[Clip Trimming]
    K --> O[Transition Selection]
    M --> P[Timeline Assembly]
    N --> P
    O --> P
    P --> Q[Rendered Video]
    Q --> R[Quality Review Agent]
    R -->|Revisions Needed| H
</mermaid>

===== Key Comparisons =====

^ System ^ Autonomy ^ Story Support ^ User Study ^
| Prompt-Driven Agentic | Fully autonomous | Narrative sequencing | Pipeline evaluation |
| LAVE | Semi-autonomous (user approves) | Brainstorming + storyboarding | 8 participants, positive |
| VideoAgent | Agentic framework | Understanding + editing | General performance |

===== See Also =====

  * [[image_editing_agents|Image Editing Agents]]
  * [[music_composition_agents|Music Composition Agents]]
  * [[game_playing_agents|Game Playing Agents]]

===== References =====