Automatic Prompt Engineer (APE)

Automatic Prompt Engineer (APE) is a framework that uses large language models to autonomously generate, evaluate, and select optimal natural-language instructions for downstream tasks. It treats prompt design as a black-box optimization problem over discrete instruction space, eliminating the need for manual prompt engineering.¹⁾

APE leverages LLMs in a two-stage process:

A base LLM (e.g., Codex) acts as an “inference model” to produce multiple candidate instructions by conditioning on few-shot input-output examples for a task. These candidates are treated as natural-language “programs” that guide the target LLM.

Each candidate instruction is executed on held-out validation data using a target LLM (e.g., GPT-3). Performance scores (e.g., accuracy) are computed, and the highest-scoring instruction is selected. This Monte Carlo-style search can iterate, generating new candidates from top performers.

APE formalizes prompt engineering as black-box optimization:²⁾

1. Start with task demonstrations (input-output pairs).
2. Generate diverse candidate prompts via LLM proposals.
3. Score candidates on a validation set using target model performance.
4. Select the best candidate; optionally refine through iterative search.

Advanced variants incorporate evolutionary search or reinforcement learning, but the original APE uses simple LLM-driven Monte Carlo search for efficiency.

OPRO (Optimization by PROmpting) by Yang et al. builds on APE-like ideas but uses LLMs to iteratively generate and score prompts via natural-language critiques:³⁾

• The LLM evaluator provides relative rankings (“better than” comparisons) to guide refinement.
• This reduces compute needs compared to APE's data-heavy evaluation approach.
• OPRO emphasizes critique-driven evolution rather than scoring on validation data.

APE-generated prompts matched or exceeded expert human prompts across multiple benchmarks.⁴⁾

• Big-Bench tasks: APE improved GPT-3 performance by 10-40% over zero-shot baselines, matching human prompts on navigation, sentiment analysis, and other tasks.
• Zero-shot CoT: APE discovered prompts that outperformed the manually crafted “Let's think step by step” trigger.
• Generalization: Selected prompts transferred well to held-out data and different models (e.g., Codex-generated instructions boosted PaLM and GPT-3).
• Later work (PE2) demonstrated further gains through iterative APE variants on counterfactual tasks.

• Validation data required: APE needs representative validation examples to score candidate prompts.
• Computational cost: Generating and evaluating many candidates requires significant LLM inference.
• Quality ceiling: Generated prompts are bounded by the base model's instruction-generation capabilities.
• Edge case sensitivity: May underperform on unusual inputs not represented in validation data.

• Prompt Engineering
• Meta Prompting
• Zero-Shot Prompting
• Few-Shot Prompting
• Active-Prompt
• chain_of_thought_prompting