Automated Prompt Optimisation
Automated prompt optimisation uses algorithms (evolutionary, feedback-driven, gradient-based, or evaluation-loops) to systematically refine prompts for LLMs, reducing manual engineering and improving task accuracy/performance.
https://www.awesomepython.org/?q=auto-prompting
agent-lightning (Microsoft)
Repo: github.com/microsoft/agent-lightning
Description: Framework to “train” agents by optimizing prompts (APO) and other components with minimal/near-zero code changes, and works across many agent stacks. (GitHub)
FAQ
- What’s the core idea? Iterate: run your agent → score outputs → generate improved prompts/components → repeat. (microsoft.github.io)
- Does it fine-tune model weights? It supports multiple training-style approaches, including prompt optimization and (optionally) other methods like RL/SFT depending on setup. (GitHub)
- Do I need a labeled dataset? Not strictly—any repeatable evaluation/scoring function can drive optimization. (microsoft.github.io)
dspy (StanfordNLP)
Repo: github.com/stanfordnlp/dspy
Description: A declarative framework for building modular LM “programs” that can be compiled/optimized into better prompts (and sometimes learned parameters) for tasks like classification, RAG, and agents. (GitHub)
FAQ
- How is it different from “prompting”? You write structured components/signatures; DSPy handles prompt construction and optimization loops. (GitHub)
- What do I optimize against? Whatever metric you can compute from outputs (accuracy/F1, retrieval metrics, task-specific checks, etc.). (GitHub)
- API LLMs or self-hosted? Both—DSPy is designed to sit above whatever LM backend you configure. (DSPy)
gepa (gepa-ai)
Repo: github.com/gepa-ai/gepa
Description: “Genetic-Pareto” optimization for systems made of text components (prompts/specs/code snippets), using reflective edits guided by execution + evaluation traces against any metric. (GitHub)
FAQ
- What does it optimize? Any text component in a pipeline—single prompts or multi-part textual systems. (GitHub)
- Does it need gradients? No—improvements come from iterative generation/reflection and metric-based selection. (GitHub)
- What metrics can it use? Anything you can score automatically (task success, F1, rubric scores, custom validators). (GitHub)
Resources
- https://dspy.ai/tutorials/entity_extraction/
- https://dspy.ai/tutorials/gepa_ai_program/
- https://dspy.ai/tutorials/gepa_facilitysupportanalyzer/ - structured information extraction and classification
- https://github.com/gepa-ai/gepa/tree/main/src/gepa/adapters/dspy_full_program_adapter - GEPA evolve entire DSPy programs—including signatures, modules, and control flow
Videos
- https://www.youtube.com/watch?v=rrtxyZ4Vnv8 - Matei Zaharia - Reflective Optimization of Agents with GEPA and DSPy
promptfoo
Repo: github.com/promptfoo/promptfoo
Description: Prompt/model evaluation toolkit (plus red teaming) to test prompts, compare providers/models side-by-side, and automate checks in CI/CD. (GitHub)
FAQ
- Is this an optimizer? It’s primarily an eval + testing harness (with security/red-teaming), often used around optimizers. (GitHub)
- What’s the main workflow? Define test cases + assertions, run evals, compare results, and gate changes in CI. (GitHub)
- Does it help with LLM security? Yes—includes red teaming / vulnerability scanning guidance and tooling. (Promptfoo)
AdalFlow (SylphAI)
Repo: github.com/SylphAI-Inc/AdalFlow
Description: A PyTorch-like SDK for building LLM workflows with “auto-differentiative” prompt optimization (zero-shot + few-shot) so you can train prompts against a dataset/metric. (GitHub)
FAQ
- What gets optimized? Instructions and/or few-shot examples depending on how you mark tunable prompt parameters. (GitHub)
- Do I need labels? Typically you need a reward/score; that can be labels, validators, or an LLM-judge rubric. (GitHub)
- When is it a good fit? When you want a training-loop vibe for prompts (datasets, objectives, repeatable optimization). (GitHub)
promptomatix (SalesforceAIResearch)
Repo: github.com/SalesforceAIResearch/promptomatix
Description: Framework to automate prompt creation/optimization from natural-language task descriptions; supports multiple optimization paths (including a DSPy-powered compiler per the paper). (GitHub)
FAQ
- What do I provide as input? A task description (and optionally data/examples); the system generates and refines prompts. (arXiv)
- Is it tied to DSPy? It can integrate with DSPy as one route, but also includes lighter-weight optimizer paths. (arXiv)
- Best use case? Rapidly getting strong baseline prompts without lots of manual prompt-engineering iterations. (GitHub)
PromptWizard (Microsoft)
Repo: github.com/microsoft/PromptWizard
Description: Discrete prompt optimization that “self-evolves” by generating, critiquing, and refining both instructions and in-context examples via iterative feedback. (GitHub)
FAQ
- What’s being optimized? Both the instruction prompt and the few-shot examples/ICL set. (Microsoft)
- What drives improvement? Feedback loops (often LLM-based critique + task scoring) over iterations. (GitHub)
- When should I use it? When “prompt + examples” matter a lot and you want an automated refinement loop. (GitHub)
AutoPrompt (Eladlev)
Repo: github.com/Eladlev/AutoPrompt
Description: Prompt optimization pipeline aimed at real-world use: auto-generates prompts tailored to intent and iteratively “calibrates” them using challenging edge cases. (GitHub)
FAQ
- How does it improve prompts? Generate → evaluate → add hard/edge cases → refine prompt iteratively. (LinkedIn)
- Do I need a test set? Strongly recommended—calibration is only as good as the cases you evaluate against. (LinkedIn)
- Good fit for what tasks? Moderation/classification-ish pipelines and other repeatable “prompt as policy” tasks. (GitHub)
Repo: github.com/meta-llama/prompt-ops
Description: Open-source prompt optimization centered on PDO (Prompt Duel Optimizer): a label-free method using dueling bandits + Thompson sampling to pick better prompts via pairwise comparisons. (GitHub)
FAQ
- What does “label-free” mean here? Optimization can be driven by preference/duel outcomes (often via an LLM judge) rather than ground-truth labels. (arXiv)
- Why duels instead of scoring everything? Pairwise comparisons can be more sample-efficient and robust when absolute scoring is noisy. (arXiv)
- When is it useful? When you can compare outputs reliably (A vs B) even if absolute metrics/labels are hard. (arXiv)
sammo (Microsoft)
Repo: github.com/microsoft/sammo
Description: Structure-aware multi-objective metaprompt optimization: treats prompts as structured “programs” and searches over transformations (e.g., add/remove/replace components) to improve outcomes. (GitHub)
FAQ
- What’s the key idea? Optimize prompt structure (not just wording) with transformations over prompt “objects.” (arXiv)
- What objectives can it handle? Multiple objectives (quality, cost, latency proxies, etc.) depending on how you define scoring. (GitHub)
- Where does it shine? Complex prompts like RAG pipelines where modular structure matters. (Microsoft)
automatic_prompt_engineer (keirp)
Repo: github.com/keirp/automatic_prompt_engineer
Description: Research code for APE (“Large Language Models Are Human-Level Prompt Engineers”): generates many instruction candidates with an LLM and selects the best via a score function. (GitHub)
FAQ
- Is this production-ready tooling? It’s primarily research code; you’ll likely adapt ideas rather than drop it into prod. (GitHub)
- What do I need to run it? A way to propose candidate instructions (LLM) and a repeatable scoring/eval function. (arXiv)
- What’s the main output? A stronger instruction prompt (often for zero-shot or few-shot settings). (arXiv)
prompt-optimizer (vaibkumr)
Repo: github.com/vaibkumr/prompt-optimizer
Description: Prompt compression tooling: minimizes token complexity to reduce API cost/compute while tracking token reduction and semantic similarity; supports “protected tags” to preserve key parts. (GitHub)
FAQ
- Is it about quality or cost? Mostly cost/latency via shorter prompts, while trying to preserve meaning. (GitHub)
- How do protected tags work? You mark sections that must not be altered/removed during optimization. (GitHub)
- When should I avoid it? If small wording changes can break strict formats/contracts and you can’t robustly validate outputs. (GitHub)