DSPy — Automatic Prompt Generation

Many engineers working with LLMs and agents — and not just AI specialists — face tasks where the correctness of the agent's behavior largely depends on the prompt.

For simple tasks a prompt like "You are a smart agent, answer the user's questions" is enough. But if the agent has to make complex function calls and handle non-standard requests, writing a stable prompt for the calls and for the final answer is far from easy, and when you switch to a different LLM model, a prompt that used to work may stop working. How to write such prompts by hand is covered in the article on prompt engineering, and how an agent calls tools in the article on RAG and function calling.

DSPy (GitHub) turns this from writing text into a measurable process. What it does, in simple terms:

• Instead of a long text you describe the task as a form: what inputs there are and what outputs are needed.
• The rules for calling tools are fixed in code, not in "words" inside the prompt.
• Quality is measured by a metric, and DSPy picks the wording to fit that metric.
• When you change the model or the functions, you run the tuning again instead of rewriting everything by hand.

What this gives you:

• Scalability: it is easier to add tools and new answer formats.
• A cheap model switch: it becomes a re-run of the tuning, with a score to compare before and after.
• Reproducibility: the prompt is stored as an artifact and can be restored.
• Control: behavior is defined by structure and a metric, not by "magic text".

The price is the data set and the time a tuning run takes; both are covered below. The example is an agent that searches a local database, to show that agents can be quite complex.

Signature and tools

The signature says which fields arrive at the agent and what it must return. Tools are ordinary functions with clear inputs and outputs:

import sqlite3
from contextlib import closing

import dspy

# A local model in LM Studio; any model LiteLLM can reach will do.
dspy.configure(
    lm=dspy.LM(
        "openai/google/gemma-3n-e4b",
        api_base="http://localhost:1234/v1",
        api_key="lm-studio",
        temperature=0,
    )
)


def search_kb(query: str) -> list[str]:
    """Searches the knowledge base and returns the matching fragments."""
    with closing(sqlite3.connect("kb.db")) as db:
        rows = db.execute(
            "SELECT text FROM articles WHERE text LIKE ? LIMIT 5", (f"%{query}%",)
        ).fetchall()
    return [row[0] for row in rows]


class AnswerFromKb(dspy.Signature):
    """Answer the question using only the knowledge base."""

    question: str = dspy.InputField()
    answer: str = dspy.OutputField(desc="a short answer, no explanations")


agent = dspy.ReAct(AnswerFromKb, tools=[search_kb], max_iters=5)

Data

The data set is a list of questions with correct answers:

[
  {
    "question": "...",
    "answer": "..."
  }
]

According to the DSPy documentation, 30 examples often bring substantial value, but you should aim for at least 300, and for MIPROv2 it advises 200 or more to prevent overfitting. What matters is that they are diverse.

The split is the opposite of what you are used to from neural networks: 20 percent for training and 80 for validation, because prompt optimizers easily overfit to a small training set. On top of that comes a held-out test set the optimizer never sees, and its score is the one to trust:

import json
import random

with open("dataset.json", encoding="utf-8") as f:
    examples = [dspy.Example(**row).with_inputs("question") for row in json.load(f)]

random.Random(0).shuffle(examples)
holdout = len(examples) // 5
testset, rest = examples[:holdout], examples[holdout:]
split = len(rest) // 5
trainset, valset = rest[:split], rest[split:]

Metric and tuning

MIPROv2 runs the agent to collect good traces, proposes variants of the instructions and few-shot examples, and searches among them by the metric:

def metric(example, prediction, trace=None):
    return example.answer.lower() in prediction.answer.lower()


evaluate = dspy.Evaluate(devset=testset, metric=metric, num_threads=4)
print("before:", evaluate(agent))

optimizer = dspy.MIPROv2(metric=metric, auto="light", num_threads=4)
tuned = optimizer.compile(agent, trainset=trainset, valset=valset)

print("after:", evaluate(tuned))
tuned.save("agent.json")

A run is not cheap. With auto="light" this agent gets 20 trials on mini-batches of 35 examples, plus a full pass over up to 100 validation examples every 5 trials, and every agent run is several model calls. On a local model that is a long wait, on a cloud one a bill. The proposals can go to a stronger model than the one the agent runs on, through prompt_model and task_model.

Using the result

agent.json is an artifact: it goes into the repository and loads without any tuning. When the data or the model changes, run the tuning again and compare the scores on the test set.

agent = dspy.ReAct(AnswerFromKb, tools=[search_kb], max_iters=5)
agent.load("agent.json")
print(agent(question="How long is the refund period?").answer)