Optimize agent instructions, skills, tools, and models (preview)

The agent optimizer supports four optimization targets: instruction tuning (rewrites your agent's system prompt), skill improvement (refines reusable capabilities), tool optimization (improves tool descriptions and parameters), and model selection (evaluates across multiple model deployments). The optimizer automatically determines which targets to improve based on your agent's baseline configuration.

Prerequisites

• A Foundry project with a deployed hosted agent
• The azure.ai.agents CLI extension installed (see Quickstart: Optimize a hosted agent)
• A model deployed for evaluation (for example, gpt-4.1-mini) and an optimization model from the supported list (for example, gpt-5.1)
• Your agent is optimizer-ready (calls load_config())

Which agent gets optimized

The optimizer needs to know which deployed hosted agent to target. It resolves the agent name using the following priority order:

In most cases, you don't need to specify the agent name explicitly. The CLI uses your current azd environment and your project's local agent.yaml file to detect the local agent. Use the --agent flag when you have multiple agents in your project or want to override the default:

azd ai agent optimize

Optimize instructions

The optimizer rewrites and refines your agent's system prompt to improve performance on your evaluation dataset. Instruction tuning activates automatically when your baseline config includes an instructions.md file.

How it works

1. Baseline evaluation. Your agent runs with its current instructions against every task in the dataset. The evaluator scores each response against the task's criteria.
2. Instruction generation. The optimizer analyzes the baseline scores and generates alternative system prompts designed to improve weak areas while maintaining strong areas.
3. Candidate evaluation. The optimizer injects each candidate instruction set into your agent through the OPTIMIZATION_CANDIDATE_ID environment variable and evaluates it against the same dataset.
4. Ranking. The optimizer ranks candidates by composite score and marks the best candidate with ★.

Run instruction optimization

azd ai agent optimize

With a custom config:

# eval.yaml
agent:
  name: my-agent

dataset:
  local_uri: ./eval.jsonl

evaluators:
  - builtin.task_adherence

options:
  eval_model: gpt-4.1-mini
  optimization_model: gpt-5.1
  max_candidates: 5

azd ai agent optimize --config eval.yaml

When dataset includes local_uri and doesn't include name or version, the optimizer treats the dataset as a local file.

What gets changed

The optimizer rewrites the system prompt. Your code stays the same because load_config() returns the new instructions automatically. Common improvements include:

• Adding explicit constraints that the original prompt implied but didn't state
• Restructuring instructions for clarity
• Adding output format specifications
• Strengthening safety and scope boundaries

Example: Before and after

Before (your default instructions):

You are a helpful assistant.

After (optimized):

You are a helpful coding assistant. Follow these guidelines:
1. Always include working code examples
2. Explain your reasoning step by step
3. If a question is outside your expertise, say so clearly
4. Use markdown formatting for code blocks
5. Handle edge cases in code examples

Max candidates

The max_candidates option sets the expected number of candidate instruction sets for the optimization run. The optimizer typically returns after it reaches that count, unless the run stops early because of an error or another stopping condition.

Higher values explore more variations but take longer. The optimizer learns from earlier candidates, so later candidates tend to score higher.

Eval model

The eval model scores agent responses against criteria. Any chat-completion model deployed in your Foundry project works.

azd ai agent optimize --eval-model gpt-4.1-mini

Optimization model (reflection)

You must specify the deployed optimization model that generates candidate configurations. Supported optimization models include the gpt-5 family and DeepSeek models. For the current list and model roles, see Models.

Specify the optimization model in your config file:

options:
  optimization_model: gpt-5.1

Or use the CLI flag:

azd ai agent optimize --optimize-model gpt-5.1

For more details on how these models are used, see Models.

Optimize skills

The optimizer improves existing skills your agent uses. It refines skill descriptions, implementations, and activation criteria. Skill optimization activates automatically when your baseline config includes a skills/ directory.

How it works

1. Baseline evaluation. Same as instruction tuning. The optimizer evaluates your agent against the dataset.

2. Skill improvement. The optimizer analyzes weak areas and refines skill definitions. A skill is a named capability with:

   • Name: For example, "step_by_step_reasoning"
   • Description: What the skill does and when to use it
   • Body: Implementation details or procedure

3. Injection. The agent loads improved skills through load_config(), which makes them available to your agent's instruction set.

   # load_config() returns skills from .agent_configs/baseline/skills/
   config = load_config()
   full_prompt = config.compose_instructions()
   # Returns: "You are a helpful assistant.\n\n## Available Skills\n- **step_by_step_reasoning**: ..."
   

4. Evaluation. The optimizer evaluates the agent with improved skills against the dataset.

Run skill optimization

Ensure your baseline config has a skills/ directory with at least one skill, then run:

azd ai agent optimize

With a config file:

# eval.yaml
agent:
  name: my-agent

dataset:
  local_uri: ./eval.jsonl

evaluators:
  - builtin.task_adherence

options:
  eval_model: gpt-4.1-mini
  optimization_model: gpt-5.1
  max_candidates: 5

azd ai agent optimize --config eval.yaml

Skill file downloads

For candidates that include skill files, the config loader can download them through the resolver API. Skills use the open Agent Skills format and are stored in a local directory.

from azure.ai.agentserver.optimization import load_config, load_skills_from_dir
from pathlib import Path

config = load_config()

if config.skills:
    print(f"Skills loaded from optimization config:")
    for skill in config.skills:
        print(f"  - {skill.name}: {skill.description}")
elif config.skills_dir:
    # Load skills from local directory
    skills = load_skills_from_dir(Path(config.skills_dir))
    config.skills.extend(skills)
    for skill in skills:
        print(f"  - {skill.name}: {skill.description}")

Learn more about the Agent Skills format at agentskills.io.

Optimize tools

The optimizer improves tool descriptions and parameters in your tools.json file to help the model call tools more accurately. Tool optimization activates automatically when your baseline config includes a tools.json file.

How it works

1. Baseline evaluation. The optimizer evaluates your agent against the dataset, including any tool calls it makes.
2. Tool analysis. The optimizer identifies tool calls that fail or produce suboptimal results and analyzes the root cause—unclear descriptions, missing parameters, or ambiguous naming.
3. Description refinement. The optimizer generates improved tool definitions with clearer descriptions, better parameter documentation, and more precise function names.
4. Evaluation. The optimizer evaluates the agent with improved tool definitions against the dataset.

Run tool optimization

Ensure your baseline config has a tools.json file, then run:

azd ai agent optimize

What gets changed

The optimizer refines your tools.json definitions. Common improvements include:

• Clearer function descriptions that help the model know when to call a tool
• More specific parameter descriptions that reduce inaccurate arguments
• Added constraints (enums, required fields) that prevent invalid inputs

Your tool implementation code stays the same. Only the definitions the model sees change.

Optimize model selection

The optimizer evaluates your agent across multiple model deployments to find the best quality-to-cost trade-off. Each model runs against the same dataset, so you can compare results directly. Model optimization activates when you specify model candidates in optimization_config.model_search_space.

Configure model candidates

Specify the models to evaluate in your eval.yaml:

# eval.yaml
agent:
  name: my-agent

dataset:
  local_uri: ./eval.jsonl

evaluators:
  - builtin.task_adherence

options:
  eval_model: gpt-4.1-mini
  optimization_model: gpt-5.1
  max_candidates: 5
  optimization_config:
    # Target model candidates for optimization.
    model_search_space:
      - gpt-4.1
      - gpt-4.1-mini
      - gpt-4o

Each model listed under optimization_config.model_search_space must be deployed in your Foundry project.

Run model optimization

azd ai agent optimize --config eval.yaml

The optimizer evaluates your agent with each specified model deployment and ranks the results by score and token cost.

Combine with other targets

When your baseline includes instructions, skills, and model candidates, the optimizer runs all targets together in a single run:

# eval.yaml — all targets active
agent:
  name: my-agent

dataset:
  local_uri: ./eval.jsonl

evaluators:
  - builtin.task_adherence
  - builtin.intent_resolution

options:
  eval_model: gpt-4.1-mini
  optimization_model: gpt-5.1
  max_candidates: 5
  # Free-form per-attribute overrides sent to the optimization service.
  # Each key is an attribute name; the value is a JSON-encoded payload.
  optimization_config:
    # Target model candidates for optimization.
    model_search_space:
      - gpt-4.1
      - gpt-5

This produces candidates that combine improved instructions with different model options, giving you the full picture of what works best.

Interpret results

After optimization completes, review the results table. The ★ marks the best candidate. For the results table columns, scoring details, score-improvement thresholds, and the portal view, see Understand optimization results.

Deploy the winner

The recommended workflow is to apply the optimized config locally, then deploy:

# Apply the winning candidate locally
azd ai agent optimize apply --candidate <candidate-id>

# Deploy with the optimized config
azd deploy

This downloads the optimized configuration into .agent_configs/<candidate_id>/ in your project. On next deploy, your agent uses the improved instructions and tool descriptions.

Alternatively, you can deploy directly via the API (useful for quick A/B testing):

azd ai agent optimize deploy --candidate <candidate-id>

If all candidates score lower than the baseline, don't deploy any candidate. The baseline configuration remains active.

Troubleshooting

Related content

• Agent optimizer overview
• Create a custom evaluation dataset
• Make your agent optimizer-ready
• Quickstart: Optimize a hosted agent