Skip to main content

Agents

An agent is the runnable logic Spec27 evaluates. It receives the input from an entry, sends that input to your service or model, and returns the final output to be scored. Agents are reusable across many evals.

Three ways to create an agent​

You do not have to start from a blank editor. Spec27 offers three paths, and they all produce the same kind of agent:

  • Agent Builder — describe the agent in plain language and have Spec27 generate the code.
  • Registry integrations — copy a prebuilt integration for a popular platform (OpenAI, Gemini, LangGraph, Botpress, and more) and supply your credentials.
  • Writing the code yourself — author the agent directly in the code editor, described below.

For an agent that holds a conversation across several turns rather than answering a single input, see Writing a multi-turn agent.

Write an agent by hand​

When you create an agent, the page opens on the Preview tab with a JavaScript code editor. Enter a name in Name the agent, then write the agent as a small client that takes an input, calls your system, and returns the output:

return async function process(input) {
  const response = await fetch("https://your-service.example.com/respond", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ input }),
  });

  const data = await response.json();
  return data.output;
}

Before you save, confirm that:

  • the request URL and method match your API,
  • the request body matches your API contract, and
  • the returned value is the final output you want Spec27 to score.

Configure secrets and rate limits​

The Configuration panel beside the editor has two tabs:

  • Secrets — declare the project secret keys this agent needs at runtime. Select Add secret for each key (for example OPENAI_API_KEY). You declare keys here and set their values as project secrets; preview is blocked until any missing secrets exist.
  • Rate limit — switch On to cap preview and runtime traffic, then set the maximum number of runs per time window.

Select Create Agent to save.

Preview an agent​

Use preview to validate an agent before attaching it to an eval. It runs the agent in the sandbox and does not create a persisted run.

  1. Open the saved agent and select Editor.
  2. Enter sample text in Input and select Run preview.
  3. Review Output and the Console tab for logs.

If execution fails, the page shows Execution failed with the error type, message, and stack trace — use that to tell whether the problem is in the agent code, the API contract, or the upstream service.

Review the agent​

The agent detail page is the hub for an agent. It shows:

  • Created, Updated, Used in evals, and Results at a glance,
  • the Configuration (rate limit and secret counts),
  • Required secrets, with a Create project secret link for any that are missing, and
  • the agent Content.