# WarpMetrics

> Observability for AI agents. WarpMetrics tracks every LLM call -- cost, latency, tokens, outcomes -- and exposes it back to agents via MCP tools so they can query their own history and improve.

WarpMetrics is an observability platform purpose-built for AI agents. It models agent workflows as a hierarchy of five primitives -- Run, Group, Call, Outcome, and Act -- and records every LLM API invocation with full context: model, messages, response, tokens, cost, latency, tool calls, and status.

Unlike tracing tools designed for human dashboards (Langfuse, Helicone, etc.), WarpMetrics exposes all tracked data to agents themselves via 17 MCP tools. This enables a feedback cycle where agents query their own performance history, identify failures, and self-correct without human intervention.

The SDK is open source (MIT licensed). The platform is hosted at warpmetrics.com.

- Website: https://warpmetrics.com
- Documentation: https://warpmetrics.com/docs
- SDK (npm): https://www.npmjs.com/package/@warpmetrics/warp
- MCP server (npm): https://www.npmjs.com/package/@warpmetrics/mcp
- API reference: https://api.warpmetrics.com/v1/docs
- OpenAPI spec: https://api.warpmetrics.com/v1/docs/openapi.json
- Contact: https://warpmetrics.com/contact
- Chief Prompt Whisperer: Nikolai Onken (@nonken on X)

---

## How WarpMetrics Works

WarpMetrics operates in four steps:

1. **Wrap your LLM client** -- Use `warp(client)` to wrap your OpenAI or Anthropic client. All calls are automatically intercepted and tracked in memory.
2. **Organize with runs and groups** -- Create runs for top-level workflows and groups for nested phases. Link calls to build the execution tree.
3. **Record outcomes** -- Mark runs with outcomes like "Completed" or "Failed" to classify results and track success rates over time.
4. **Query via dashboard or MCP** -- Your team sees costs, latency, success rates, and detailed call logs in the dashboard. Your agents query the same data via 17 MCP tools to find failures and self-correct.

### How WarpMetrics differs from other tools

| WarpMetrics | Other tools (Langfuse, Helicone, etc.) |
|---|---|
| 17 MCP tools for agent self-querying | Human-only dashboards, no agent access |
| Runs with structured outcomes and success rates | Flat traces without run-level outcomes |
| Agents query, adjust, and verify in a loop | No programmatic query interface for agents |
| Automatic tracking of streaming, tool calls, and errors | No MCP integration |
| Async SDK -- no proxy, no added latency | No outcome classification or success rate tracking |

---

## Quickstart

### Prerequisites

- Node.js 18+
- An OpenAI or Anthropic API key
- A WarpMetrics account (free tier available)

### Step 1: Create an account

Sign up at https://warpmetrics.com/signup

### Step 2: Create an API key

Go to your API keys page at https://warpmetrics.com/app/api-keys and create a new key. Keys start with `wm_live_` (production) or `wm_test_` (testing).

### Step 3: Install the SDK

```bash
npm install @warpmetrics/warp
```

Set the `WARPMETRICS_API_KEY` environment variable to your API key:

```bash
export WARPMETRICS_API_KEY=wm_live_your_api_key_here
```

### Step 4: Instrument your first call

```javascript
import { warp, run, call, outcome } from '@warpmetrics/warp';
import OpenAI from 'openai';

// 1. Wrap your client
const openai = warp(new OpenAI());

// 2. Create a run
const r = run('my-agent');

// 3. Make LLM calls (automatically captured)
const res = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'Hello!' }],
});

// 4. Emit the call to the run
call(r, res);

// 5. Record the outcome
outcome(r, 'Completed');
```

### Using with Anthropic

```javascript
import { warp } from '@warpmetrics/warp';
import Anthropic from '@anthropic-ai/sdk';

const anthropic = warp(new Anthropic());

const res = await anthropic.messages.create({
  model: 'claude-sonnet-4-5-20250514',
  max_tokens: 1024,
  messages: [{ role: 'user', content: 'Hello' }],
});
```

---

## Core Concepts

WarpMetrics models AI agent workflows as a hierarchy of five primitives. Every agent execution is modeled as a tree:

```
Run
+-- Group (planning phase)
|   +-- Call (gpt-4o)
|   +-- Call (gpt-4o)
+-- Group (execution phase)
|   +-- Group (sub-task)
|   |   +-- Call (gpt-4o-mini)
|   +-- Call (gpt-4o)
+-- Outcome ("failed")
+-- Act ("retry")
    +-- Run (follow-up attempt)
```

### Quick reference

| Function | Description | ID prefix |
|---|---|---|
| `run(label, opts?)` | Create a top-level run | `wm_run_` |
| `group(target, label, opts?)` | Create a group inside a run or group | `wm_grp_` |
| `call(target, response, opts?)` | Link an LLM call to a run or group | -- |
| `outcome(target, name, opts?)` | Record a result on any entity | `wm_oc_` |
| `act(outcome, name, opts?)` | Record an action after an outcome | `wm_act_` |
| `ref(target)` | Get the tracking ID for any entity | -- |
| `flush()` | Manually send pending events | -- |

---

## Runs

A run is the top-level unit of work. It represents a single, complete execution of your AI agent -- from start to finish.

Every time your agent processes a request, it performs a run. A run is the container for everything that happens during that execution: the LLM calls your agent makes, the groups it organizes them into, and the outcomes it produces. Think of a run as one row in a log -- "the agent did this task, made these calls, and produced this result." Runs are how you track what your agent is doing over time.

**ID format:** `wm_run_<ulid>`

### When to create a run

Create a run at the start of each distinct task your agent handles:

- **Support agent** -- one run per customer conversation
- **Code reviewer** -- one run per pull request
- **Content generator** -- one run per article or document
- **Data pipeline** -- one run per extraction job
- **RAG system** -- one run per user query
- **Multi-step agent** -- one run per task attempt

### Labels

Every run has a label that categorizes it. Runs with the same label are grouped together in the dashboard, so you can compare performance across executions of the same agent or workflow.

Labels should describe the type of work, not the specific instance. Use the opts bag for instance-specific metadata.

```javascript
// Good: label describes the workflow type
run('Code review');
run('Support ticket');
run('Content generation');

// Bad: label contains instance-specific data
run('Code review PR #42');       // use opts instead
run('Support ticket for John');  // use opts instead
```

### API

```
run(label: string, opts?: object): Run
run(ref: Act, label: string, opts?: object): Run
```

Creates a new run with the given label. Returns a frozen run handle. When ref is an act, creates a follow-up run linked to that act.

**Parameters:**
- `label` -- Category name for this run. Runs with the same label are grouped in dashboards.
- `opts` -- Optional metadata object. Stored with the run and visible in the dashboard. Use for instance-specific data like PR numbers, user IDs, etc.

**Returns:** A frozen object with `id` (string) and `_type` (`'run'`). Pass this to `group()`, `call()`, or `outcome()` to build the execution tree.

### Basic example

```javascript
import OpenAI from 'openai';
import { warp, run, call, outcome, flush } from '@warpmetrics/warp';

const openai = warp(new OpenAI());

// Start a run
const r = run('Summarizer');

// Make an LLM call
const res = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'Summarize this article...' }],
});

// Link the call to the run
call(r, res);

// Record the outcome
outcome(r, 'Completed');

await flush();
```

### Passing metadata with opts

The opts bag lets you attach arbitrary metadata to a run. This is useful for filtering, debugging, and correlating runs with your own systems.

```javascript
const r = run('Code review', {
  name: 'Review PR #42',
  pr: 42,
  repo: 'acme/api',
  author: 'alice',
  link: 'https://github.com/acme/api/pull/42',
});
```

### Follow-up runs

When an agent retries or iterates on a task, you can chain runs together using the act primitive. A follow-up run is linked to the act that triggered it, creating a traceable improvement chain.

```javascript
import { run, outcome, act, flush } from '@warpmetrics/warp';

// First attempt
const r1 = run('Code review');
// ... calls ...
const oc = outcome(r1, 'Failed', { reason: 'Tests failing' });

// Decide to retry
const a = act(oc, 'retry');

// Second attempt -- linked to the act
const r2 = run(a, 'Code review');
// ... more calls ...
outcome(r2, 'Completed');

await flush();
```

When passing an act as the first argument, the second argument becomes the label and the third becomes opts:

```javascript
run(actRef, 'Code review', { attempt: 2 });
```

### What gets tracked on a run

| Field | Description |
|---|---|
| Label | Category name for grouping runs |
| Opts | Custom metadata you provide |
| Groups | All groups created under this run |
| Calls | All LLM calls linked directly to this run |
| Outcomes | Results recorded on this run |
| Timestamp | When the run was created |
| Cost | Total cost across all calls (computed server-side) |
| Latency | Total latency across all calls (computed) |
| Tokens | Total token usage across all calls (computed) |

### Tips for runs

- Use consistent labels across your codebase. The label is how WarpMetrics groups runs for comparison.
- Put instance-specific data in opts, not in the label. This keeps your dashboards clean.
- You can attach calls directly to a run without groups. Groups are optional -- use them when your agent has distinct phases.
- Runs are created instantly. The SDK queues events and flushes them in batches, so there's no latency impact on your agent.

---

## Groups

Groups organize LLM calls into logical phases within a run. They give your agent's execution tree structure and make it easy to see what happened at each step.

A group represents a phase, step, or logical unit within a run. If a run is the "what" (the task), groups are the "how" (the steps the agent took to complete it).

Groups are optional. For simple agents with a single LLM call, you can link calls directly to the run. But for multi-step agents, groups provide crucial visibility into which phase produced which calls, how long each phase took, and where failures occur.

**ID format:** `wm_grp_<ulid>`

### When to use groups

Use groups when your agent has distinct phases:

- **Plan -> Execute** -- a planning group that decides what to do, then an execution group that does it
- **Analyze -> Synthesize** -- one group gathers information, another produces the final output
- **Triage -> Route -> Handle** -- classification step, routing decision, then specialized handling
- **Research -> Draft -> Review** -- multi-pass content generation with self-review
- **Validate -> Transform -> Output** -- data processing pipelines with validation steps

### Labels

Like runs, groups have a label for categorization. Groups with the same label are aggregated across runs, so you can see how your "planning" phase performs across all executions.

```javascript
// Labels describe the phase, not the instance
group(r, 'Planning');
group(r, 'Code generation');
group(r, 'Review');
```

### API

```
group(target: Run | Group, label: string, opts?: object): Group
```

Creates a new group linked to a run or parent group. Returns a frozen group handle.

**Parameters:**
- `target` -- the parent run or group. The new group is automatically linked as a child.
- `label` -- category name for this group. Groups with the same label are aggregated in dashboards.
- `opts` -- optional metadata object.

**Returns:** A frozen object with `id` and `_type: 'group'`. Pass this to `call()`, `outcome()`, or nest further with another `group()`.

### Basic example

```javascript
import OpenAI from 'openai';
import { warp, run, group, call, outcome, flush } from '@warpmetrics/warp';

const openai = warp(new OpenAI());

const r = run('Code review');

// Planning phase
const planning = group(r, 'Planning');
const planRes = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'What should we check in this PR?' }],
});
call(planning, planRes);

// Execution phase
const execution = group(r, 'Execution');
const execRes = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'Review the code for these issues...' }],
});
call(execution, execRes);

outcome(r, 'Completed');
await flush();
```

### Nesting groups

Groups can be nested to any depth. Pass a group as the target to create a sub-group. This is useful for complex agents with nested control flow.

```javascript
const r = run('Data pipeline');

const validation = group(r, 'Validation');
const schemaCheck = group(validation, 'Schema check');
call(schemaCheck, await openai.chat.completions.create({...}));

const dataCheck = group(validation, 'Data quality');
call(dataCheck, await openai.chat.completions.create({...}));

const transform = group(r, 'Transform');
call(transform, await openai.chat.completions.create({...}));
```

This produces:

```
Run (Data pipeline)
+-- Validation
|   +-- Schema check
|   |   +-- gpt-4o call
|   +-- Data quality
|       +-- gpt-4o call
+-- Transform
    +-- gpt-4o call
```

### Dynamic branching

Groups work naturally with dynamic control flow. Create different groups depending on the agent's decisions -- WarpMetrics captures whichever path is taken.

```javascript
const r = run('Support agent');

// Triage
const triage = group(r, 'Triage');
const triageRes = await openai.chat.completions.create({
  model: 'gpt-4o-mini',
  messages: [{ role: 'user', content: 'Classify this ticket...' }],
});
call(triage, triageRes);

const category = triageRes.choices[0].message.content;

// Branch based on classification
if (category === 'billing') {
  const billing = group(r, 'Billing handler');
  call(billing, await openai.chat.completions.create({...}));
} else if (category === 'technical') {
  const tech = group(r, 'Technical handler');
  const research = group(tech, 'Research');
  call(research, await openai.chat.completions.create({...}));
  const response = group(tech, 'Response');
  call(response, await openai.chat.completions.create({...}));
}
```

### Outcomes on groups

You can record outcomes on individual groups, not just runs. This lets you track success/failure at the phase level.

```javascript
const validation = group(r, 'Validation');
// ... calls ...

if (isValid) {
  outcome(validation, 'Passed');
} else {
  outcome(validation, 'Failed', { errors: validationErrors });
}
```

### Tips for groups

- Groups are optional. For simple single-call agents, link calls directly to the run.
- Keep labels consistent. If you sometimes call a phase "Planning" and sometimes "Plan", they'll appear as separate groups in dashboards.
- Use nesting sparingly. One or two levels deep is usually enough. Deep nesting makes trees harder to read.
- Groups are created instantly with no performance overhead. The SDK queues events in memory.

---

## Calls

Calls are individual LLM API invocations. The SDK automatically captures the full context -- messages, response, tokens, cost, latency, and status -- so you never have to log any of it manually.

### How call tracking works

Call tracking is a two-step process:

1. **Intercept** -- When you wrap a client with `warp()`, the SDK intercepts every API call. It records the request metadata (model, messages, tools) and the response (content, tokens, latency, status) in memory -- but doesn't send anything yet.

2. **Link** -- When you call `call(target, response)`, the SDK links the intercepted call to a run or group and queues it for transmission. Only calls that are explicitly linked are ever sent to the API.

This **lazy linking** design means you can use your wrapped client anywhere -- in utility functions, libraries, tests -- and only the calls you explicitly link will be tracked. No noise.

### API

```
call(target: Run | Group, response: Response, opts?: object): void
```

Links an intercepted LLM response to a run or group. Queues the call for batch transmission.

**Parameters:**
- `target` -- the run or group to link this call to.
- `response` -- the response object returned by the wrapped LLM client. For streams, pass the stream object after consuming it.
- `opts` -- optional metadata to attach to this call.

**Returns:** Nothing. This is a fire-and-forget operation. The call is queued for batch transmission.

### Basic example

```javascript
import OpenAI from 'openai';
import { warp, run, group, call, flush } from '@warpmetrics/warp';

const openai = warp(new OpenAI());
const r = run('Summarizer');
const g = group(r, 'Analysis');

// Make the LLM call as usual
const res = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'Summarize this...' }],
});

// Link it to the group
call(g, res);

// The response works exactly as before
console.log(res.choices[0].message.content);

await flush();
```

### What gets captured

The SDK automatically extracts all of this from the LLM response:

| Field | Description |
|---|---|
| `provider` | openai or anthropic |
| `model` | The model used (e.g., gpt-4o, claude-sonnet-4-5-20250514) |
| `messages` | The input messages sent to the LLM |
| `response` | The text content of the LLM's response |
| `tools` | Tool/function names if tools were provided |
| `toolCalls` | Tool calls made by the LLM (id, name, arguments) |
| `tokens.prompt` | Input token count |
| `tokens.completion` | Output token count |
| `tokens.total` | Total tokens used |
| `tokens.cachedInput` | Cached input tokens (OpenAI) |
| `tokens.cacheWrite` | Cache write tokens (Anthropic) |
| `tokens.cacheRead` | Cache read tokens (Anthropic) |
| `cost` | Computed cost in USD (server-side) |
| `latency` | Wall-clock time in milliseconds |
| `status` | success or error |
| `error` | Error message if the call failed |
| `timestamp` | ISO 8601 timestamp |

### Linking to runs vs groups

You can link a call to either a run or a group. Both work the same way:

```javascript
// Link directly to a run (no groups needed)
const r = run('Simple agent');
const res = await openai.chat.completions.create({...});
call(r, res);

// Or link to a group within a run
const r2 = run('Complex agent');
const g = group(r2, 'Analysis');
const res2 = await openai.chat.completions.create({...});
call(g, res2);
```

### Streaming

Streaming works automatically. The SDK wraps the async iterator to collect content chunks and capture usage data as the stream completes. Call `call()` after consuming the stream.

```javascript
const stream = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'Hello!' }],
  stream: true,
});

// Consume the stream as usual
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content || '');
}

// Link after the stream is consumed
call(g, stream);
```

Token counts and latency are captured when the stream finishes. Cost is calculated server-side based on the model's pricing.

### Error tracking

When an LLM call throws an error, the SDK captures it and attaches tracking data to the error object. You can still link failed calls:

```javascript
try {
  const res = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: [{ role: 'user', content: 'Hello' }],
  });
  call(g, res);
} catch (err) {
  // call() automatically extracts tracking data from the error
  call(g, err);

  // Record an outcome for the failure
  outcome(g, 'Error', { message: err.message });
}
```

Failed calls appear in the dashboard with `status: 'error'` and include the error message, latency, and any partial data available.

### Outcomes on calls

You can record outcomes on individual calls. Use the response object as the target:

```javascript
const res = await openai.chat.completions.create({...});
call(g, res);

// Record an outcome on the call itself
outcome(res, 'Helpful');
outcome(res, 'Hallucination Free');
```

### Supported providers

**OpenAI:**
- `chat.completions.create()` -- standard and streaming
- `responses.create()` -- standard and streaming
- Automatic tracking of tool calls, cached tokens

**Anthropic:**
- `messages.create()` -- standard and streaming
- Automatic tracking of cache write/read tokens

### Tips for calls

- Each response can only be linked once. The second `call()` with the same response is silently ignored.
- Unlinked responses are never transmitted. If you forget to call `call()`, the data stays in memory until garbage collected -- no noise in your dashboard.
- The wrapped client is fully compatible. It returns the same types, supports the same options, and works with TypeScript autocomplete.
- `call()` is a fire-and-forget operation. It never throws and adds no latency to your agent.

---

## Outcomes

Outcomes record the result of a run, group, or call. They are the bridge between "the agent did something" and "it worked" -- the foundation of success rate tracking.

An outcome is a named result attached to any entity in your execution tree. It answers the question: "What happened?"

LLM calls give you tokens, latency, and cost. But those don't tell you if the agent actually did its job. Outcomes close that gap. They let you define what success means for your agent and track it over time.

**ID format:** `wm_oc_<ulid>`

### Why outcomes matter

Most LLM observability tools show you what the model said and how much it cost. That doesn't answer the question that matters: is your agent actually working?

Outcomes let you track:

- **Success rate** -- what percentage of runs completed successfully?
- **Failure patterns** -- which outcome names appear most? What metadata do they carry?
- **Phase-level quality** -- which step in your pipeline fails most often?
- **Improvement over time** -- are retries helping? Is the new prompt better?

### API

```
outcome(target: Run | Group | Response, name: string, opts?: object): Outcome
```

Records an outcome on a run, group, or call response. Returns a frozen outcome handle.

**Parameters:**
- `target` -- the run, group, or LLM response to attach this outcome to.
- `name` -- a short label for the outcome. Use consistent names across your codebase -- these are what get classified and aggregated.
- `opts` -- optional metadata. Use for context: reasons, error details, scores, etc.

**Returns:** A frozen object with `id` and `_type: 'outcome'`. Pass this to `act()` to trigger a follow-up action.

### Basic examples

```javascript
import { run, outcome, flush } from '@warpmetrics/warp';

const r = run('Code review');
// ... groups and calls ...

// Simple outcome
outcome(r, 'Completed');

// Outcome with metadata
outcome(r, 'Completed', {
  linesReviewed: 342,
  issuesFound: 3,
  severity: 'medium',
});

// Failure outcome
outcome(r, 'Failed', {
  reason: 'Rate limit exceeded',
  retryable: true,
});

await flush();
```

### Outcome targets

You can attach outcomes to any level of the execution tree:

```javascript
// On a run -- "the whole task succeeded"
outcome(r, 'Completed');

// On a group -- "this phase succeeded"
const validation = group(r, 'Validation');
outcome(validation, 'Passed');

// On a call -- "this specific LLM output was good"
const res = await openai.chat.completions.create({...});
call(g, res);
outcome(res, 'Accurate');
```

Multiple outcomes can be attached to the same target. This is useful for tracking multiple quality dimensions:

```javascript
outcome(res, 'Accurate');
outcome(res, 'Well Formatted');
outcome(res, 'No Hallucinations');
```

### Classifications

Outcome names are free-form strings. To compute success rates, WarpMetrics maps outcome names to one of three classifications:

- **Success** -- completed, approved, passed, resolved, shipped
- **Failure** -- failed, error, rejected, timeout, invalid
- **Neutral** -- skipped, deferred, partial, unknown

Classifications are configured in the WarpMetrics dashboard under Outcomes. When the system sees an outcome name it hasn't classified yet, it appears as unclassified until you assign it.

Only the **last outcome** on a run determines its success/failure status. If a run has both a "Failed" and then a "Completed" outcome, the run counts as a success.

### Naming conventions

Use consistent, human-readable names with Title Case and spaces. The outcome name is what gets classified and aggregated, so consistency matters.

```javascript
// Good: consistent, descriptive names
outcome(r, 'Completed');
outcome(r, 'Failed');
outcome(r, 'Rate Limited');
outcome(r, 'Validation Error');

// Bad: inconsistent casing or vague names
outcome(r, 'COMPLETED');      // use Title Case, not all caps
outcome(r, 'error_123');      // too specific -- use opts for details
outcome(r, 'ok');             // too vague
outcome(r, 'rate-limited');   // use spaces, not dashes
```

Put specific details in the opts bag, not the name. This keeps your classifications clean:

```javascript
// Good: generic name + specific opts
outcome(r, 'Failed', { reason: 'Rate limit on gpt-4o', code: 429 });

// Bad: encoding details in the name
outcome(r, 'rate-limit-gpt-4o-429');
```

### Tips for outcomes

- Always record an outcome on your runs. Without outcomes, you can track cost and latency but not whether your agent is working.
- Use outcomes on groups to pinpoint which phase of your pipeline fails most.
- Outcome opts are searchable in the dashboard. Use them for debugging context.
- The outcome handle returned by `outcome()` can be passed to `act()` to create follow-up actions.

---

## Acts & Feedback Loops

Acts record the action to be taken after an outcome and create traceable chains between runs. They're how you model retry logic, self-improvement, and iterative refinement.

An act represents a decision to do something in response to an outcome. If an outcome says "what happened," an act says "what we're going to do about it."

**ID format:** `wm_act_<ulid>`

### Why acts exist

The best AI agents don't just run once. They evaluate their own output, detect problems, and try again. This creates a loop:

Run -> Outcome -> Act -> Run -> Outcome -> Act -> Run -> ...

Acts make this loop observable. Without them, you'd see 3 runs with the same label and no way to know they're related. With them, the dashboard shows the full chain: which attempt led to which, what the agent decided to do differently, and whether the retry actually helped.

```
Run 1 (Code review)
+-- Outcome: "failed" (tests failing)
    +-- Act: "retry" (fix and rerun)
        +-- Run 2 (Code review) <-- follow-up
            +-- Outcome: "completed"
```

### Important constraint

Acts can **only** be created from outcomes. You can't create an act from a run or group directly. The chain is always:

    outcome -> act -> run -> outcome -> act -> run -> ...

This is by design. An act is a response to a specific outcome. Without knowing what happened (the outcome), you can't meaningfully decide what to do next (the act).

### API

```
act(target: Outcome, name: string, opts?: object): Act
```

Records an action to take after an outcome. Returns a frozen act handle.

**Parameters:**
- `target` -- the outcome this act responds to. Must be an outcome handle or a `wm_oc_` ref string.
- `name` -- what action is being taken: "retry", "refine-prompt", "switch-model", "escalate", etc.
- `opts` -- optional metadata about the action (what changed, why, strategy, etc.).

**Returns:** A frozen object with `id` and `_type: 'act'`. Pass this as the first argument to `run()` to create a follow-up run.

To create a follow-up run:

```
run(act: Act, label: string, opts?: object): Run
```

### Basic example

```javascript
import { run, group, call, outcome, act, flush } from '@warpmetrics/warp';

// First attempt
const r1 = run('Content generator');
const draft = group(r1, 'Drafting');
call(draft, await openai.chat.completions.create({...}));

// Evaluate the result
const oc = outcome(r1, 'Low Quality', {
  reason: 'Too generic, needs more specific examples',
});

// Decide to retry with a different approach
const a = act(oc, 'Refine Prompt', {
  change: 'Added domain-specific examples to prompt',
});

// Second attempt -- linked to the act
const r2 = run(a, 'Content generator');
const draft2 = group(r2, 'Drafting');
call(draft2, await openai.chat.completions.create({...}));

outcome(r2, 'Completed', { quality: 'good' });

await flush();
```

### Common act names

| Name | Description |
|---|---|
| Retry | Same approach, try again (transient failure, rate limit) |
| Refine Prompt | Adjusted the prompt based on output quality |
| Switch Model | Trying a different model (e.g., gpt-4o -> claude) |
| Decompose | Breaking the task into smaller sub-tasks |
| Escalate | Handing off to a human or more capable system |
| Add Context | Adding more context or examples to the input |
| Fix and Retry | Fixing a specific issue and retrying |

### Self-improving agent pattern

Acts shine in agents that evaluate and improve their own output. Here's a pattern for an agent that loops until it's satisfied:

```javascript
async function selfImprovingAgent(task, maxAttempts = 3) {
  let actRef = null;

  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    // Create run (first attempt or follow-up)
    const r = actRef
      ? run(actRef, 'Writer', { attempt })
      : run('Writer', { attempt });

    // Generate
    const gen = group(r, 'Generate');
    const result = await openai.chat.completions.create({
      model: 'gpt-4o',
      messages: [{ role: 'user', content: task }],
    });
    call(gen, result);

    // Evaluate
    const review = group(r, 'Self-review');
    const evaluation = await openai.chat.completions.create({
      model: 'gpt-4o',
      messages: [{
        role: 'user',
        content: `Rate this output 1-10: ${result.choices[0].message.content}`,
      }],
    });
    call(review, evaluation);

    const score = parseInt(evaluation.choices[0].message.content);

    if (score >= 8) {
      outcome(r, 'Completed', { score });
      break;
    }

    // Not good enough -- act and retry
    const oc = outcome(r, 'Below Threshold', { score });
    actRef = act(oc, 'retry', { targetScore: 8, actualScore: score });
  }

  await flush();
}
```

### What gets tracked with acts

In the dashboard, act chains let you:

- See the full retry history for any run -- how many attempts, what changed each time
- Measure improvement -- did the retry produce a better outcome?
- Identify patterns -- which act types lead to successful retries?
- Track act frequency -- how often does your agent need to retry?

### Tips for acts

- Not every agent needs acts. They're for agents that retry, iterate, or self-improve. Simple single-pass agents can skip them entirely.
- Put the "what changed" in the act opts. This makes it easy to see what the agent tried differently in each iteration.
- Keep act chains short. If your agent is retrying more than 3-5 times, the problem is likely in the prompt or approach, not the retry logic.
- Use act stats in the dashboard to identify which actions actually lead to improvement.

---

## Evaluation Strategies

WarpMetrics supports three strategies for evaluating agent output and recording outcomes:

### Deterministic

Unit tests, exact match. When tests pass, the outcome passes. Use this for structured outputs where correctness is binary.

### Heuristic

Programmatic checks. Length, format, keywords, regex, score thresholds. No "right answer" needed -- you define programmatic rules that check output quality.

### LLM-as-Judge

A second LLM scores the output. Use this for subjective quality assessment: tone, clarity, completeness, helpfulness. The judge LLM produces a score or verdict that you use to record the outcome.

---

## SDK Reference

The `@warpmetrics/warp` SDK instruments your LLM clients and sends telemetry to WarpMetrics.

### Installation

```bash
npm install @warpmetrics/warp
```

### Configuration

The SDK reads configuration from environment variables:

| Variable | Required | Description |
|---|---|---|
| `WARPMETRICS_API_KEY` | Yes | Your API key (starts with `wm_live_` or `wm_test_`) |
| `WARPMETRICS_API_URL` | No | API base URL. Default: `https://api.warpmetrics.com` |
| `WARPMETRICS_ENABLED` | No | Set to `false` to disable tracking. Default: `true` |

### warp

```
warp(client: OpenAI | Anthropic): ProxiedClient
```

Wraps an OpenAI or Anthropic client. Returns a proxied version that automatically tracks all API calls including tokens, cost, latency, and status.

```javascript
import { warp } from '@warpmetrics/warp';
import OpenAI from 'openai';
import Anthropic from '@anthropic-ai/sdk';

const openai = warp(new OpenAI());
const anthropic = warp(new Anthropic());
```

### run

```
run(label: string, opts?): Run
run(ref: Act, label: string, opts?): Run
```

Creates a new run. The label is used to categorize and group runs (e.g., "Code review", "Support agent"). When ref is an act, creates a follow-up run linked to that act. Returns a frozen run object.

```javascript
import { run } from '@warpmetrics/warp';

const r = run('Code review');

// Or as a follow-up to an act
const r2 = run(a, 'Code review');
```

### group

```
group(ref: Run | Group, label: string, opts?): Group
```

Creates a new group linked to a run or parent group. Groups organize related calls into phases (e.g., "Planning", "Execution"). The ref is required and auto-links the group.

```javascript
import { run, group } from '@warpmetrics/warp';

const r = run('Code review');
const g = group(r, 'Planning');
```

### call

```
call(ref: Run | Group, response: Response, opts?): void
```

Emits a tracked LLM call and links it to a run or group. Only responses passed to `call()` are sent to the API -- unclaimed responses are never transmitted.

```javascript
import { call } from '@warpmetrics/warp';

const res = await openai.chat.completions.create({...});
call(r, res); // Emit and link call to run

// Or link to a group
call(g, res);
```

### outcome

```
outcome(ref: Run | Group | Call, name: string, opts?): Outcome
```

Records an outcome for a run, group, or call. The name should be a human-readable label in Title Case (e.g., "Completed", "Failed", "Rate Limited"). Use classifications in the dashboard to map these to success/failure. The opts bag can carry arbitrary metadata.

```javascript
import { outcome } from '@warpmetrics/warp';

outcome(r, 'Completed', { reason: 'All checks passed' });
```

### act

```
act(ref: Outcome, name: string, opts?): Act
```

Records an action to take after an outcome. Use this to close the improvement loop -- declare a next step (retry, change prompt, switch model) and link it to a follow-up run.

```javascript
import { act, run } from '@warpmetrics/warp';

const o = outcome(r, 'Failed', { reason: 'timeout' });
const a = act(o, 'Retry');
const r2 = run(a, 'Code review'); // follow-up run
```

### ref

```
ref(target: Run | Group | Call | Response): string
```

Returns the WarpMetrics tracking ID for any tracked entity. Useful for logging or correlating with external systems.

```javascript
import { ref } from '@warpmetrics/warp';

console.log(ref(r));   // wm_run_01abc...
console.log(ref(res)); // wm_call_01abc...
```

### flush

```
flush(): Promise<void>
```

Manually flush all pending events to the API. Events are automatically batched and flushed, but you can call this to ensure delivery before process exit.

```javascript
import { flush } from '@warpmetrics/warp';

await flush(); // Ensure all events are sent
```

### Streaming support

The SDK automatically handles streaming responses. Token counts and latency are captured as the stream completes. Costs are calculated server-side. No extra code needed.

```javascript
const stream = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'Hello!' }],
  stream: true,
});

for await (const chunk of stream) {
  // Process chunks as usual
}

// Emit the tracked call after stream completes
call(r, stream);
```

---

## MCP Integration

WarpMetrics provides 17 MCP tools that let AI assistants and agents query your telemetry data programmatically.

### What is MCP?

The Model Context Protocol (MCP) is an open standard (https://modelcontextprotocol.io) that allows AI assistants to connect to external tools and data sources. With the WarpMetrics MCP server, agents can:

- Query runs, calls, and outcomes using natural language
- Check costs, latency, and success rates conversationally
- Get time series data and trend analysis
- Investigate individual call details and agent workflows

### Setup

#### 1. Get your API key

Create an API key from your account settings at https://warpmetrics.com/app/api-keys. This key allows the MCP server to access your WarpMetrics data.

#### 2. Install the MCP server

```bash
npm install -g @warpmetrics/mcp
```

#### 3. Configure Claude Desktop

Add the following to your Claude Desktop configuration file:

**macOS:** `~/.claude/claude_desktop_config.json`
**Windows:** `%APPDATA%\Claude\claude_desktop_config.json`

```json
{
  "mcpServers": {
    "warpmetrics": {
      "command": "warpmetrics-mcp",
      "env": {
        "WARPMETRICS_API_KEY": "wm_live_your_api_key_here"
      }
    }
  }
}
```

Replace `wm_live_your_api_key_here` with your actual API key.

#### 4. Restart Claude Desktop

Quit and reopen Claude Desktop. The WarpMetrics tools will now be available. You can verify by asking Claude "What WarpMetrics tools do you have access to?"

### All 17 MCP tools

**Stats (3 tools):**
- `get_stats` -- Retrieve summary statistics including totals, trends vs previous period, and previous period values. Optionally filtered by date range.
- `get_timeseries` -- Retrieve time series data with automatic hourly/daily resolution. Returns runs, calls, cost, latency, and success/failure per bucket.
- `get_global_counters` -- Retrieve all-time counters for the project: total runs, groups, calls, outcomes, tokens, cost, and latency.

**Runs (5 tools):**
- `list_runs` -- Retrieve a paginated list of runs with optional filtering by label and date range. Returns success rates, costs, and durations.
- `get_run` -- Retrieve a single run by ID with full details including groups, calls, outcomes, and totals.
- `get_run_timeline` -- Retrieve the execution timeline for a run, showing the sequence of calls and groups.
- `list_run_labels` -- Retrieve all distinct run labels for the project, optionally filtered by date range.
- `get_run_matrix` -- Retrieve a comparison matrix for runs with a given label, showing metrics across runs.

**Groups (2 tools):**
- `list_groups` -- Retrieve a paginated list of groups with optional filtering by label and date range.
- `get_group` -- Retrieve a single group by ID with full details including calls and outcomes.

**Calls (2 tools):**
- `list_calls` -- Retrieve a paginated list of LLM calls with optional filtering by date, model, status, and sort order.
- `get_call` -- Retrieve a single LLM call by ID with full details including messages, response, tools, and ancestry.

**Outcomes (3 tools):**
- `list_outcomes` -- Retrieve a paginated list of outcomes with optional filtering by name, classification, and date range.
- `get_outcome` -- Retrieve a single outcome by ID with full details including acts, provenance chain, target resolution, and classification.
- `get_outcome_stats` -- Retrieve aggregated outcome statistics grouped by name. Returns success/failure counts and rates per outcome name, with trend data over time.

**Acts (2 tools):**
- `list_acts` -- Retrieve a paginated list of acts with optional filtering by name and date range.
- `get_act` -- Retrieve a single act by ID with full details including the referenced outcome, target entity, and follow-up run.

### Example prompts

After setup, try asking your AI assistant:

- "How many runs did I have today?"
- "What's my total LLM spend this week?"
- "Show me the most expensive calls"
- "What's the success rate for my Code review agent?"
- "List recent failed runs"
- "Show me time series data for the last 7 days"
- "What happened in run wm_run_01abc...?"
- "Which model is most expensive?"

### Troubleshooting

**"WarpMetrics tools not available"** -- Make sure you've restarted Claude Desktop after editing the config file. Check that the config JSON is valid and the API key is correct.

**"Authentication failed"** -- Verify your API key is active in your API keys settings. Keys can be revoked or may have expired.

**"Command not found: warpmetrics-mcp"** -- Make sure the package is installed globally: `npm install -g @warpmetrics/mcp`

### Open source

The WarpMetrics MCP server is open source. View the code or report issues on npm: https://www.npmjs.com/package/@warpmetrics/mcp

---

## API Reference

The WarpMetrics API gives you programmatic access to your AI agent telemetry. Query runs, calls, outcomes, and statistics. Build dashboards, automations, or integrate with your CI pipeline.

**Base URL:** `https://api.warpmetrics.com`

### Authentication

All API requests require authentication via API key. Include your key in the Authorization header:

```bash
curl "https://api.warpmetrics.com/v1/runs" \
  -H "Authorization: Bearer wm_live_your_api_key"
```

Create and manage API keys from your API keys page at https://warpmetrics.com/app/api-keys.

### Rate limiting

API requests are limited to **200 requests per minute**. Monitor your usage via response headers:

| Header | Description |
|---|---|
| `X-RateLimit-Limit` | Maximum requests per window |
| `X-RateLimit-Remaining` | Requests remaining in current window |
| `X-RateLimit-Reset` | Window reset timestamp |

When rate limited, the API returns:

```json
{
  "success": false,
  "error": {
    "message": "Rate limit exceeded",
    "code": "RATE_LIMITED"
  }
}
```

### OpenAPI specification

The full OpenAPI specification with all endpoints, parameters, request/response schemas, and curl examples is available at:

    https://api.warpmetrics.com/v1/docs/openapi.json

### Endpoint categories

The API is organized into these categories:

- **Stats** -- summary statistics, time series, global counters
- **Runs** -- list, get, timeline, labels, matrix
- **Groups** -- list, get
- **Calls** -- list, get (with full message/response detail)
- **Outcomes** -- list, get, aggregated stats
- **Acts** -- list, get
- **Ingest** -- SDK event ingestion (used by the SDK, not typically called directly)

---

## Pricing

### Free -- $0/month

For individual developers getting started.

- 10,000 calls/month
- 2 projects
- 1 team member
- Runs, groups, calls tracking
- Real-time dashboard
- 7 days data retention
- Community support
- Does NOT include: outcome tracking, advanced stats, data export

### Pro -- $49/month (most popular)

For teams building production AI agents.

- 500,000 calls/month
- 20 projects
- 10 team members (extra seats $10/mo each)
- Everything in Free
- Outcome tracking
- Advanced stats
- Data export
- 90 days data retention
- Email support

### Enterprise -- Custom pricing

For organizations with advanced needs.

- 2,000,000+ calls/month
- 100+ private projects
- Unlimited team members
- Everything in Pro
- Priority support
- Custom integrations
- SLA guarantee
- Unlimited data retention

### Plan comparison

| Feature | Free | Pro | Enterprise |
|---|---|---|---|
| API calls per month | 10,000 | 500,000 | 2,000,000+ |
| Projects | 2 | 20 | 100+ |
| Data retention | 7 days | 90 days | Unlimited |
| Team members | 1 | 10 | Unlimited |
| Runs, groups, calls tracking | Yes | Yes | Yes |
| Real-time dashboard | Yes | Yes | Yes |
| Outcome tracking | No | Yes | Yes |
| Advanced stats | No | Yes | Yes |
| Data export | No | Yes | Yes |
| Custom integrations | No | No | Yes |
| Community support | Yes | Yes | Yes |
| Email support | No | Yes | Yes |
| Priority support | No | No | Yes |
| SLA guarantee | No | No | Yes |

### FAQ

**What happens if I exceed my monthly call limit?**
On the Free plan, additional calls are dropped once you hit the limit. On Pro and Enterprise, overage calls are billed at a per-call rate so your agents never stop working.

**Can I upgrade or downgrade at any time?**
Yes. Upgrades take effect immediately and you only pay the prorated difference. Downgrades apply at the start of your next billing cycle.

**How long is my data retained?**
Free plans retain data for 7 days, Pro for 90 days, and Enterprise plans can retain data indefinitely. You can export your data at any time on paid plans.

**How does team billing work?**
Each plan includes a set number of team members. On Pro you can add extra seats for $10/mo each. Enterprise plans include unlimited seats.

**Is the SDK open source?**
Yes. The @warpmetrics/warp SDK is MIT licensed and fully open source. The hosted dashboard and API are proprietary, but you always own your data.

**Do you offer discounts for startups or open source projects?**
Yes. 50% off Pro for early-stage startups and free Pro for qualifying open source projects. Contact us to apply.

---

## About WarpMetrics

WarpMetrics is an AI-first company. The platform -- from backend to frontend -- is built by LLMs. Founded by Nikolai Onken (@nonken on X).

**Mission:** Give AI agents deep visibility into their own runs, calls, outcomes, and performance -- so they can improve themselves.

**Contact:** https://warpmetrics.com/contact
**Sales:** sales@warpmetrics.com