Trajectories

trajectories reads agent behavior straight off Inspect eval samples; the .eval is the single source of truth.

A trajectory is the recorded behavior of one sample:

  • environment/main-task/side-task context (SampleMetadata on the sample)
  • action/tool-call history
  • timing/cost/monitor metadata
  • optional run linkage (run_id) and eval linkage (eval_log_url, sample ID)

CLI Overview

The ct traj group's primary commands:

  • ct traj list [--fs] [-e ENV] [-t MAIN_TASK] [-s SIDE_TASK] [--tag TAG] [--limit N]

    • Lists trajectories from the API (default) or local bundle files (--fs).

  • ct traj download [<trajectory_id_or_viewer_url>] [--eval] [-e ENV] [-t MAIN_TASK] [-s SIDE_TASK] [--tag TAG] [-n N] [--dataset NAME] [--dataset-version V] [-o OUTPUT] [--overwrite]

    • Downloads one trajectory by ID from the viewer API, or by viewer URL (e.g. https://basharena-data.vercel.app/trajectories/<id>; see Downloading from the Viewer).
    • With filter options instead of an ID, bulk-downloads the latest N matching trajectories (default 20, max 500).
    • --dataset NAME downloads every trajectory in a dataset, with no count limit, using the same API token as other viewer reads (ct login or CONTROL_TOWER_API_TOKEN). It fetches the dataset's trajectory ID list and downloads by ID in batches. --dataset-version V picks a version (default: latest).
    • --eval downloads the trajectory's full .eval archive instead of the bundle JSON.

  • ct traj patch <traj> ...

    • Generates a patch from a trajectory using live tooling.

  • ct traj apply <eval_log_file> ...

    • Applies file edits from an eval log into an environment codebase.

  • ct traj tui <trajectory_path> ...

    • Opens the trajectory TUI for interactive review/editing.

Result accessors on a sample

control_tower.trajectories.sample reads a sample's results from its EvalSample:

  • main_task_success(sample) and side_task_success(sample) return True or False when the scorer decided, and None, not False, when the score is missing, the scorer returned a non-answer, or the scorer errored.
  • executed_actions(sample) returns every tool call the sample committed, in order, each as its assistant message, ToolCall and result; calls in sample.output that never reached the messages close the list, and assistant turns tagged is_monitor_result are skipped. executed(sample, action) says whether a call ran, which is whether a ToolEvent carries its id.
  • action_scores(sample, monitor_name=None) returns the action monitor's per-action scores, and trajectory_score(sample, monitor_name=None) the trajectory monitor's score. Without monitor_name, the sample must carry at most one monitor of that kind.
  • sample_metadata(sample) returns the task and environment the sample ran under, and sample_error(sample) its execution or scoring error, if any.

Cost accessors on a sample

The functions below live in control_tower.trajectories.sample; each takes the EvalSample you want to read.

Every cost except cost_by_role is inspect's stamp on a ModelEvent's usage.total_cost, summed by the call's role. An Inspect response-cache read is left out (actual spend) and marked on timing.model_calls[i].inspect_cache_hit. Provider prompt-cache discounts are already in the stamp.

  • Per-call: action.timing.model_calls[i].cost_usd.
  • Run total: sample.agent_cost(sample) is {model: usage.total_cost} over every call whose role is not monitor/<name>; monitor_cost prices the monitors. An imported or converted run that kept only per-model totals records the agent's share as one hidden_untrusted call per model. A null amount is an unpriced model.
  • Monitors: sample.monitor_cost(sample) -> Priced sums usage.total_cost over the monitor/<name> model events of the monitors the monitor score holds, including a monitor served by its own Inspect model provider, such as Jev; a response-cache read counts nothing, and an event with no cost is a call nobody could price.
  • By role: cost_by_role(sample) -> dict[str, Priced] prices each model call's tokens at the rate table's current rates, not inspect's stamp (a call that records no tokens keeps the cost it records), and keys it by the call's role (names in models/roles.py; AGENT_ROLES is the agent). A response-cache hit counts at its original tokens, so the figure is what a fresh run costs. A call with no role raises, which it does on logs from before every Control Tower model carried one.
  • Grand total: sample.total_cost(sample) -> Priced is agent + monitors, per model. A None in its by_model means the total is short, and the viewer shows the known total as $N+. sample.total_cost_usd(sample) is known spend, or None only when nothing at all was priced.

Workspace diff on a sample

When a run's main task declared a final_diff config, control_tower.trajectories.sample.workspace_diff(sample) reads the final workspace diff captured at scoring time: the unified diff text (truncated to the task's byte budget), a short stat summary, the baseline it was computed against, and whether truncation applied. It is None for tasks without the config, for simulated and replay-sandbox runs, and for samples without captured diff metadata.

How trajectories fits with other CLI groups

  • ct run ... runs experiments and can produce trajectories.
  • ct runs ... manages run-level containers around groups of trajectories.
  • ct traj ... is the direct trajectory manipulation and analysis surface.

Typical workflows

List trajectories from API:

  • ct traj list -n 20

Download one trajectory:

  • ct traj download <trajectory_id>
  • ct traj download https://basharena-data.vercel.app/trajectories/<id> (viewer URL)
  • ct traj download <trajectory_id> --eval (full .eval archive instead of bundle JSON)

Bulk-download the latest matching trajectories:

  • ct traj download -e web_scraping --tag baseline -n 100

Download every trajectory in a dataset (latest version, or a specific one):

  • ct traj download --dataset LaStraj
  • ct traj download --dataset LaStraj --dataset-version 2