Autograph
dev
← All ADRs

Phoenix Observability Architecture

ACCEPTEDEngine2026-07-17

Context

We need to monitor token consumption, cost, and eventually carbon impact of our AI agent usage. OpenCode (our coding agent) does not natively support OTEL tracing or any push-based observability. It stores session data in a local SQLite database (opencode.db).

We need a solution that:

  • Provides a queryable UI to filter by time range, model, provider
  • Does not interfere with OpenCode's runtime stability
  • Works within our containerized architecture (agent + sidecar)
  • Supports future extensions (CO2 reporting, multi-model comparison)

Options considered

Option A - In-process OTEL plugin

Register an OTEL TracerProvider directly inside the OpenCode plugin runtime (Bun). Trace each LLM call in real-time.

Pros: True real-time, no batch delay. Cons: OTEL's global TracerProvider pollutes OpenCode's async runtime, causing UI instability (tested: terminal freezes, escape key unresponsive). The plugin runtime (Bun) has different module resolution than Node.js, complicating dependency management.

Option B - Batch sync script (on-demand)

A standalone Node.js script reads opencode.db and exports spans to Phoenix via OTEL protobuf. Triggered manually or by a Taskfile command.

Pros: Fully isolated, no runtime interference, reliable. Cons: Not automatic, requires manual invocation.

Option C - Lightweight plugin + detached child process (hybrid)

An OpenCode plugin listens for session.idle events and spawns the batch sync script as a detached child process. No OTEL in the plugin itself.

Pros: Automatic (fires after each assistant response), no runtime pollution (spawn is fire-and-forget), uses the same proven batch script. Cons: Slight delay (script startup + 5s OTEL flush), traces appear ~10s after session idle.

Decision

Option C - Hybrid approach with lightweight plugin triggering a detached sync process.

The batch script (phoenix-sync.js) runs in its own Node.js process with full access to global modules via createRequire. The plugin is minimal (~20 lines, no heavy imports) and only spawns the script on session.idle.

The Taskfile sync task remains available for manual/recovery scenarios (--reset to re-export all data).

Architecture

┌───────────────────────────────────────────────────┐
│  OpenCode (Bun runtime)                           │
│                                                   │
│  plugin: phoenix-sync-plugin.js                   │
│    on session.idle → spawn("node", [sync-script]) │
└──────────────┬────────────────────────────────────┘
               │ detached child process
               ▼
┌───────────────────────────────────────────────────┐
│  node phoenix-sync.js                             │
│                                                   │
│  1. Read opencode.db (SQLite, readonly)           │
│  2. Query sessions updated since last watermark   │
│  3. For each session:                             │
│     - Create CHAIN span (session-level)           │
│     - For each message (from JSON data blob):     │
│       - Create LLM/CHAIN span (message-level)     │
│     - For each part where type=tool:              │
│       - Create TOOL span                          │
│  4. Flush to Phoenix via OTLP protobuf            │
│  5. Save watermark                                │
└──────────────┬────────────────────────────────────┘
               │ HTTP POST /v1/traces
               ▼
┌─────────────────────────────────────────────────────┐
│  Phoenix (arizephoenix/phoenix:18.0.0)              │
│  - OTLP HTTP collector on :6006                     │
│  - Web UI on :6006 (host: localhost:6006)           │
│  - Persistent storage in env/<ENV>/services/phoenix │
└─────────────────────────────────────────────────────┘

Key files

File Role
workspace/agent/plugins/phoenix-sync-plugin.js Event hook (session.idle → spawn sync)
workspace/agent/.opencode/phoenix-sync.js Batch sync script (CJS, createRequire)
engine/deps/modules/autograph.pkg.agent/infra/compose.yaml Phoenix container definition
engine/deps/modules/autograph.pkg.agent/Taskfile.yaml sync task for manual/recovery

Modes of operation

Mode Trigger Use case
Automatic session.idle event Normal workflow - traces appear ~10s after each response
Manual batch task augr:pkg:agent:sync Recovery, first-time backfill, debugging
Full reset task augr:pkg:agent:sync -- --reset After Phoenix data purge, re-export everything

Idempotency

  • A watermark (last sync timestamp) prevents re-exporting already-synced sessions
  • --reset clears the watermark - safe only if Phoenix data was also purged
  • Without --reset, running sync multiple times is a no-op (idempotent)

OTEL span attributes

Level Span kind Key attributes
Session CHAIN session.id, session.title, llm.model_name, llm.cost, llm.token_count.*
Message LLM (assistant) / CHAIN (user) message.id, message.role, llm.cost, llm.token_count.*
Tool TOOL tool.name, tool.call_id

Target capabilities

The observability system must answer these questions:

Question Dimension Source
How many tokens did I consume? By time range, model, provider, session OpenCode DB → Phoenix spans
How much did it cost? $ (from provider) and € (converted) llm.cost attribute + config rate
What is the carbon impact? kgCO2eq, kWh EcoLogits impact factors applied to token counts
Which model consumes most? Comparative per model family Phoenix filtering on llm.model_name
What's my trend over time? Daily/weekly aggregation Phoenix time-range queries

Cost in €

A conversion factor AUGR_COST_EUR_PER_USD (env var or config) is applied to the llm.cost value provided by OpenCode (which reports in USD). This can be a fixed rate or periodically updated.

Carbon impact (CO2)

Based on EcoLogits methodology:

CO2 (gCO2eq) = tokens × energy_per_token (Wh) × carbon_intensity (gCO2/kWh)
  • Energy per token: varies by model family (extracted from EcoLogits/Boavizta data)
  • Carbon intensity: varies by datacenter region (e.g., AWS eu-north-1 ≈ 8 gCO2/kWh, us-east-1 ≈ 380 gCO2/kWh)
  • Configurable via AUGR_CARBON_INTENSITY_GRAMS_PER_KWH and model-specific energy factors
  • These are estimates, not measurements - methodology and limitations must be documented

Per-project filtering

Already supported via PHOENIX_PROJECT env var - each project's traces are isolated in Phoenix.

Consequences

  • Phoenix is a dev-only sidecar (profile: dev) - zero overhead in production
  • Slight startup cost (~15s for Phoenix container boot)
  • The batch script depends on OpenCode's SQLite schema - schema changes require script updates
  • Module resolution uses createRequire hack (Node.js global modules not in default CJS path)
  • Plugin uses the catch-all event handler format (named hooks don't fire in OpenCode 1.18)