AI agent observability is the ability to reconstruct what an agent tried to do, which evidence it used, which tools it called, where policy intervened, and how the run ended. A stream of log lines helps, yet it rarely answers the questions an operator asks after something goes wrong.
The useful unit is an action with lineage. It begins with a task and an owner. It passes through context assembly, planning, tool execution, policy checks, and review. The record should connect those stages without forcing a person to search several unrelated stores.
This matters because agents can fail while looking busy. A scheduler can return success even though its feed is empty. A tool can complete while writing stale data. A run can produce a polished result with no link to the source task. Observability has to show operational meaning, not activity alone.
What AI agent observability should answer
An operator usually starts with a plain question: what happened? The answer should include the triggering request, the scope, the chosen execution path, the evidence loaded, the tools called, the policy decisions, the final artifact, and any caveat that remains open.
From there, the system should support sharper questions. Why did the agent choose this action? Which source supports the customer-facing claim? Did a human approve the irreversible step? Was the result reviewed independently? Did the failure occur during planning, tool use, delivery, or state persistence?
These questions require connected events. A timestamped message such as "tool completed" has little value without a task identifier and a result reference. A result file has limited value when no record ties it back to the request. Observability turns separate artifacts into a navigable chain.
The chain should stay readable by humans. Dense traces are useful for debugging, while an operator also needs a concise view of the same run. Both views should point to the same underlying events.
Start with a stable identity for the run
Every task needs a stable identifier that follows it through delegation, retries, and review. Child work should preserve the parent relationship. Tool calls should carry the same lineage. Results should name the task they satisfy.
Stable identity prevents a common form of operational amnesia. A system may have a queue entry, several logs, and a file on disk, yet nobody can prove they belong together. When the identifiers align, a reviewer can move from request to evidence and from evidence to outcome.
Retries deserve their own attempt identity beneath the run. Otherwise a successful retry can overwrite the evidence of the earlier failure. The operator needs to see what changed between attempts, which context was reused, and whether the retry began from a clean state.
The historical reliability record behind Gevurah makes this distinction explicit. 833 of 839 measured tasks ran once only, while 6 received a second attempt that already carried information from the first. That record cannot support a clean repeated-attempt reliability score. The observability lesson is direct: attempt lineage changes what the data can prove.
Record decisions alongside actions
Tool traces reveal what code executed. They do not explain why the action was allowed. An observable agent system records policy decisions as first-class events, including the rule evaluated, the evidence presented, the verdict, and the reason.
Gevurah's frozen record contains 29 blocked decisions out of 137 policy decisions. A block is useful evidence. It shows that the control path ran and refused an action under a stated rule. Hiding blocked attempts would make the system appear smoother while removing the proof that restraint works.
Allow decisions matter too. They should state which rule granted the path and whether approval was attached. When the action involves money, publishing, or a customer message, the record should show the human checkpoint before execution.
Unknown state needs a distinct verdict. If the policy check cannot reach its source, the event should say unknown and close the path. Reading an empty response as permission destroys the value of the gate.
Trace context and evidence provenance
An agent's output depends on what it saw. Observability should capture the context sources that influenced the run without copying sensitive material into every log. Store references, versions, retrieval times, and scope labels. Keep the protected content in its owner system.
For public claims, the trace should connect each measured statement to the frozen source that approved it. For live relationship state, it should name the live check used at decision time. For a memory record, it should preserve the record identifier and version.
This makes post-run review possible. A person can decide whether the source was authoritative, current, and within scope. If a result later proves wrong, the team can tell whether the problem began with retrieval, reasoning, policy, or execution.
Evidence provenance also protects good work. When an agent reaches the right conclusion, the trace shows how. That path can become a test case or an operating pattern instead of disappearing inside a transcript.
Separate activity, availability, and reliability
These three categories are easy to mix. Activity measures that something ran. Availability measures whether a component responded. Reliability asks whether the system completed the intended work correctly under a defined test.
The distinction changes how dashboards should read. Gevurah's source records 484 uptime checks per day. Those checks describe availability. They do not establish agent reliability. The scheduler record also includes tasks that return a non-zero exit code, while an exit code of zero only shows that a script did not crash.
Activity volume can still be useful. The evidence record contains 7,321 outbound actions logged over 92 days. That proves the ledger was receiving real action events during the measured window. It does not prove that every action was correct, complete, or independently reviewed.
Label each metric with its class. Operators should never have to infer whether a green number means traffic, uptime, completion, or verified quality.
Use denominators that survive scrutiny
An observability system becomes misleading when it shows a percentage without the population behind it. Missing results, unlabeled outcomes, and self-reported success all affect the denominator.
The delegated work record illustrates the issue. Between 44.5% and 63.4% of delegated work orders completed cleanly on the first attempt. The upper end counts 532 clean results out of 839 tasks with a linked result. The lower end treats 356 work orders with no result record as failures against all 1,195 work orders.
The range has to stay intact. Publishing the upper edge alone would hide the missing-result problem. The outcome field also comes from the executing agent about its own run, which makes it self-assessment rather than independent review.
Good observability keeps the denominator next to the metric. It shows missing data as a category. It exposes who assigned the outcome label. A reviewer can then decide what the number supports.
Make missing results visible
A run without a linked result is an operational event. It may indicate a failed task, a broken handoff, an orphaned artifact, or a state-writing problem. Treating it as an empty space removes the most valuable clue.
Create explicit terminal states for completed, partial, failed, blocked, and missing result. Do not collapse them into success and error. A blocked action may show healthy policy behavior. A partial result may still contain useful work and a clear caveat. A missing result points to broken durability even when execution logs look normal.
The dashboard should let an operator open the task, its attempts, and any nearby files. If an artifact exists without a task link, show it as orphaned. If a task claims success without an artifact, show the contract mismatch.
This is one place where strict build rules help. Requiring every result to name the task it resolves turns observability into a structural property instead of a reporting convention.
Design events for delivery failures
Detection and delivery are separate parts of an alerting system. A monitor may compute the right condition and write it to a file that nobody reads. The detector is healthy, while the operating outcome is failure.
Observability should trace the alert from condition to emission and from emission to delivery. Record the stable event identity, severity, deduplication decision, delivery channel, and final state. If a rate cap suppresses alerts, expose that fact rather than silently dropping them.
Stable identity matters here. A queue depth that changes from one value to another is still the same continuing incident. Deduplicating by the changing value creates a fresh alert each time and teaches people to mute the channel.
The useful heartbeat says whether an event was emitted or paged. A healthy process exit cannot substitute for that evidence.
Build dashboards around decisions
An operations dashboard should help a person decide what to inspect next. Start with runs that lack results, policy blocks that need review, repeated tool failures, stale evidence, and delivery events that never reached their destination.
Summary counts belong at the top, followed by drill-down paths. Every count should open the population behind it. If the dashboard shows blocked decisions, the operator should be able to see the reasons. If it shows completion, the denominator and outcome owner should be visible.
Avoid one universal health score. It compresses different failure classes into a number that looks precise and explains little. A small set of clearly named panels gives a better operating picture: task flow, policy, tool health, evidence quality, and alert delivery.
Time windows must be explicit. The Gevurah evidence page keeps measurement windows attached to its figures. The same practice belongs in internal observability. A count without a window cannot be compared or reproduced.
Protect privacy inside the trace
Full prompts and tool payloads can contain customer data, private notes, credentials, or material from another scope. Logging everything creates a second data store with weaker ownership.
Prefer references and structured summaries. Record that a customer record was consulted, along with its owner and version, while leaving the content in the customer system. Redact secrets before persistence. Keep access controls aligned with the source domain.
Debug mode still needs boundaries. Temporary expanded traces should have a clear owner, a short retention rule, and an audit record. They should never become the default because they are convenient during development.
Privacy-aware tracing improves analysis too. Structured event fields are easier to query than giant prompt dumps. The operator sees the action, source class, and verdict without reading unrelated content.
Turn incidents into evaluation cases
The strongest observability systems feed evaluation. When a run fails because context was stale, convert that sequence into a test where the current source must win. When a policy check was skipped, create a negative case that verifies the path stays closed. When delivery failed after detection, test the entire chain.
Keep the original event references with the case. Future reviewers can see why the test exists and which incident it prevents. This gives the evaluation suite operational memory.
Retries should start from a defined state. The evidence source describes a harness that starts each attempt from a clean workspace, enforces a negative control, and requires explicit reversibility approval. That design prevents information from an earlier attempt from quietly improving the next one.
Evaluation results belong beside production evidence with clear labels. One describes controlled trials. The other describes historical operation. Combining them would weaken both.
An implementation sequence that stays useful
Begin with a shared run identifier and a strict result contract. Connect task, attempt, tool call, policy decision, artifact, and review. Add timestamps and scopes. Make missing links fail validation where possible.
Next, define a small event vocabulary. Include requested, started, context assembled, tool called, policy evaluated, result written, review completed, and delivery confirmed. Add failure states at each stage. Keep event payloads structured and privacy-aware.
Then build the operator view around open decisions. Show missing artifacts, blocked actions, stale context, repeated failures, and undelivered alerts. Attach denominators and windows to every metric.
Finally, review traces as part of normal work. Pick a successful run and prove the evidence chain. Pick a failure and identify the first broken link. Pick a blocked action and confirm that the policy reason is understandable.
AI agent observability earns its place when it shortens the path from "something feels wrong" to a specific, inspectable event. For the public measurement record and its caveats, visit the Gevurah evidence page. It shows the standard that internal traces should support: figures with windows, denominators, and limits attached.