Monitor verification outcomes¶

This guide is for whoever runs a service that checks manifests and needs to know it is working. You count the verdicts it returns, the errors it hits, and how long each check takes, using the Prometheus monitoring system, and you learn which of those numbers deserve an alert.

Keep three things apart: verdicts, unexpected errors, and time taken. A healthy verifier rejects bad manifests, so a rising share of VALID results is not a goal in itself.

Run a metrics example¶

Complete the first-manifest example, install prometheus-client with python -m pip install prometheus-client, then append this block to first_manifest.py and run it again. It records one accepted result, one mismatch, and one deliberate exception.

import time
from prometheus_client import CollectorRegistry, Counter, Histogram, generate_latest

def make_verifier_metrics(operation=verify_manifest):
    registry = CollectorRegistry()
    outcomes = Counter(
        "agent_manifest_verifications_total", "Verification attempts by outcome",
        ["result"], registry=registry,
    )
    latency = Histogram(
        "agent_manifest_verification_duration_seconds", "Time inside the verifier",
        buckets=[0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0],
        registry=registry,
    )
    def measured_verify(received, approved_context, revocations):
        started = time.perf_counter()
        outcome = "ERROR"
        try:
            result = operation(received, approved_context, revocations)
            outcome = result.result.value
            return result
        finally:
            outcomes.labels(result=outcome).inc()
            latency.observe(time.perf_counter() - started)
    return registry, measured_verify

registry, measured_verify = make_verifier_metrics()
assert measured_verify(record, context, RevocationStore()).result.value == "VALID"
assert measured_verify(record, drift, RevocationStore()).result.value == "MISMATCH"
assert registry.get_sample_value("agent_manifest_verifications_total", {"result": "VALID"}) == 1
assert registry.get_sample_value("agent_manifest_verifications_total", {"result": "MISMATCH"}) == 1
assert registry.get_sample_value("agent_manifest_verification_duration_seconds_count") == 2
print("PASS: accepted and rejected verdicts are counted separately")

def unavailable_verifier(*args):
    raise RuntimeError("synthetic verifier failure")

error_registry, failing_verify = make_verifier_metrics(unavailable_verifier)
try:
    failing_verify(record, context, RevocationStore())
except RuntimeError:
    pass
else:
    raise AssertionError("Metrics wrapper swallowed the exception")
assert error_registry.get_sample_value("agent_manifest_verifications_total", {"result": "ERROR"}) == 1
assert error_registry.get_sample_value("agent_manifest_verification_duration_seconds_count") == 1
print("PASS: unexpected error counted and propagated")
print(generate_latest(registry).decode("utf-8"))

The timing histogram covers the wrapped verifier call, including exceptions. It excludes HTTP transport, queueing, and any evidence fetching performed before the call. ERROR is this wrapper's exception label, not an SDK verdict. The counter resets when its process restarts.

Connect it to the service¶

To collect the same numbers from the verification service, wrap its verify call and add a /metrics route.

Query what was measured¶

These queries, written in PromQL (the Prometheus query language), turn the raw counts into rates and timings. Configure Prometheus to scrape the service under a job named manifest-verifier, or replace that job selector in these queries.

Choose alerts from service policy¶

Set alert levels from your own traffic, and read each spike for what it actually counts. Select thresholds and evaluation windows from measured traffic and the service's objectives. This guide supplies no measured latency, uptime, or revocation-propagation guarantee.

• A MISMATCH spike means more failed comparisons or signatures. Examine details before concluding tampering or replay.
• A REVOKED spike counts verification attempts against revoked IDs. Repeated requests for one ID can cause it; it does not count new revocations.
• INCOMPLETE commonly means a declared artifact lacks the required independent runtime observation. Inspect the actual result fields rather than assuming missing human approval.
• Higher latency does not identify a slow CRL or transparency service. Instrument those operations separately if the application calls them; the demo uses an in-memory revocation store.
• Track revocation freshness and propagation at the distribution/refresh boundary. Neither of the two metrics above measures them.

Separate availability and processing-error objectives from authorization outcomes. Rejecting a bad record is expected behavior, and a high acceptance rate can hide a verifier that accepts too much.

Other telemetry backends¶

The same approach works with other monitoring tools. With OpenTelemetry, instrument the same call boundary and preserve verdict/exception distinctions. Configure a metric reader and exporter in the hosting application; creating a meter and instruments alone does not deliver metrics to a backend. Use your deployment's existing telemetry configuration and test delivery with an intentional request and failure before relying on alerts.