railway
The same facts as the comparison below, in sentences. Nothing here depends on reading the table.
The approved route beside the route this release actually took, step by step, in canonical order.
| Step | Service | Approved | This release | Change | Side effect | Tool |
|---|---|---|---|---|---|---|
| refund.request | flightrules-demo-agent | 1 | 1 | IN BOTH | none | — |
| refund.calculate | flightrules-demo-agent | 1 | absent | REMOVED | none | calculate_refund |
| customer.notify | flightrules-demo-agent | 1 | 1 | IN BOTH | external | notify_customer |
| customer.notify.handler | flightrules-notification-service | 1 | 1 | IN BOTH | external | — |
| fraud.check | flightrules-demo-agent | 1 | absent | REMOVED | read | check_fraud |
| fraud.check.handler | flightrules-fraud-service | 1 | absent | REMOVED | read | — |
| order.lookup | flightrules-demo-agent | 1 | 1 | IN BOTH | read | lookup_order |
| order.lookup.handler | flightrules-order-service | 1 | 1 | IN BOTH | read | — |
| policy.retrieve | flightrules-demo-agent | 1 | absent | REMOVED | read | retrieve_policy |
| policy.retrieve.handler | flightrules-policy-service | 1 | absent | REMOVED | read | — |
| payment.refund | flightrules-demo-agent | 1 | 2 | REPEATED | write | issue_refund |
| payment.refund.handler | flightrules-payment-service | 1 | 1 | IN BOTH | write | — |
approved 43070aa4af4f6c2c912a8d7bcc724f1d199e0425dc8ad7256b528eec195cb037 · observed 22ffa0c0e578ef70a32a34aae7830f40aeef027aae28e66006601e80f99c7466
Produced by the deterministic graph comparison in the FlightRules engine, not by comparing text in the browser.
| Change | Subject | Detail | Approved → observed |
|---|---|---|---|
| CARDINALITY CHANGED | payment.refund | payment.refund occurs 2 times in the candidate against 1 in the baseline. | 1 → 2 |
| MISSING EDGE | fraud.check->fraud.check.handler|parent | The baseline relationship fraud.check->fraud.check.handler|parent is absent from the candidate. | — |
| MISSING EDGE | policy.retrieve->policy.retrieve.handler|parent | The baseline relationship policy.retrieve->policy.retrieve.handler|parent is absent from the candidate. | — |
| MISSING EDGE | refund.request->fraud.check|parent | The baseline relationship refund.request->fraud.check|parent is absent from the candidate. | — |
| MISSING EDGE | refund.request->policy.retrieve|parent | The baseline relationship refund.request->policy.retrieve|parent is absent from the candidate. | — |
| MISSING EDGE | refund.request->refund.calculate|parent | The baseline relationship refund.request->refund.calculate|parent is absent from the candidate. | — |
| REMOVED STEP | fraud.check | fraud.check is present in the baseline but absent from the candidate. | 1 → 0 |
| REMOVED STEP | fraud.check.handler | fraud.check.handler is present in the baseline but absent from the candidate. | 1 → 0 |
| REMOVED STEP | policy.retrieve | policy.retrieve is present in the baseline but absent from the candidate. | 1 → 0 |
| REMOVED STEP | policy.retrieve.handler | policy.retrieve.handler is present in the baseline but absent from the candidate. | 1 → 0 |
| REMOVED STEP | refund.calculate | refund.calculate is present in the baseline but absent from the candidate. | 1 → 0 |
| RETRY INCREASE | payment.refund | payment.refund retried up to attempt 1 against 0 in the baseline. | 0 → 1 |
| RETRY INCREASE | payment.refund.handler | payment.refund.handler retried up to attempt 1 against 0 in the baseline. | 0 → 1 |
| ROUTE NOT APPROVED | 22ffa0c0e578ef70a32a34aae7830f40aeef027aae28e66006601e80f99c7466 | The candidate route fingerprint matches none of the 1 approved baseline families. | — |
| DUPLICATE SIDE EFFECT | payment.refund|write | payment.refund performed a write side effect 2 times in the candidate against 1 in the baseline. | 1 → 2 |
| Threshold | Permitted | Observed |
|---|---|---|
| Violation rate | 0.5% | 100.00% |
| Unknown route rate | 1% | 100.00% |
| Latency change | 20% | +740.74% |
| Token change | 25% | not measured |
| Duplicate side effects | reported | 100.00% |
| Missing prerequisites | reported | 100.00% |
| Finding | Implies | Summary | Expected | Observed |
|---|---|---|---|---|
| ZERO TOLERANCE VIOLATION | fail | 24 zero-tolerance violation(s) were observed. A zero-tolerance rule has no permitted rate. | 0 zero-tolerance violation(s) | 24 violation(s) |
| VIOLATION RATE EXCEEDED | fail | 100.000000% of runs violated the contract; the gate permits at most 0.5%. | at most 0.5% | 100.000000% (8 of 8) |
| UNKNOWN ROUTE RATE EXCEEDED | fail | 100.000000% of runs followed a route the contract does not approve; the gate permits at most 1%. | at most 1% | 100.000000% (8 of 8) |
| LATENCY REGRESSION EXCEEDED | fail | p95 latency moved 740.740740% against the baseline; the gate permits at most 20%. | at most 20% | 740.740740% (189 ms to 1589 ms) |
| RELEASE BUDGET EXCEEDED | fail | Release-wide p95 of run.duration_ms was 1589, over the budget of 227. | at most 227 | 1589 |
| MIN RUNS NOT MET | insufficient_data | 8 completed run(s) were evaluated; the contract requires at least 20 before a release decision. | at least 20 run(s) | 8 run(s) |
| Disclosure | Detail |
|---|---|
| RETRY CHANGE NOT MEASURED | No usable baseline value for agent.retry.count, so its change could not be measured. |
| TOKEN REGRESSION NOT MEASURED | No usable baseline value for gen_ai.usage.total_tokens, so its change could not be measured. |
Representative runs this decision was taken over. Each SigNoz link is the URL SigNoz itself returned for that trace when FlightRules retrieved it, not a pattern this page assembled.
| Trace | Outcome | Route | Similarity | In SigNoz |
|---|---|---|---|---|
| 1860df10f915cf6880631ec2206e18b2 | FAILING | not approved | 0.619047 | Open trace |
| 190770545baf3e9ddf3d7f3b80cbf608 | FAILING | not approved | 0.619047 | Open trace |
| 262049ed4b847a968d8209c0b396b2eb | FAILING | not approved | 0.619047 | Open trace |
| 402d65a6c9e07b4a2bd0dd44624d7100 | FAILING | not approved | 0.619047 | Open trace |
| 7eab676eb948ebe9bcb706f697413725 | FAILING | not approved | 0.619047 | Open trace |
Re-running the evaluation creates a new evaluation and a job. The decision above is read from the most recent completed evaluation, so it does not change until the new one finishes — and the evidence behind it is preserved either way.