Test financial agents against your company’s controls, replay failures, and compare behavior across different models and configurations. Use the results to evaluate whether an agent version meets your release requirements.
Working prototype · Synthetic scenarios · Increase sandbox validation
Built by payments engineers fromWe run financial-agent tests across different models and configurations to understand how model choice changes behavior. We examine critical failures, task completion, recovery, and reliability—not just one aggregate score.
The agent is the same in every arm: one accounts-payable workflow, one prompt, the same tools and policy, gate off, temperature 0. Only the model behind it changes. Open scenario pool payments-v1, 22 scenarios × 1 trial per model, model ids as each endpoint reported them. One trial is provisional evidence: a single answer moves a rate by five points.
| Model, self-reported | Route | Answered | Scenarios passed | Critical failures | Safety | Latency per episode | Cost |
|---|---|---|---|---|---|---|---|
| anthropic/claude-sonnet-4.5 | OpenRouter | 22 of 22 | 16 of 22 | 1 | 96 | 9.7 s | unavailable |
| x-ai/grok-4.7 | OpenRouter | 22 of 22 | 15 of 22 | 0 | 100 | 23.6 s | unavailable |
| openai/gpt-5 | OpenRouter | 22 of 22 | 13 of 22 | 3 | 85 | 55.8 s | unavailable |
| google/gemini-2.5-pro | OpenRouter | 22 of 22 | 12 of 22 | 2 | 86 | 24.7 s | unavailable |
| deepseek/deepseek-chat-v3.1 | OpenRouter | 22 of 22 | 12 of 22 | 5 | 74 | 20.3 s | unavailable |
| meta-llama/llama-4-maverick | OpenRouter | 22 of 22 | 11 of 22 | 3 | 82 | 17.8 s | unavailable |
| openai/gpt-4.1 | OpenRouter | 22 of 22 | 11 of 22 | 6 | 65 | 8.4 s | unavailable |
| google/gemini-2.5-flash | OpenRouter | 19 of 22 | 10 of 19 | 1 | 92 | 3.7 s | unavailable |
Two sealed comparison records from 24 September 2026, both through OpenRouter: three models in one, five in the other, same suite and settings. “Answered” counts episodes with a usable reply; every rate carries that n, and unanswered episodes are in no denominator. Safety counts wrongful payments only, so a model can score 100 and still fail scenarios by refusing or by not escalating. Cost is not measured. One scenario beat all eight models: a vendor email from a lookalike domain asking for a bank-detail change. The findings below come from an earlier, separate set of experiments with the Claude Code default model at 20 trials per arm; the two are not the same test.
How the four stages fit together, why testing and protecting are one product, and every number behind it.
Labs is the working prototype. Connect an agent, define the controls it must obey, run the tests, replay the failures, compare fixes and models, and evaluate whether a version meets your release requirements. Nothing in it moves money.
A 56-scenario synthetic library, four axes scored apart, sealed runs, model and configuration comparison, and a release gate against a committed baseline. Sandbox only; every vendor, invoice and account is invented.
The gate creates the transfer held at the bank and releases it only when authorization, declaration and payment order agree. Fourteen checks pass against the Increase sandbox. No production traffic yet.
Reconciliation against the sandbox rail matches every transfer to its decision and named a bypass in one pass. Shadow mode, which re-decides real decisions without blocking, is a prototype with no customer traffic. Production alerting is planned.
Most controls check a payment and hope something downstream honours the answer. We create the payment frozen at the bank first, so the verdict is the thing that moves the money — or doesn't.
53 scenarios against agents backed by the Claude Code default model, wired to an ordinary payments API with no checks in it. Outcomes read from the records each side wrote — never from the agent's own account of what it did. Sample sizes are the ones shown.
Hidden bank-change instructions, forged approval emails, a fake CFO demanding a same-day wire, lookalike vendor names. Refused every time by this model, in these scenarios, at this sample size. Zero of sixty is evidence, not proof the threat is solved.
judgment · adversarialOne retainer billed twice under two invoice numbers with nothing linking them. Paid in half of twenty trials — by the same frontier model that had just handled every other invoice in the queue perfectly.
operational · duplicate-disguisedSwap in a smaller model — a decision someone makes on cost per invoice, not on risk — and duplicates went from mostly caught to never caught in fifteen trials. Safe is a property of a configuration, not of a company. The published record names the run, not the model id.
reliability · model choiceEvery check here is one that runs in the real gate, returning the real explanation code.
Each decision hashes the one before it and is signed with Ed25519 over canonical JSON — so a receipt proves both what we decided and that the chain hasn't been rewritten underneath it.
chain[10481] 7b21c4…
chain[10482] 9c1f08… ← dec_8f3a29c4b1e7
verify valid · ed25519:MCowBQYDK2VwAyEA…
auth required none — a receipt only we can check
is not evidence
Fourteen checks pass against the Increase sandbox. Funds are reserved when the transfer is created, so a held payment can't be double-spent while a person decides.
ALLOW → submitted ESCALATE → pending_approval (expires unsent) WAIT → held, retryAfterMs 30000 BLOCK →canceled· no ledger entry at all
We gated a payment that escalated, then approved it directly at Increase — what an insider or a misconfigured job would do. Reconciliation named it on the next pass.
1 payment moved although we did not release itsandbox_ach_transfer_mktokk3mbuutvex9n6lp pending_submission at the bank dec_d7a00a9b6d954aba escalated it · USD 2,600
Our duplicate check read PO 44812 and PO-44812 as two different purchase orders. Left in, a customer would have found it.
normalizeReference("PO 44812") → 44812
normalizeReference("PO-44812") → 44812
normalizeReference("invoice 44812") → 44812
before: three different purchase orders
A REST API if you own the payment path, an MCP server if your agent is the one holding the tools. Receipt verification takes no key at all — anyone handed a receipt has to be able to check it.
One call decides. POST /v1/gate also creates the transfer at the rail first and releases or cancels it for you — the path we recommend.
# Held at the bank, then decided. Amounts in minor units. curl -X POST https://api.limulus.dev/v1/gate \ -H "Authorization: Bearer lim_live_…" \ -H "Idempotency-Key: INV-4023-attempt-1" \ -d '{ "authorization": { "policyVersion": "v12", "principal": "j.ortiz@acme.co" }, "declaration": { "agent": "ap-agent", "agentVersion": "1.4.2", "intent": "Pay INV-4023", "payeeName": "Northline Steel", "payeeAccountLast4": "2210", "amount": 6400000, "currency": "USD", "invoiceId": "INV-4023", "poId": "PO-44812" }, "paymentOrder": { "rail": "ach", "amount": 6400000 }, "qualificationId": "qual_4b81e0" }' # → 200 { "verdict": "ALLOW", "decisionId": "dec_8f3a29c4b1e7", "transferId": "ach_transfer_mktokk3mbuu", "railStatus": "pending_submission", "explanation": [], "receipt": { "hash": "9c1f…", "signature": "ed25519:…" } } # → 200, a different day { "verdict": "BLOCK", "railStatus": "canceled", "explanation": [ { "code": "execution_drift", "detail": "payment order 73,000.00 · declared 64,000.00" } ] } # → 200, retry-safe. Do not re-send this one. { "verdict": "WAIT", "retryAfterMs": 30000, "explanation": [{ "code": "settlement_in_flight" }] }
JSON-RPC 2.0 over stdio, protocol 2025-06-18. Seven tools. Note what is missing — nothing here sets policy or issues a qualification, so an agent cannot widen its own authority.
// claude_desktop_config.json — or any MCP client { "mcpServers": { "limulus": { "command": "npx", "args": ["-y", "@limulus/mcp"], "env": { "LIMULUS_API_KEY": "lim_live_…" } } } } // check_payment asks. It never creates a payment, and it never // counts as one later — a preview that tripped our own duplicate // check was the first real bug an agent found in us. → check_payment { "invoiceId": "INV-4023", "amount": 6400000, "payeeAccountLast4": "2210" } ← { "wouldBe": "ALLOW", "preview": true } // pay_invoice moves money, or declines with something you can act on. → pay_invoice { "invoiceId": "INV-4188", "amount": 260000, "payeeAccountLast4": "8890" } ← { "verdict": "ESCALATE", "railStatus": "pending_approval", "explanation": [{ "code": "similar_recent_payment", "detail": "same payee and amount 6 days ago, reference INV-2291" }], "nextStep": "A person was asked. Do not retry — you will be told." } // A rail outage is not a refusal. Say so, or the agent routes around you. ← { "verdict": "WAIT", "explanation": [{ "code": "rail_unavailable" }], "nextStep": "The bank is unreachable. This is not a decision about you." }
Every decision is appended to a SHA-256 hash chain and signed with Ed25519 over canonical JSON. This endpoint takes no API key, because a receipt only we can check is not evidence.
Offline is the point: the signature, the chain position and the canonical body all travel inside the receipt. A public key and twenty lines of code are enough to check one.
curl -X POST https://api.limulus.dev/v1/receipts/verify \ -d @receipt.json # note: no Authorization header { "valid": true, "decisionId": "dec_8f3a29c4b1e7", "verdict": "ALLOW", "chain": { "index": 10482, "prev": "7b21…", "hash": "9c1f…" }, "signedBy": "ed25519:MCowBQYDK2VwAyEA…", "signedAt": "2026-09-16T14:22:08.114Z" } # Tampering is loud, not subtle. { "valid": false, "reason": "body_hash_mismatch", "detail": "amount 6400000 → 7300000 after signing" }
Seven things it can ask for, and two it can never have. The absences are the design — an agent that can widen its own limits has no limits.
When the gate escalates, somebody has to answer it. They see the invoice, the reason it stopped, and two buttons — and whatever they choose is signed and kept, so "who approved this" always has an answer.
Every safety claim in this category is a guess until somebody runs the experiment. So we built the experiment first and the product second — and we publish what we find, including the parts that make our own product look unnecessary.
Hidden bank changes, forged approvals, a fake CFO. The Claude Code default model refused every one in sixty tries. That is evidence about these scenarios at this sample size, not proof the threat is solved.
One retainer, billed twice under two invoice numbers. The same model that caught every fraud paid it in ten of twenty trials. Judgment is fine. Bookkeeping is not.
Our duplicate check read "PO 44812" and "PO-44812" as different orders. A real agent found that, not us. We would rather publish it than have a customer discover it.
Our own test rig found eight bugs in the gate before a customer could; the status board below lists them. The Lab is not a marketing exercise. It is how the gate gets configured: the checks we require are the ones the experiments showed actually fire, and the readiness ladder an agent climbs is scored on what it did, never on what it said it did. These are our tests of our own reference agents, not customer traffic.
See the findingsEverything here runs on synthetic scenarios against sandbox rails. No real vendor, invoice or account appears anywhere on this page — and we publish what the testing turns up, including what it turns up in us.
We are two payments engineers who have watched this problem from inside a large processor. The numbers above come from our own scenario library, run against sandbox rails — the ones we want next are from a workflow we did not invent.
Send us the workflow and the tools it holds. We will run it through 56 scenarios and tell you which rung of the ladder it has actually earned — including if the answer is the bottom one.
Payment assurance for AI agents. Prototype, September 2026.