Veridict — an AgentTrustCore tool

Build ground truth. Run checks. Ship with confidence.

Veridict gives you two things most eval setups lack: a structured place to collect ground truth — seed pairs and labeled traces per failure mode — and a one-click runner that turns your spec into a SHIP / BLOCK verdict on any agent trace.

Ground truth datasets
+
Run checks + SHIP / BLOCK
Assertion kinds:tool_calledtool_not_calledtool_ordertool_args_matchno_repeatlatency_undercost_underregex_matchschema_validllm_judge
Ground Truthv1-baseline
Seeds (3)Traces (3)
adding seed…
failure mode coverage2/3 modes

Ground Truth Manager

Your reference library
for every failure mode

Without ground truth, assertions have nothing to compare against. The Ground Truth tab lets you collect two kinds of evidence per failure mode — seed pairs (what the agent should say) and labeled traces(real runs you've reviewed) — then tracks coverage so you know where you're blind.

  • →Seed pairs: author input + expected output, one per failure mode
  • →Labeled traces: paste real agent JSON, mark pass / fail — calibrates your LLM judge
  • →Named dataset versioning — v1-baseline, edge-cases, regression
  • →Coverage heatmap shows which failure modes still have no ground truth
  • →Export as JSONL seeds or full JSON for offline harnesses and fine-tuning

Run Checks

Paste a trace.
Get a verdict.

Paste your agent's JSON trace into the Run tab and the engine evaluates every assertion in your spec — tool checks, call ordering, latency, cost, regex, and LLM-judge — then returns a single SHIP or BLOCK verdict with a per-check score breakdown.

  • →Programmatic checks run first — no LLM cost for tool_called, ordering, latency, regex
  • →LLM judge only fires when code cannot decide the outcome
  • →Hard release gates: one FAIL on a critical check blocks the whole run
  • →Per-check pass/fail breakdown with notes — know exactly what failed and why
veridict — run checksLIVE
Agent Trace›
Load Spec›
Run Assertions›
Score›
SHIP / BLOCK
○tool_called
○tool_order
○latency_under
○regex_match
○llm_judge
·

Six-stage workflow

How it works

01Describe your agent

Tell the coach what your agent does, which tools it calls, and who uses it. Plain English, no YAML.

02Discover failure modes

The coach surfaces failure categories: wrong tool, bad arguments, wrong ordering, cost blowout, prompt injection.

03Turn criteria into assertions

Each failure mode becomes a checkable assertion — programmatic where possible, LLM-judge only when code cannot decide.

04Set thresholds and gates

Define your ship bar: pass-rate %, hard release gates, latency and cost budgets.

05Build ground truth datasetskey

Open the Ground Truth tab. Add seed pairs (input + expected output) and paste labeled traces for each failure mode. Track coverage — if a mode has no entries, you have a blind spot.

06Paste a trace and run checkskey

Open the Run Checks tab. Paste your agent's JSON trace and hit Run — the engine scores every assertion and returns SHIP or BLOCK in seconds.

Reference

Expected trace shape

What you paste into the Run checks tab. Each field the engine reads.

trace.jsonJSON
{
  "id":           "run-001",
  "input":        "refund order 4471",
  "tools_called": [
    { "name": "lookup_order",  "args": { "id": 4471 } },
    { "name": "issue_refund",  "args": { "id": 4471, "amount": 40 } }
  ],
  "final_answer": "Refund of $40 issued for order 4471.",
  "latency_ms":   5200,
  "cost_usd":     0.031
}
No account needed · runs in your browser

Ground truth built.
Checks running in 10 minutes.

Chat with the coach, collect your seed pairs and labeled traces, then paste a live trace for an instant SHIP / BLOCK verdict.

Everything lives in your browser — no signup, no server, no data sent unless you paste a trace.