Skip to content
Lore
Sign inTalk to usTalk

Production agents

Observe production. Reconstruct failures. Improve your evaluations and agent harness. Autonomously.

Production intelligence

Find the moment it broke.

Go from a failed outcome to the turn, context, and tool call behind it.

VOICE / SUPPORT AGENTrun_1042
“Actually, pause it. Don’t cancel.”
Tool callsubscriptions.cancelFlagged
Agent cancelled after the caller corrected their request.
Scenario reconstruction

Recreate what actually happened.

The same conversation, acoustic conditions, data, and tool state.

Snapshot 01 / 04
Recorded turnActually, pause it. Don’t cancel.
AcousticsOffice · 18 dB SNRChannelPhone · 8 kHzTools3 captured states
Validate improvements

Same scenario. Next version.

Check the fix against known failures.

Final intentPass rate
Current
62%
Candidate
-
An illustrative suite of 40 related cases.
Adversarial simulations

Push beyond the happy path.

Real personas. Hard conversations.

The caller corrects their request mid-turn.

Continuous evaluation

Keep your tests in step.

Production failures become reusable checks.

Regression suite3 checks
Final intentGap found
Tool argumentsCovered
Policy complianceCovered
Task completionCovered

Text / Support agent

Catch a failure in production.

Lore catches policy answers that ignore market, channel, or effective date.

Failure identified
Lore
Support agent
Customer message + order metadata

Can I return this UK marketplace order at your San Francisco store?

Support agentFlagged

Authorizes the return using the current US direct-sales policy.

LoreRun review

Apply the policy for the order's market and sales channel.

Illustrative runObserve Learn Improve

Bring Lore into the thread.

Bring production failures into the conversation. Ask Lore what happened, inspect the evidence, and decide what to improve.

In the conversation

Your context. Your assistant.

Give your assistant the context behind each run. Lore connects your workspace knowledge through MCP.

Connected with MCP
Python SDK

Make evaluation part of the job.

Make evaluation part of your own workflow. Run cases against your agent and bring the results back into your stack.

A few lines of code
agent-qualitySlack thread
J
Jamie 10:42

@Lore Why did the support agent approve this refund?

Run #1042
Lore APP

It used the old 60-day policy.
The current return window is 30 days.

01 Returns policy Current version

I’ve traced the mismatch to the policy retrieved in this run.

Reply in thread…
Claude Lore connected
Check the returns policy before I update the support agent.
Lore Retrieved workspace context

The current return window is 30 days. The agent retrieved a previous version that allowed 60 days.

01 Returns policy Source

Use the current policy, then replay the original case to check the change.

Reply to Claude…
evaluate.pyPython
import withlore
from my_agent import agent
 
withlore.init()
 
result = withlore.evaluate(
    experiment="support-agent-v2",
    data="support_cases.jsonl",
    task=agent,
)
 
withlore.print_eval_report(result)
withlore.shutdown()
Evaluation complete
20 cases19 passed1 to review
Illustrative result · JSONL cases and your own agent callable

Start with one
agent workflow.

Talk to us