Skip to content
Lore
Sign inTalk to usTalk

Multimodal models

Evaluations and environments that teach multimodal models to self-improve.

Text / Retrieval experiment RCA

Catch a regression the training curve missed.

Lore compares approved and degraded runs across data, code, configuration, and evaluation results.

Experiment regression detected
Lore
Retrieval experiment RCAExperiment RCA
Retrieval-model trainingReview required

retrieval-encoder / Run #208

Recall@10
0.71
Approved baseline
0.82
Retrieval regressionHigh

Recall@10 falls while training loss remains within its approved range.

Effective batch sizeReview

Worker count doubled without changing gradient accumulation.

Data and model hashesPassed

Training data, model code, and evaluation set match the baseline.

Illustrative scenarioObserve Learn Improve

Bring Lore into the thread.

Bring data and experiment failures into the conversation. Inspect the evidence, compare runs, and decide what to test next.

In the conversation

Your context. Your assistant.

Give your assistant the evidence behind each experiment. Lore connects datasets, evaluations, and run history through MCP.

Connected with MCP
Python SDK

Make evaluation part of the job.

Make data checks and model evaluation part of every job. Catch regressions before the next expensive run begins.

A few lines of code
model-qualitySlack thread
J
Jamie 10:42

@Lore Why did retrieval quality regress in run #151?

Experiment #151
Lore APP

The regression starts in the new embedding batch.
Site C accounts for 81% of the quality drop.

01 Embedding batch v38 Current version

I compared the dataset lineage and evaluation slices across both runs.

Reply in thread…
Claude Lore connected
Compare run #143 with #151 before I retrain.
Lore Retrieved workspace context

Run #151 introduced a new embedding batch. Site C explains most of the evaluation drop.

01 Experiment comparison Source

Validate the corrected batch on the affected slice before starting a full retrain.

Reply to Claude…
evaluate.pyPython
import withlore
from training import candidate
 
withlore.init()
 
report = withlore.evaluate(
    experiment="retrieval-v151",
    data="validation_set.jsonl",
    task=candidate,
)
 
withlore.print_eval_report(report)
withlore.shutdown()
Evaluation complete
34 checks31 passed3 to review
Illustrative result · validation data and your own model callable

Start with one
model workflow.

Talk to us