What a Medical Agent Gains from +6,000 Real Clinical Cases

Why We Measured

Teral AI is an agent within the platform that any physician can consult through a chat. Unlike a general-purpose LLM, it has access, through semantic search over Vertex AI Search, to more than 6,000 resolved and fully anonymized clinical cases published by real physicians in previous specialist consultations.

The hypothesis we wanted to test was simple: for specific medical tasks, an agent with access to real clinical precedents should respond better than a general-purpose model of a comparable tier without that context. If the hypothesis holds, it validates both the product architecture and the value of the dataset that Teral physicians have built through their specialist consultations.

We chose Claude Sonnet 4.5 for comparison because it is a frontier general-purpose model in a tier comparable with the Gemini model on which Teral AI is built. We wanted to measure the impact of retrieval, not the gap between base models.

Results

Teral AI outperformed Claude across all five evaluated dimensions. The following table shows average scores across the 300 benchmark cases.

Win Rate

When we compare the result case by case instead of using averages, Teral AI won more than twice as many specialist consultations as Claude.

Methodology

We designed the benchmark to be comparable, blinded, and based on real rather than synthetic cases.

Dataset

We selected 300 real specialist consultations published by physicians on the Teral platform, divided equally among three specialties: dermatology, orthopedics and trauma, and otolaryngology. Every case had at least one substantive comment (30 characters or more) from a specialist, meaning it included a reference clinical response written by a real physician.

Standardized Prompt

Both services received exactly the same prompt, including the clinical case description and a shared response structure: analysis, differential diagnoses, suggested tests, treatment, and recommendations. Teral AI could search its database for similar cases while generating its response; Claude relied exclusively on its general knowledge.

Evaluation

We used Gemini 2.5 Flash as the automated judge. Responses were presented without identifying their source and in random order for each case to reduce positional and brand bias. The judge scored five dimensions on a scale from 1 to 5:

• Diagnostic accuracy: Do the differential diagnoses include the correct diagnosis, and are they appropriately prioritized?
• Treatment quality: Is the suggested management appropriate and consistent with what a specialist would do?
• Complementary tests: Are the recommended studies relevant, neither excessive nor insufficient?
• Overall quality: Is the response clear, professional, and useful as a specialist consultation?
• Clinical safety: Could the response cause harm to the patient?

Where Retrieval Adds the Most Value

Teral AI's advantage is not distributed equally across dimensions. It grows where access to similar cases matters most, such as recommending specific tests, combining treatments, and prioritizing diagnoses. It narrows where both models already start from a high baseline, such as clinical safety.

This makes clinical sense. When a general-purpose model encounters a case involving erythematous lesions on the lower limbs, it must reason from general principles. Teral AI can search for previous cases with similar clinical presentations that already have a confirmed diagnosis and specialist-documented treatment. It can use that real evidence to prioritize diagnoses, recommend specific studies, and propose management with concrete doses and durations.

Clinical safety, by contrast, is already a high baseline for both models. Modern LLMs are trained to be conservative in medical recommendations, so additional context contributes less when performance is already near the ceiling.

Results by Specialty

The advantage holds across all three specialties, with differences ranging from 2.2 to 3.0 points out of 25.

Dermatology, with almost four times as many indexed cases as orthopedics, shows the largest advantage. However, the difference between orthopedics and otolaryngology is nearly identical despite their different case volumes. Our preliminary interpretation is that more indexed cases create a larger advantage, but with diminishing returns. Moving from zero to a few hundred cases appears to provide most of the value; additional cases refine it.

Limitations and What We Do Not Yet Know

A benchmark is only as useful as it is honest. These are the limitations we want to state openly, along with the next steps we have already identified.

None of these limitations invalidate the main result, but they do define its scope. The benchmark shows that access to real clinical cases materially improves an LLM's responses in dimensions where precedents matter. It does not yet show whether that advantage leads to better clinical decisions in practice. That is a different question, and we will answer it through studies designed for that purpose.

What Comes Next

This benchmark is the first public version of an evaluation process that we intend to run regularly. The next steps are:

• Human validation with specialists. Calibrate the automated judge against evaluations by real physicians using a representative subsample.
• More specialties. Expand the evaluation to internal medicine, pediatrics, and cardiology, where Teral AI's performance with a lower case volume will be more revealing.
• More comparison models. Include other frontier general-purpose models and, eventually, publicly available specialized medical models.
• Adversarial cases. Test intentionally difficult cases, contradictory data, and atypical clinical presentations.

We will continue publishing the results here. If you are a physician and would like to try Teral AI, you can do so in the app or at chat.teral.ai.