OpenAI's Mental-Health Benchmark Puts GPT-6 Astra on Top — Scored by Another OpenAI Model

OpenAI released MentalHealthBench on September 23, an open benchmark built to test how AI models handle mental-health conversations – from everyday stress to acute emergencies – and, in the process, put its own newest model at the top of the resulting leaderboard. The detail worth pausing on: the benchmark’s automated grader is also an OpenAI model.

What MentalHealthBench Actually Measures

The benchmark consists of 1,215 synthetic conversations, paired with 5,262 rubric criteria written by more than 80 licensed psychologists and psychiatrists across 22 countries. Conversations are split by acuity: 53.5% non-acute everyday topics (stress, low mood, relationship strain), 18.2% high-acuity conversations flagging serious concerns, and 28.3% outright emergencies involving immediate safety risk. Each conversation was reviewed by at least three experts before its rubric was finalized, and scoring is decomposed across behavioral axes such as context-seeking, empathy, urgency calibration, and reality testing.

The Leaderboard

On the published results, OpenAI’s own GPT-6 Astra scored highest at 57.3%, ahead of GPT-6 Sol (53.9%) and Anthropic’s Claude Opus 5.5 (52.4%). Further down, GPT-4o (the March 2025 snapshot) scored 32.1% and Google’s Gemini 2.5 Pro scored 29.5% – roughly half of Astra’s score, on the same rubric.

The Grader Is Also an OpenAI Model

The scoring itself isn’t done by the clinicians who wrote the rubrics – it’s automated, run by GPT-5.6 Sol acting as an LLM judge, comparing each model’s response against the expert-written criteria. That’s a standard technique in benchmark design (grading thousands of long-form conversations by hand doesn’t scale), but it means every score on this leaderboard, including the ones for competing labs’ models, passed through an OpenAI model’s judgment. OpenAI’s own paper, as described in coverage of the release, acknowledges that judge choice, prompt wording, and sampling variance can move the numbers – a caveat that matters more, not less, when the judge belongs to the same lab as the top-ranked model.

What the Benchmark Admits It Doesn’t Show

Coverage of the release also surfaces two limits worth keeping in mind before treating this leaderboard as a ranking of “which AI gives better mental-health support.” First, it only scores a single conversational turn – it says nothing about whether a longer exchange built trust, whether the person understood the advice, or whether they went on to seek appropriate care. Second, when OpenAI separately collected rubrics from real past users of AI for emotional support, alongside the clinicians, the two groups didn’t fully agree on what a good response looks like: clinicians weighted guardrails and appropriate caution more heavily, while users cared more about concrete next steps and tone. A model that scores higher against the clinical rubric isn’t automatically the one a struggling person would find more helpful in the moment.

Why This Matters If You’re Building With These Models

If your product touches anything adjacent to emotional or mental-health support – a wellness app, a companion bot, even general-purpose customer support that occasionally gets a distressed message – MentalHealthBench is a genuinely useful reference: it’s the first benchmark of its kind built with this much clinical input, and it’s open for anyone to run. But the leaderboard number alone shouldn’t be read as “this model is safer for vulnerable users than that one.” It’s a score against one lab’s rubric, produced by that same lab’s grading model, describing a single turn of conversation – not a study of what happens next to the person on the other end.