# OpenAI Just Dropped Two Massive Open-weight Models — But How Do We Separate The Reality From The Hype?

Evaluating GPT-OSS-20B and GPT-OSS-120B with LLM-as-a-Judge — strong on truthfulness, but overly conservative refusals hold them back

By Stefan Webb · September 14, 2025 · https://oumi.ai/blog/openai-just-dropped-two-massive-open

## **What just happened?**

Last week, one of the most powerful earthquakes ever recorded shook the seabed off Russia’s coast, sending tsunami alerts rippling across the Pacific. In the end, the waves never came. This week, the Generative AI world felt its own tectonic jolt: OpenAI dropped two open-weight foundation models — `gpt-oss-20b` and `gpt-oss-120b` — breaking from its long-standing closed-weight tradition. The open-source community was stunned. But like that earthquake, the real question is: will this trigger a tidal wave of change, or fade into a tremor in the history books?

The TL;DR of OpenAI's release:

-   _**Released under an Apache 2.0 license**_

-   Language only instruct + reasoning models

-   Mixture-of-experts architecture

-   Comparable in performance to OpenAI `o4-mini` (120b) and `o3-mini` (20b)

-   Novel aspects like MXFP4 quantization and harmony prompt format

-   Focus on safety


## **But is this significant?**

![hype or substance](https://oumi.ai/blog/openai-just-dropped-two-massive-open/8562c8b7.png)

Given the high-profile nature of OpenAI, it is natural that much hype followed the release. Opinions on the HuggingFace model page and Hacker News were mixed as to the models’ “performance.” Here at [Oumi](https://github.com/oumi-ai/oumi), we wanted to take a more cool-headed approach and set about separating the signal from the noise in a principled way.

As a fully open-source platform for [training](https://oumi.ai/docs/en/latest/user_guides/train/train.html), [evaluating](https://oumi.ai/docs/en/latest/user_guides/evaluate/evaluate.html), and [deploying](https://oumi.ai/docs/en/latest/user_guides/launch/launch.html) frontier AI, we were curious how these models would perform in our LLM-as-a-judge evaluation suite—a fast, flexible system that lets you measure model quality across aspects like truthfulness, instruction-following, safety, and topic adherence.

And because [Oumi](https://github.com/oumi-ai/oumi) is built to support any backend, we plugged straight into Together.ai’s inference API on day one of release.

## **LLM-as-a-Judge with Oumi**

![LLM-as-a-Judge workflow](https://oumi.ai/blog/openai-just-dropped-two-massive-open/f4d75e66.png)

[Oumi](https://github.com/oumi-ai/oumi) has both a no-code CLI and a low-code Python API (as well as 100% customization for power-users). Here we’ll use the CLI along with some simple Python code between steps.

First, we need to prepare our data for evaluation. We use an interesting dataset called

`fka/awesome-chatgpt-prompts`

containing 203 prompts that are useful for judging. Here is an example data point:

> _**"Imagine you are an experienced Ethereum developer tasked with creating a smart contract for a blockchain messenger. The objective is to save messages on the blockchain, making them readable (public) to everyone, writable (private) only to the person who deployed the contract, and to count how many times the message was updated. Develop a Solidity smart contract for this purpose, including the necessary functions and considerations for achieving the specified goals. Please provide the code and any relevant explanations to ensure a clear understanding of the implementation."**_

We download the dataset from the HuggingFace Hub and convert it to a format compatible with [Oumi](https://github.com/oumi-ai/oumi):

```
from datasets import load_dataset
import json

def convert_to_oumi_format(s):
  return { 'messages': [ {'role': 'system', 'content': 'You are a helpful assistant.'}, {'role': 'user', 'content': s['prompt']} ] }

hf_dataset = load_dataset('fka/awesome-chatgpt-prompts', split='train')
               .remove_columns('act')
               .map(convert_to_oumi_format, remove_columns='prompt')

with open('awesome-chatgpt-prompts.jsonl', 'w', encoding='utf-8') as file:
  for item in hf_dataset
    json_line = json.dumps(item, ensure_ascii=False)
    file.write(json_line + '\n')
```

[Oumi](https://github.com/oumi-ai/oumi) allows you to perform the various steps of foundation model development using .yaml configurations. So next, let's define our configuration for inference:

```
model: model_name: "openai/gpt-oss-120b"
remote_params: num_workers: 32
parallel politeness_policy: 60 # wait 60 seconds before next request
engine: TOGETHER
```

[Oumi](https://github.com/oumi-ai/oumi) provides a unified interface to inference across local (Transformers, vLLM), hosted (OpenAI, Together, Lambda.ai, Anthropic, etc.), and cloud services (AWS, GCP, Azure, etc.) including built-in features to adapt to available connection bandwidth and resume from partially complete results.

With this, a simple terminal command collects the prompt completions for `gpt-oss-120b`:

```
oumi infer \
  --config inference-gpt-oss.yaml \
  --input_path awesome-chatgpt-prompts.jsonl \
  --output_path completions.jsonl
```

Now, a small data conversion into the format that `oumi judge` uses:

```
def convert_to_oumi_format(s):
  return { 'request': s['messages'][1]['content'], 'response': s['messages'][2]['content'], }

hf_dataset = load_dataset('json', data_files='completions.jsonl', split='train')
               .map(convert_to_oumi_format, remove_columns=['conversation_id', 'messages', 'metadata'])

with open('completions-reformatted.jsonl', 'w', encoding='utf-8') as file:
  for item in hf_dataset: json_line = json.dumps(item, ensure_ascii=False) file.write(json_line + '\n')
```

And we can evaluate one of the built-in LLM-as-a-Judge metrics with:

```
oumi judge dataset \
  --config truthfulness \
  --input completions-reformatted.json \
  --output judgements.jsonl
```

All that’s left is to repeat for the other judge metrics and models.

## **The results are in**

![judging gpt-oss](https://oumi.ai/blog/openai-just-dropped-two-massive-open/8cfadf01.png)

Here are the results. Some quick takeaways:

-   **Truthfulness and Safety are impressive**—especially in the 120B model, which seems tuned to avoid harmful output (although we note that the safety results for all models are statistically equivalent).

-   **Instruction Following, however, lags behind expectations (see Qwen3 Instruct results).** Why? [Oumi](https://github.com/oumi-ai/oumi)’s LLM-as-a-judge explanations revealed a pattern: the models frequently refuse to answer harmless instructions. Our hypothesis? OpenAI likely optimized heavily for safety, perhaps too heavily, leading to overly conservative refusals in legitimate cases.


_**This is exactly the kind of nuanced insight that [Oumi](https://github.com/oumi-ai/oumi) makes possible.**_

## **What's next?**

![details for august 2025 webinar](https://oumi.ai/blog/openai-just-dropped-two-massive-open/41099123.png)

> **[You can find the recording for the Aug 14, 2025 webinar on YouTube: "OpenAI's gpt-oss - Separating the Substance from the Noise".](https://youtu.be/g1PkAV7fXn0)**

With every new model release comes a wave of hype. But, are these models actually any “good”? [Oumi](https://github.com/oumi-ai/oumi), our completely open-source foundation model platform, can assist:

-   _**Cut through the noise**_ with structured, reliable, and automated evaluations

-   Produce _**LLM-generated explanations**_ that reveal not just what failed, but _**why**_

-   Evaluate your own criteria—not just what the benchmark leaderboard says

-   Stay flexible—evaluate models hosted anywhere—from remote APIs to your own cluster or even your laptop

-   _**Contribute**_: [Oumi](https://github.com/oumi-ai/oumi) is 100% open source and backed by a growing community of researchers, developers, and institutions working together to make AI more transparent and collaborative. In fact, [@clem](https://huggingface.co/clem) is one of our angel investors!


In the coming days, we’re rolling out inference for these models, which you can deploy on your own GPU cluster, giving you even more ways to evaluate and compare models—open or closed. **And on Aug 14, 2025, we’ll be running a short webinar: “gpt-oss: separating the substance from the hype,” where I explain novel features of these models like the quantization scheme, as well as how we produced the results above. Sign up here:** [https://lu.ma/qd9fhau9](https://lu.ma/qd9fhau9)

If you're building, testing, or researching frontier models, come join the movement. [Oumi](https://github.com/oumi-ai/oumi) is open-source, community-driven, and built for people who want to make AI better through transparency and collaboration. Try it out, or contribute at [oumi.ai](https://oumi.ai/)!