Last update over a year ago, so I hope (2025) gets added to the title:
> [2025/05/26] (Step 1 completed!) We release Mixture-of-Thoughts--a curated reasoning dataset of 350k verified traces distilled from R1. The dataset spans tasks in mathematics, coding, and science, and is designed to teach language models to reason step-by-step. We also provide a recipe to train OpenR1-Distill-7B, which replicates the reasoning capabilities of deepseek-ai/DeepSeek-R1-Distill-Qwen-7B and marks the completion of step 1 in the Open R1 project.
Doesn't look like they managed to actually reproduce R1, and only stopped on Step 1 out of their 3-step plan.
One of my favorite code comments of all time is still in the src:
"# TODO: implement a proper validator to compare against ground truth. For now we just check for exact string match on each line of stdout." [1]
This was one of my chief complaints about the entire R1 news cycle, it felt like no one actually read the technical report. They were being heralded for their openness, but they left out the most meaningful details that you'd need to reproduce their work.
I had some contract work years ago helping some PhD program astrophysicists write scripts to make their research algorithms compute their data before their great grandchildren retire and one of the action items we never implemented was because someone wrote an “…etc” in the middle of the math.
They knew that everything was verifiably correct on the input, and verifiably correct on the output, and they swore they had figured it out at one point and just never wrote it down. I was asked if I could “just extrapolate it” and had to explain that computer programs work the same as math - I can give you a literally infinite number of ways to reach an output from an input.
I often wonder why OLMo and Nemotron aren't more popular -- they are gold-standard / "frontier" of a year ago. If we had more support behind these, seeing a true open-source AI system that legitimately challenges OpenAI & Anthropic might not be far away!
It might change soon. Nemotron 120b was never flashy but always well regarded in the community and had material strengths at long context. The 550b next gen version is out now and still very fresh. It is too early to tell but for some reason I believe the impact it will eventually have is quite strong. NVIDIA open weight models are really good. They're not flashy but they're always well put together, well documented, well licensed, and in general make for truly great bases for customization - whether it's Nemotron or Cosmos.
Cosmos 2 in particular already has taken the image diffusion world by storm in a finetune (Anima) essentially replacing/dethroning the previous budget king SDXL. I wonder if the newest Nemotron could have the same impact for open weights LLM?
I'm not really familiar with either, but I'm more familiar with Olmo. My impression is Nemotron is newer -- why is it less applicable? Is it not totally open like Olmo?
Open-source data coverage: The released datasets cover an estimated 8–10T tokens
(~40–50% of the internal 25T blend). Missing categories include code (~14% of blend),
nemotron-cc-code (~2%), crawl++ (~2%), and academic text (~2%). Users should
supplement with their own data for these categories and adjust train_iters
accordingly.
K2 Think V2 is another fully open model like Olmo, with full datasets released.
Note that the Nemotron models are generally stronger than Olmo and K2 Think V2 (according to Artificial Analysis benchmarks), and there is a lot of overlap in their datasets (lots of datasets are based on the same sources with different filtering, Olmo and K2 Think V2 both have used some Nemotron datasets).
But yeah, Nemotron is a modern and fairly capable LLM, even the 122b is more capable than Deepseek R1 (a 671b model) on most benchmarks, and there's also the recently released 550b Ultra now.
It does have a fully open training recipe, just some data missing from its datasets, but if you want a fully open pipeline it's going to be a good place to start, you just need to find some more data to fill in the datasets to get up to the token count with reasonably high quality data.
Check out OpenThoughts. It has a widely used dataset, a model that beats the deepseek's smaller reasoning models, and a paper that talks in detail about the data curation methodology.
From the blog, it looks like there hasn't been much progress for a few months, but if you check their HF it looks like they have a series of 32B models trained on top of Qwen3 32B with different numbers of training examples that they've uploaded a few days ago: https://huggingface.co/collections/open-thoughts/openthinker...
So looks a little bit more research oriented than intended for production use, but still neat to see this effort.
Olmo 3 claims that if they paid market rates for their training, it would have cost $2.75m. It was trained by a non-profit and probably had some of the compute donated, hence why they have to estimate.
Last update over a year ago, so I hope (2025) gets added to the title:
> [2025/05/26] (Step 1 completed!) We release Mixture-of-Thoughts--a curated reasoning dataset of 350k verified traces distilled from R1. The dataset spans tasks in mathematics, coding, and science, and is designed to teach language models to reason step-by-step. We also provide a recipe to train OpenR1-Distill-7B, which replicates the reasoning capabilities of deepseek-ai/DeepSeek-R1-Distill-Qwen-7B and marks the completion of step 1 in the Open R1 project.
Doesn't look like they managed to actually reproduce R1, and only stopped on Step 1 out of their 3-step plan.
One of my favorite code comments of all time is still in the src:
"# TODO: implement a proper validator to compare against ground truth. For now we just check for exact string match on each line of stdout." [1]
This was one of my chief complaints about the entire R1 news cycle, it felt like no one actually read the technical report. They were being heralded for their openness, but they left out the most meaningful details that you'd need to reproduce their work.
[1] https://github.com/huggingface/open-r1/blob/1416fa0cf21595d2...
Reminds me of my days in a computational physics PhD program.
I had some contract work years ago helping some PhD program astrophysicists write scripts to make their research algorithms compute their data before their great grandchildren retire and one of the action items we never implemented was because someone wrote an “…etc” in the middle of the math.
They knew that everything was verifiably correct on the input, and verifiably correct on the output, and they swore they had figured it out at one point and just never wrote it down. I was asked if I could “just extrapolate it” and had to explain that computer programs work the same as math - I can give you a literally infinite number of ways to reach an output from an input.
> This was one of my chief complaints about the entire R1 news cycle
For me it was the headline that a group of students replicated GPT-3 for $5000
If you really want to see fully open training pipelines for modern LLMs, Olmo and to a lesser extent Nemotron are what you should look at.
https://github.com/allenai/OLMo
https://github.com/NVIDIA-NeMo/Nemotron
After my own very exhaustive survey, I can just say '+1' and also good to note that OLMo has actually had one independent reproduction (albeit not open) done: https://www.amd.com/en/developer/resources/technical-article...
I often wonder why OLMo and Nemotron aren't more popular -- they are gold-standard / "frontier" of a year ago. If we had more support behind these, seeing a true open-source AI system that legitimately challenges OpenAI & Anthropic might not be far away!
It might change soon. Nemotron 120b was never flashy but always well regarded in the community and had material strengths at long context. The 550b next gen version is out now and still very fresh. It is too early to tell but for some reason I believe the impact it will eventually have is quite strong. NVIDIA open weight models are really good. They're not flashy but they're always well put together, well documented, well licensed, and in general make for truly great bases for customization - whether it's Nemotron or Cosmos.
Cosmos 2 in particular already has taken the image diffusion world by storm in a finetune (Anima) essentially replacing/dethroning the previous budget king SDXL. I wonder if the newest Nemotron could have the same impact for open weights LLM?
I'm not really familiar with either, but I'm more familiar with Olmo. My impression is Nemotron is newer -- why is it less applicable? Is it not totally open like Olmo?
Olmo releases their full datasets.
Nemotron only releases portions of some of their datasets, like the source code dataset that they pretrain on.
For example, from https://docs.nvidia.com/nemotron/latest/nemotron/super3/pret... :
K2 Think V2 is another fully open model like Olmo, with full datasets released.Note that the Nemotron models are generally stronger than Olmo and K2 Think V2 (according to Artificial Analysis benchmarks), and there is a lot of overlap in their datasets (lots of datasets are based on the same sources with different filtering, Olmo and K2 Think V2 both have used some Nemotron datasets).
But yeah, Nemotron is a modern and fairly capable LLM, even the 122b is more capable than Deepseek R1 (a 671b model) on most benchmarks, and there's also the recently released 550b Ultra now.
It does have a fully open training recipe, just some data missing from its datasets, but if you want a fully open pipeline it's going to be a good place to start, you just need to find some more data to fill in the datasets to get up to the token count with reasonably high quality data.
great to see OLMO mentioned here; interesting project
Check out OpenThoughts. It has a widely used dataset, a model that beats the deepseek's smaller reasoning models, and a paper that talks in detail about the data curation methodology.
https://www.open-thoughts.ai/
Oh, neat, I hadn't heard of that.
From the blog, it looks like there hasn't been much progress for a few months, but if you check their HF it looks like they have a series of 32B models trained on top of Qwen3 32B with different numbers of training examples that they've uploaded a few days ago: https://huggingface.co/collections/open-thoughts/openthinker...
So looks a little bit more research oriented than intended for production use, but still neat to see this effort.
neat
"This will likely involve curating new, large-scale datasets for math, reasoning, and code.". ... everybody likes to hand-wave on this .
What is the estimated cost these days to train something like this to conclusion?
DeepSeek themselves claimed that R1 cost $294k to train. Folks are skeptical of how low that is, however.
https://www.techspot.com/news/109542-rare-disclosure-deepsee...
Olmo 3 claims that if they paid market rates for their training, it would have cost $2.75m. It was trained by a non-profit and probably had some of the compute donated, hence why they have to estimate.
https://arxiv.org/pdf/2512.13961
So, probably somewhere between hundreds of thousands and tens of millions.
Too old now