Genesis 1B: Run 2 Extended - 80,000 Steps

Model: Genesis 1B

Training Configuration

Run 2: Training Progress (20k → 80k Extension)

Run 2 launched March 24, 2026 with a redesigned 32-layer architecture and reached 20,000 steps on March 31, 2026. The run was extended first to 40,000 steps (~21B tokens), completing April 7, 2026 with final loss ~1.93, then to 60,000 steps (~31.5B tokens), and is now continuing to 80,000 steps (~42B tokens) for deeper pre-training before the next SFT round. ETA April 24, 2026.

At 20k steps, the log shows loss 2.2103 (noisy single-step value). The run continued cosine decay toward 1e-5 through 40k steps. Average loss over steps 38k–40k is ~1.93. The run was then extended to 60,000 steps across several resumed segments (April 9–17, 2026). Loss at step 60,000 was 1.9193, throughput held at ~19,050 tok/s throughout.

Checkpoints are backed up locally every 10 minutes and uploaded to HuggingFace. Try them in the live playground.

The Dataset

~60B tokens, curated from public sources:

• FineWeb-Edu (English web, educational filter)
• DCLM baseline + extra slices
• StarCoderData (code)
• FineMath (mathematics)
• Wikipedia (multilingual)
• CulturaX (Arabic, German, Spanish, French, Japanese, Korean, Portuguese, Chinese)
• OpenHermes, Orca AgentInstruct (instruction data)
• Function calling datasets (Glaive, Gorilla, Hermes, xLAM)
• Cosmopedia (synthetic textbooks)

All tokenized with a custom SentencePiece BPE tokenizer trained on the corpus itself.

The Road to Genesis 1B v0.1

Pre-training is only the first phase. The full pipeline has four stages:

Phase 1: Pre-training - In Progress (60,000 / 80,000)

Run extended to 80,000 steps. ETA April 24, 2026.

Phase 2: SFT (Supervised Fine-Tuning) - running

SFT is now running on top of the step 40,000 base. The dataset is 510,577 examples across constitutional data (generated with Claude Haiku), SmolTalk, OpenHermes 2.5, Tulu 3, and MetaMathQA. Training runs for 15,955 steps (1 epoch) at 2e-5 peak learning rate. The approach is inspired by Constitutional AI: define a set of principles and train the model to follow them. The goal is a model with genuine personality, not a model optimized for refusal rates.

Phase 3: DPO (Direct Preference Optimization)

Refine taste and style. Train the model to prefer interesting, thoughtful responses over generic safe ones. Preference pairs are constructed to reward curiosity and penalize hedging.

Phase 4: Continued pre-training cycles

Run SFT and DPO on the 40k base, then continue pre-training to 80,000 and beyond. Each cycle produces a better pre-trained foundation, which produces a better aligned model.

The 60B token corpus means zero data repetition even at extended step counts. Every token the model sees is genuinely new data.

Run 1: What Happened (Historical)

Try It Yourself

The model is ready to inspect. Select a checkpoint and generate text to see how it evolved across the run: