Our Gemma QA fine-tune, aligned with Direct Preference Optimization on 500 AI-judged preference triplets — no reward model.
The Gemma QA adapter, further trained with DPO against a frozen reference. Preferences were written by Gemini and verified by a blind LLM judge that the chosen answer is genuinely better.
750 steps, ~7 min on an L4. Judged correctness 0.924 [0.906–0.942]; the paired win-rate against the SFT checkpoint is 46% (ties split), so DPO did not raise judged accuracy on the already-strong Gemma — it mostly shifted style rather than correctness.
| Class | Gemma2ForCausalLM |
| Layers | 26 |
| Hidden size | 2,304 |
| Attention | 8 heads / 4 KV · dim 256 · GQA |
| Feed-forward | GeGLU · inner 9,216 |
| Attention window | sliding 4,096 (alternating) · logit soft-cap |
| Norm | RMSNorm |
| Context | 8,192 tokens |
| Vocabulary | 256,128 |
| Embeddings | tied input/output |
| Init from | gemma-2-2b QA-SFT adapter |
| Method | QLoRA-DPO · rank-16 |
| Trainable params | 20.8M LoRA (0.8%) |
| Training data | 500 preference pairs |
| Training tokens | 335K (chosen + rejected) |
| Optimizer steps | 750 |
our cost begins at fine-tuning — the base is Google's, imported free.
| Stage | Detail | Cost |
|---|---|---|
| QLoRA QA fine-tune | the adapter DPO starts from | $9.84 |
| QLoRA-DPO alignment | direct preference optimization on L4 | $0.43 |
| Evaluation (shared) | 13 versions on 500 held-out questions, this model's share | $1.48 |
| Total | $11.75 |
Figures are Modal GPU usage (time × rate) across this model's lineage; shared datasets are charged at this model's share. Whether base pretraining is included is stated above — it is for the models pretrained here, and excluded for imported bases. Evaluation-derived metrics come from an independent blind-judge harness on a frozen, decontaminated held-out set. Serving is billed separately and scales to zero.