All projects
06/2026
DPO / ORPO Alignment Lab
Fine-tuning Qwen2.5-3B with preference optimization
Lab author
Overview
Most of my LLM work is at the systems level (agents, RAG, telemetry). This lab goes one layer down: aligning a small open model with preference data.
I fine-tuned Qwen2.5-3B using DPO and ORPO, working through preference-pair data preparation, training runs, and before/after output comparison.
Tech Stack
PythonJupyterQwen2.5-3BDPOORPOTransformers
Lessons Learned
- Preference optimization is data-bound: pair quality dominates hyperparameters.
- Small models make alignment trade-offs visible fast — useful intuition even when production runs on hosted APIs.