LLM POST-TRAINING QWEN · PEFT · TRL
Production-grade engineering patterns

Fine-tune the model.
Optimize the preference.

A source-first engineering reference for Qwen2.5 LoRA, supervised fine-tuning, Direct Preference Optimization and ORPO. Explore isolated training stacks, data contracts, adapter outputs and detailed Python implementations using PyTorch, Hugging Face Transformers, PEFT and TRL.

Explore the architecture View source on GitHub ↗
Qwen2.5-0.5B-InstructLoRA / PEFTSFT → DPOORPOPython + Jupyter
2Independent training environments
49Qwen instruction/answer records described in README
6Embedded SFT examples
5Embedded preference pairs

SYSTEM DESIGN

Two workflows. Clear boundaries.

The repository separates modern Qwen adapter training from a strictly pinned legacy TRL preference-optimization pipeline. They should not share one Python environment.

Workflow A · Qwen LoRA

Transformers 5.x
01
Input datasetJSONL fields: instruction and output
02
Chat template + tokenizerTruncate to 256 tokens; collate batches
03
Pretrained Qwen2.5-0.5BAutoModelForCausalLM with causal loss
04
PEFT LoRA trainingRank 8 · alpha 16 · dropout 0.05
05
Adapter and tokenizer artifactsRequires base Qwen checkpoint for inference

The released script trains a causal language-model objective across the formatted sequence; it does not implement completion-only masking or a held-out validation split.

Workflow B · TRL alignment

TRL 0.8.6
01
Embedded data preparationSix instructions + five chosen/rejected pairs
02
SFT on SmolLM-135MCompletion-only response supervision
03
Reference adapterFrozen SFT adapter for DPO comparisons
04
DPO preference stageChosen versus rejected response modeling
OR
Independent ORPO routeRuns separately on a fresh model state

The legacy workflow includes an optional Phi-2 path with additional CUDA and quantization prerequisites; the default path uses SmolLM-135M.

Precision matters: DPO is a second stage after SFT in one workflow; ORPO is a separate alternative. The Qwen workflow and legacy TRL workflow have incompatible dependency requirements.

ENGINEERING CAPABILITIES

A complete view of the post-training stack.

Implementation details drawn from the released Python scripts, requirements and README—not hypothetical claims about benchmark results.

01

Qwen2.5 LoRA

A causal-language-model fine-tuning path using Qwen/Qwen2.5-0.5B-Instruct, query/value projection adapters and a dedicated modern Transformers stack.

02

SFT training

Supervised fine-tuning using instruction/response text and a completion-only training collator in the separately pinned TRL workflow.

03

Direct Preference Optimization

Pairwise preference updates using chosen/rejected responses and a frozen copy of the SFT adapter as reference.

04

Odds Ratio Preference Optimization

An independent preference-optimization training route rather than a third stage after SFT and DPO.

05

Parameter-efficient adaptation

PEFT LoRA adapters with rank 8, alpha 16 and 0.05 dropout; adapter checkpoints remain coupled to their base model.

06

Input dataset contracts

JSONL instruction/output records for Qwen and embedded SFT/preference examples for the separate TRL workflow.

07

Training optimization

AdamW-family optimizers, configurable precision, microbatching, gradient accumulation and learning-rate schedules.

08

Environment isolation

Transformers 5.x for the Qwen workflow; exact Transformers 4.40.0 / TRL 0.8.6 versions for the second path.

09

Hardware-aware execution

CPU validation and training paths, optional NVIDIA CUDA, memory-aware sequence lengths and runtime dependency checks.

10

Checkpoint artifacts

Qwen LoRA output plus SFT, DPO and ORPO adapter-save routines, subject to successful execution.

11

Notebooks and scripts

Separate Python programs and Jupyter companions for inspecting and running the post-training workflow.

12

Validation discipline

Syntax, data-schema and legacy sample checks documented; full training and GPU performance are not independently benchmarked.

MODEL TRAINING

Select the right optimization path.

The project exposes three distinct optimization concepts across two independently managed training environments.

Modern Qwen path

Qwen LoRA adaptation

Use Qwen2.5-0.5B-Instruct with PEFT adapters trained on an instruction/answer corpus. The model and tokenizer are downloaded from Hugging Face when not cached.

  • Default 5 epochs, batch size 1, accumulation 4
  • LoRA modules: q_proj, v_proj
  • FP16 on CUDA; FP32 on CPU
  • Adapter weights saved after a successful run
Pinned TRL path

SFT → DPO or independent ORPO

Use the dedicated TRL 0.8.6 interface with SmolLM-135M by default, completion-only SFT masking, and chosen/rejected response training.

  • Six embedded supervised records and five preference pairs
  • Frozen SFT adapter used for DPO reference
  • ORPO starts as a separate experiment
  • Model training requires package compatibility and sufficient hardware
ConfigurationQwen LoRALegacy SFT / DPO / ORPO
Default pretrained modelQwen2.5-0.5B-InstructSmolLM-135M
FrameworkTransformers 5.x + PEFTTransformers 4.40.0 + TRL 0.8.6
Training objectiveCausal LM on formatted prompt + answerCompletion-only SFT; DPO or ORPO preference loss
Sample data49 records documented in README; supply required JSONL6 SFT records + 5 preference pairs embedded in source
OutputQwen adapter + tokenizer if training completesSFT, DPO or ORPO adapters if training completes
Validation statusSyntax and README-described dataset checks; no published full-training benchmarkEmbedded example validation; no published full-training benchmark

RUN THE CODE

Reproducible setup, with the correct filenames.

Use two Python 3.12 environments. The checked-in Qwen script is qwen_alien_romulus_finetune_fixed.py; the legacy script is LLM_Training_Fixed_v2.py.

Qwen installation · Windows PowerShell
git clone https://github.com/su394611-del/Pre-training-and-Fine-tuning.git
cd Pre-training-and-Fine-tuning
py -3.12 -m venv .venv-qwen
.\.venv-qwen\Scripts\python.exe -m pip install --upgrade pip
.\.venv-qwen\Scripts\python.exe -m pip install -r requirements.txt
.\.venv-qwen\Scripts\python.exe -m pip check
Qwen · prepare data and start training
# The script expects: .\data\alien_romulus_qa.jsonl
# Inspect data.zip and supply the required JSONL file at that path.
# Each line needs nonempty instruction and output fields.
New-Item -ItemType Directory -Force .\data
# Once the JSONL file is in place, run:
.\.venv-qwen\Scripts\python.exe qwen_alien_romulus_finetune_fixed.py
TRL · independent pinned environment
# Run in a new terminal from the repository root.
py -3.12 -m venv .venv-trl
.\.venv-trl\Scripts\python.exe -m pip install --upgrade pip
.\.venv-trl\Scripts\python.exe -m pip install -r requirements_legacy_llm_v2.txt
.\.venv-trl\Scripts\python.exe LLM_Training_Fixed_v2.py --mode validate
.\.venv-trl\Scripts\python.exe LLM_Training_Fixed_v2.py --mode check
TRL · choose an optimization workflow
# Option A: SFT, then DPO (same legacy environment)
.\.venv-trl\Scripts\python.exe LLM_Training_Fixed_v2.py --mode sft-dpo

# Option B: ORPO as a separate experiment
.\.venv-trl\Scripts\python.exe LLM_Training_Fixed_v2.py --mode orpo
Production verification boundary: The source and README document schema checks, syntax checks and training configurations. They do not provide independently verified GPU training runs, held-out performance metrics, load benchmarks, deployment reliability, safety evaluation or model-quality evidence. The page uses production-grade engineering language for its design approach, not as a certification of the resulting models.

SOURCE & DOCUMENTATION

Inspect the implementation, not just the headline.

Direct links to the exact source files, environment specifications and project documentation.

Repository files