DISTRIBUTED AI ENGINEERING / PYTORCH

Train across processes.
Understand every gradient.

An inspectable implementation of PyTorch DistributedDataParallel: learn how torchrun launches workers, how DistributedSampler partitions training examples, and how gradient synchronization supports causal Transformer training and Qwen2.5 LoRA experiments.

Run the project ↗Explore source ↗

Code and notebook are public. Two-process CPU execution is reported as tested; GPU/Qwen configurations are example pathways, not published performance benchmarks.

DATA PARALLEL TRAINING / OVERVIEWPROCESS-GROUP MODEL

torchrun · world_size = 2

RANK 0 CPU / GPU 0

Model replica A

Data shard A
Forward → loss → backward

RANK 1 CPU / GPU 1

Model replica B

Data shard B
Forward → loss → backward

DDP · GRADIENT ALLREDUCE Synchronize gradients → optimizer step

TORCH DISTRIBUTEDGLOO / NCCLDATA PARALLELISM

02Training paths
tiny Transformer / Qwen LoRA

2 ranksCPU Gloo demonstration
reported as verified in README

0.5BQwen2.5-Instruct
pretrained-model option

100%Full model replica
per DDP worker

01 / THE IMPLEMENTATION

What the source code covers.

Each capability below corresponds to the published Python implementation or README. Start with a two-rank CPU experiment, then study the CUDA-oriented configurations.

PYTHON · PYTORCH · PEFT

DistributedDataParallel

One process per rank, with a replicated model and coordinated gradient updates through PyTorch DDP.

DDPprocess group

torchrun orchestration

Launch one or more workers using the torch distributed runner and environment-defined ranks.

WORLD_SIZELOCAL_RANK

Partitioned data loading

DistributedSampler assigns partitions of the training dataset to workers, with epoch-aware shuffling.

DistributedSampler

Gradient AllReduce

DDP coordinates gradient communication during backward passes; the script also reduces local losses for logging.

AllReduceAdamW

Custom causal Transformer

A compact byte-level model uses positional embeddings, causal masking, self-attention, and next-token loss.

PyTorchfrom scratch

Qwen with LoRA

A separate path loads Qwen2.5-0.5B-Instruct and adds PEFT LoRA adapters to query and value projections.

Qwen2.5PEFTLoRA

CPU / GPU backend selection

Gloo is selected for CPU or Windows, while CUDA-enabled Linux can use NCCL when multiple ranks are launched.

GlooNCCL

Custom corpus support

Supply UTF-8 text samples, one per line, using --corpus-file to replace the built-in teaching corpus.

UTF-8CLI

Checkpoints and adapters

Rank zero saves a tiny-model checkpoint or the Qwen LoRA adapters, keeping per-rank file writes coordinated.

rank 0save_pretrained

02 / SYSTEM ARCHITECTURE

From launch to synchronized weights.

The pipeline below follows the implementation's control flow. DDP is data parallelism: workers use different data, while each keeps the complete model.

01 / LAUNCHtorchrun

Starts worker processes and provides rank metadata.

02 / CONNECTProcess group

Initializes Gloo or NCCL as appropriate.

03 / SHARDData sampler

Assigns training examples to each rank.

04 / TRAINForward & loss

Computes next-token loss on local batches.

05 / SYNCDDP gradients

Coordinates gradient reduction during backward.

06 / SAVERank zero

Writes model checkpoint or adapters to disk.

DDP replicates; it does not combine GPU memory.

Two 4 GB GPUs do not create an 8 GB memory pool for one model. For models that do not fit on one device, look into FSDP, tensor parallelism, or other forms of model sharding. The code here is an example of DDP, not those other strategies.

03 / TWO TRAINING PATHS

One orchestration pattern, two models.

The same distributed launch concepts are exercised with a tiny model and a pretrained Qwen checkpoint.

PATH A / NO MODEL DOWNLOAD

Tiny causal Transformer

Train a small byte-level autoregressive model with learned token and positional embeddings, one Transformer encoder block with causal masking, and a next-token training objective.

Runs on CPU without a pretrained-model download
Can demonstrate two ranks with Gloo
Uses a small built-in corpus or a text file
Writes tiny_transformer.pt on rank zero

PATH B / CUDA-ORIENTED

Qwen2.5 + LoRA

Load Qwen/Qwen2.5-0.5B-Instruct and configure PEFT LoRA adapters for parameter-efficient next-token training on the provided text corpus.

Adapter targets: q_proj and v_proj
LoRA rank 8, alpha 16, dropout 0.05
Runs with one GPU or multiple CUDA workers where supported
Saves adapters and tokenizer from rank zero

04 / HARDWARE & VALIDATION

Know what is demonstrated.

These are repository-documented capabilities and test statements, not invented benchmark scores.

Training modes, hardware, backend, and verification status
Scenario	Backend	Hardware	Evidence
1-rank tiny model	Local (no process group)	CPU	README reports completed 2-step test and checkpoint
2-rank tiny model	Gloo	CPU	README reports completed 2-rank test and checkpoint
Qwen LoRA (1 rank)	Local CUDA process	One suitable GPU	Runnable code path; no published GPU execution result
Qwen LoRA (multiple ranks)	NCCL on Linux CUDA	At least two supported GPUs	Example launch path; physical multi-GPU results not reported

05 / GETTING STARTED

Run a distributed training example.

Start on CPU to learn rank synchronization. Move to pretrained-model LoRA mode only after checking available CUDA devices and memory.

TERMINAL / POWERSHELL

WINDOWS POWERSHELL · INSTALL

git clone https://github.com/su394611-del/Distributed-Data-Parallel-Processing.git
cd Distributed-Data-Parallel-Processing

py -3.12 -m venv .venv
.\.venv\Scripts\python.exe -m pip install --upgrade pip
.\.venv\Scripts\python.exe -m pip install -r requirements.txt
.\.venv\Scripts\python.exe -m pip check

# Check whether this environment can access CUDA
.\.venv\Scripts\python.exe -c "import torch; print(torch.cuda.is_available()); print(torch.cuda.device_count())"

Run two CPU processes with real DDP

CPU · GLOO · TWO RANKS

.\.venv\Scripts\python.exe -m torch.distributed.run --standalone --nproc_per_node=2 llm_ddp_parallel_training.py --model tiny --cpu --batch-size 2 --max-steps 2

Run a single-process CPU smoke test

CPU · SINGLE PROCESS

.\.venv\Scripts\python.exe llm_ddp_parallel_training.py --model tiny --cpu --max-steps 2

Qwen LoRA example — one CUDA GPU

CUDA · SINGLE RANK

python -m torch.distributed.run --standalone --nproc_per_node=1 llm_ddp_parallel_training.py --model qwen --batch-size 1 --seq-len 64 --max-steps 2

Qwen LoRA example — two CUDA GPUs (Linux)

CUDA · TWO RANKS / LINUX

python -m torch.distributed.run --standalone --nproc_per_node=2 llm_ddp_parallel_training.py --model qwen --batch-size 1 --seq-len 64 --max-steps 2

06 / DOCUMENTATION & SOURCE

Read, run, and inspect.

Follow the implementation directly from this page. Each link points to the current repository name and file paths.

GitHub repositorySource tree, commits, and project files README and run instructionsModes, distributed backends, tests, and limitations Python training implementationDDP orchestration, causal Transformer, and PEFT LoRA Jupyter notebookNotebook-driven subprocess training experiments DependenciesPyTorch, Transformers, PEFT, Accelerate, Jupyter GitHub WikiAdditional design documentation, where published

07 / FAQ

Distributed training, explained.

Is DDP the same as tensor parallelism?

No. DDP gives each rank a complete model replica and divides the dataset. Tensor parallelism partitions parts of the model across devices. DDP does not pool VRAM.

Can I demonstrate distributed training on a CPU-only PC?

Yes. The script and README include a two-process CPU command using the Gloo communication backend. It is suitable for learning the process-group and gradient synchronization mechanics.

Is Qwen2.5 trained from scratch in this project?

No. The Qwen path downloads a pretrained Qwen2.5-0.5B-Instruct checkpoint and applies parameter-efficient LoRA updates. The tiny causal Transformer is the model constructed from scratch.

Is this QLoRA or 4-bit quantization?

No. The repository explicitly identifies the Qwen training path as LoRA, without 4-bit QLoRA quantization.

What does the global batch size mean?

For this example, global batch size equals the per-rank batch size multiplied by the number of ranks. The implementation does not include gradient accumulation.