LLM systems · parallel inference

Parallel LLM inference.
Independent workers. Ordered results.

A professional-grade Python inference workflow built around PyTorch, Qwen2.5-0.5B-Instruct and Hugging Face Transformers. Distribute JSONL review-classification batches across independent CPU or GPU workers, track execution and restore every result to its original input position.

Explore architecture View source on GitHub ↗

Python multiprocessingQwen2.5 0.5BJSONL pipelineCUDA workersHugging Face

INFERENCE ORCHESTRATOR / FLOW● BATCH PIPELINE

reviews.jsonl Validated records · indexed input

↓

Parent process Length-aware batching · task queue

↙ ↘

Worker 0 cuda:0 · model replica A

Worker 1 cuda:1 · model replica B

INDEPENDENT GENERATION / SHARED RESULT QUEUE

↓

results.jsonl Stable ordering · normalized labels

0.5BQwen2.5 model family

48Sample records in reviews.jsonl

1 / deviceIndependent model replica

JSONLInput and ordered output

Engineering capabilities

A complete batch-processing path, from input to output.

The checked-in Python implementation covers orchestration, device-aware generation, record validation and run instrumentation. These capabilities are available as a standalone CLI with a companion Jupyter notebook.

Process-based concurrency

Launch isolated Python workers with the spawn context and route batches through a bounded shared task queue.

Qwen2.5 inference

Load Qwen/Qwen2.5-0.5B-Instruct using Transformers, with chat-template prompts and greedy text generation.

Device-level replicas

Assign one full model replica per selected CUDA GPU. CPU inference and an offline heuristic mode are also supported.

Dynamic task distribution

Workers consume available batches as they become ready, using a common producer queue rather than fixed file partitions.

Length-aware batching

Group reviews by approximate text length before tokenization to reduce unnecessary padding in each mini-batch.

Strict JSONL intake

Validate each input object, required text field and nonempty review before scheduling inference work.

Deterministic ordering

Track original row indices across asynchronous workers, then write completed predictions in input order.

Normalized sentiment

Preserve the raw generated output and separately parse positive or negative labels, leaving unrecognized outputs null.

Execution instrumentation

Report model loading time, inference time, rows per second, output tokens per second and per-worker records.

Timeout and failure handling

Check worker exit codes, reject duplicate responses, enforce timeouts and terminate remaining processes on failure.

Offline pipeline testing

Run multiple CPU workers without downloading a model to exercise queue distribution and file processing.

Notebook integration

Use the self-contained Colab/Kaggle notebook, which launches the multiprocessing CLI through subprocesses.

System architecture

Parallelism at the worker boundary.

The parent process owns validation, scheduling and result assembly. Each worker owns model initialization and batch generation on its assigned device. All model replicas remain independent.

Batch inference execution graph

INDEPENDENT WORKERS

JSONL input or generated sample records Input schema and text validation

↓

Parent coordinator Sort indices by text length · create mini-batches

↓

Shared task queue Batch IDs + original row indices

↙ ↘

Worker 0 · GPU 0 Full Qwen model copy · generate()

Worker 1 · GPU 1 Full Qwen model copy · generate()

↘ ↙

Shared results queue Predictions + token counts + worker ID

↓

Ordered JSONL output Restore row order · add parsed sentiment

CPU MOCK USES THE SAME WORKER / QUEUE STRUCTURE

Read and validate

Load review records from JSONL, or generate synthetic examples. Reject invalid or empty review text.

Schedule by batch

Sort original row indices by review length and place groups into a queue, retaining the record-to-index mapping.

Initialize workers

Spawn processes with a device assignment. Model loading is serialized to reduce simultaneous memory and download spikes.

Generate and collect

Each worker produces a classification string and returns its batch output. The coordinator validates batch completeness.

Reassemble and measure

Write outputs in the original order and report throughput, initialization timing and worker-level counts.

Parallelism model: This is independent-worker, data-parallel inference—not DistributedDataParallel training, tensor parallelism, or pooled GPU memory. Every selected GPU must fit the complete model and its inference buffers.

Execution strategies

One orchestration engine. Two practical execution paths.

Start with the offline process test to verify the data path, then select real Qwen inference when the model weights and supported hardware are available.

MODE A · OFFLINE PIPELINE

CPU mock workers

Runs the actual multiprocessing and JSONL workflow with a deterministic keyword-based sentiment heuristic. It does not initialize an LLM and is suitable for validating orchestration independently of model downloads.

Two or more cpu worker slots
Queueing, batching and stable output ordering
Input/output JSONL validation
Not an LLM-quality or model-speed benchmark

MODE B · PRETRAINED MODEL

Qwen2.5 generation

Loads Qwen2.5-0.5B-Instruct as a causal language model on each selected device. A constrained prompt requests a single sentiment label; output is normalized only if it matches the expected format.

Single CUDA GPU or independent multi-GPU workers
Chat templates, left-padding and greedy decoding
Configurable batch and token limits
Separate load and inference timing

Property	CPU mock	Qwen inference
Model loaded	No; deterministic heuristic	Qwen2.5-0.5B-Instruct per device
Devices	`cpu cpu`, etc.	`cuda:0`, `cuda:0 cuda:1`, or CPU
Batch size default	2	2 (often reduce to 1 on limited VRAM)
Max input tokens	Not used by heuristic	256 default; configurable
Max new tokens	Not used by heuristic	8 default; configurable
Prediction output	Positive / negative heuristic	Decoded LLM text with strict label normalization
Validation purpose	Queue and file handling	Model inference behavior on selected runtime

Data engineering

A predictable contract for batch sentiment inference.

Each input row must be a JSON object containing a nonempty review string, unless another field is selected with --text-field. Existing metadata travels through the pipeline unchanged.

Input · reviews.jsonl

The repository includes 48 synthetic, labeled product-review rows. Labels are optional; when provided, they can be used for a simple accuracy calculation.

JSONL / INPUT

{"id":"R001","product":"smartphone","review":"The display is excellent and works perfectly.","label":"positive"}
{"id":"R002","product":"SSD","review":"The product stopped working.","label":"negative"}

Output · results.jsonl

Every record retains its original fields and adds the model's raw llm_output and a strictly normalized predicted_label. An unexpected response produces null for that label.

JSONL / OUTPUT SHAPE

{"id":"R001","review":"The display is excellent.","label":"positive","llm_output":"positive","predicted_label":"positive"}
{"id":"R002","review":"The product stopped working.","label":"negative","llm_output":"negative","predicted_label":"negative"}

Output semantics: The JSONL examples above illustrate the file contract, not recorded results from a GPU benchmark. In the implementation, predicted_label is null if the generated response is not a recognized one-word sentiment label.

Developer workflow

Clone, validate and run from a single CLI.

Commands below use filenames verified in the current GitHub repository. The project works as a standalone Python script; the notebook launches that script in subprocesses to accommodate multiprocessing.

01 · Clone repository / Windows PowerShell

git clone https://github.com/su394611-del/Parallel-LLM-Inference.git
cd Parallel-LLM-Inference
py -3.12 -m venv .venv
.\.venv\Scripts\python.exe -m pip install --upgrade pip
.\.venv\Scripts\python.exe -m pip install -r requirements.txt

02 · Offline 2-worker pipeline verification

.\.venv\Scripts\python.exe parallel_llm_inference.py --mock --devices cpu cpu --input reviews.jsonl --output results_mock.jsonl --batch-size 4

03 · Inspect CUDA availability

.\.venv\Scripts\python.exe -c "import torch; print('CUDA:',torch.cuda.is_available()); print('GPU count:',torch.cuda.device_count()); print('PyTorch CUDA:',torch.version.cuda)"

04 · Real inference / one CUDA GPU

.\.venv\Scripts\python.exe parallel_llm_inference.py --devices cuda:0 --input reviews.jsonl --output results_gpu.jsonl --batch-size 1 --max-input-tokens 192 --max-new-tokens 8

05 · Real inference / two CUDA GPUs

.\.venv\Scripts\python.exe parallel_llm_inference.py --devices cuda:0 cuda:1 --input reviews.jsonl --output results_2gpu.jsonl --batch-size 2 --max-input-tokens 192 --max-new-tokens 8

CUDA installation

A compatible CUDA-enabled PyTorch build is required for NVIDIA acceleration. A GPU visible to nvidia-smi does not guarantee that the active PyTorch environment can use CUDA.

Official PyTorch installer ↗

Model replica sizing

Each selected GPU loads the full Qwen model. Start with --batch-size 1 and smaller token limits on limited VRAM. Duplicating the same CUDA device in the worker list is rejected by the CLI.

Jupyter and Kaggle

Open Parallel_LLM_Inference_Colab_Kaggle.ipynb for a self-contained workflow. Its default path runs the offline mock; Qwen inference must be opted into separately.

Measured fields

After the run, the script prints model load time, inference duration, row throughput, approximate output-token throughput, per-worker counts and optional labeled-row accuracy.

Execution evidence and scope

The code implements queue-based multiprocessing, offline mock processing and a Qwen inference path. Mock results do not establish actual LLM accuracy or GPU scaling, and no measured two-GPU speedup is claimed. This project is a batch CLI workflow, not a persistent HTTP serving system or model-parallel inference engine.

Source and documentation

Inspect the implementation behind the architecture.

Links below point to repository files verified on the current legacy default branch. The separate GitHub Wiki is linked as a documentation resource.

Parallel LLM inference.Independent workers. Ordered results.

A complete batch-processing path, from input to output.

Process-based concurrency

Qwen2.5 inference

Device-level replicas

Dynamic task distribution

Length-aware batching

Strict JSONL intake

Deterministic ordering

Normalized sentiment

Execution instrumentation

Timeout and failure handling

Offline pipeline testing

Notebook integration

Parallelism at the worker boundary.

Batch inference execution graph

Read and validate

Schedule by batch

Initialize workers

Generate and collect

Reassemble and measure

One orchestration engine. Two practical execution paths.

CPU mock workers

Qwen2.5 generation

A predictable contract for batch sentiment inference.

Input · reviews.jsonl

Output · results.jsonl

Clone, validate and run from a single CLI.

Inspect the implementation behind the architecture.

GitHub repository files

Parallel LLM inference.
Independent workers. Ordered results.