Hand-built attention
Q, K and V projections, scaled attention scores, padding masks and a learned output projection.
PYTORCH · TRANSFORMERS · REPRODUCIBLE LEARNING
Transformer encoder,
built from first principles.
Explore a hand-built Transformer encoder for binary sentiment classification: multi-head self-attention, positional encoding, residual blocks, and PyTorch training—from an offline smoke test to Yelp Polarity experiments.
Implementation walkthrough · Source code and dependencies are public · Real-data modes use an existing MiniLM tokenizer and frozen token embeddings.
ENCODER / ATTENTION MAPPYTORCH 01
The
service
was
great
Contextual token representations
Q · K · V → Encoder blocks → Masked pooling
NegativePositive
03
Encoder layersDefault real-data configuration
06
Attention headsPer encoder layer
02
Sentiment classesNegative · Positive
INSIDE THE IMPLEMENTATION
Model components, explained.
Rather than calling a pretrained encoder for classification, the project constructs attention and Transformer blocks directly from PyTorch layers. The same code supports a manual attention path and an optimized SDPA path.
Optimized SDPA option
Switch between explicit attention math and PyTorch’s scaled dot-product attention using --sdpa.
Positional encoding
Fixed sinusoidal positions give the encoder sequence-order information without learned position parameters.
Pre-LayerNorm blocks
Configurable encoder layers with residual paths and LayerNorm before both attention and feed-forward branches.
GELU feed-forward
Each block applies two linear transformations with a GELU nonlinearity, followed by a residual connection.
Padding-aware pooling
Masked mean pooling excludes padding tokens before the linear classification head.
Training and checkpoints
AdamW, cross-entropy, gradient clipping, a fixed seed and best-validation-accuracy checkpoint selection.
Evaluation and visualization
Held-out test evaluation after checkpoint reload; optional model-diagram export using torchview and Graphviz.
What “from scratch” means here: The encoder blocks and classification head are custom-trained. For the real Yelp experiments, the tokenizer and 384-dimensional frozen input embeddings are obtained from all-MiniLM-L6-v2. This is a bidirectional sentiment classifier, not a GPT-style generative LLM.
SYSTEM ARCHITECTURE
From text to prediction.
Follow the flow of a review through tokenization, frozen wordpiece embeddings, the custom encoder and a binary classification head. The drawing reflects the structure described in the Python source.
END-TO-END MODEL PIPELINE01 → 05
ReviewText input
TokenizeIDs + mask
EmbeddingFrozen / synthetic
PositionSinusoidal
Encoder × NAttention + FFN
REPEATED ENCODER BLOCKPRE-LAYERNORM
LayerNorm → Q/K/VAttention → residualLayerNorm → GELU FFNResidual add
Final LayerNorm→ Masked mean pooling→ 2-class logits
THREE EXECUTION MODES
Start small. Scale the experiment.
All modes share the same custom encoder; dataset loading and training configurations differ. The real-data modes download Yelp Polarity data and MiniLM resources unless already cached.
MODE 01OFFLINE / CPU
Smoke test
Synthetic Yelp-style review samples, a tiny tokenizer and random frozen embeddings. Exercises the complete training, checkpoint reload and evaluation path without a model download.
Training64
Validation8
Test8
Epochs2
MODE 02YELP / SUBSET
Yelp small run
Uses the Yelp Polarity dataset, a pretrained MiniLM tokenizer and frozen input embeddings. Enables the attention implementation and training configuration to be studied on real reviews.
Requested train1,000
Validation200
Test200
Optimized path--sdpa
MODE 03YELP / LARGER
Extended Yelp run
A larger requested dataset and longer training preset. GPU acceleration is optional when PyTorch detects CUDA; memory and runtime depend on hardware and review filtering.
Requested train50,000
Validation5,000
Test10,000
Epochs5
Real-data sample counts are requested values: length filtering can reduce the number of eligible examples. The smoke test is a correctness exercise, not a benchmark for Yelp accuracy.
RUN IT LOCALLY
Clone, install, train.
Use the commands below from Windows PowerShell. The script also runs in compatible Linux and macOS environments. For a first test, choose the offline smoke mode.
TERMINAL / Install dependenciesPython 3.12 · Windows PowerShell
git clone https://github.com/su394611-del/LLM-from-scratch.git
cd LLM-from-scratch
py -3.12 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip checkTERMINAL / Offline smoke testNo Hugging Face download
python transformer_from_scratch.py --smoke-testTERMINAL / Train on Yelp subsetUses pre-trained tokenizer and embeddings
python transformer_from_scratch.py --train --sdpaTERMINAL / Extended Yelp experimentResource-intensive preset
python transformer_from_scratch.py --full --sdpaTERMINAL / Custom training optionsPowerShell backtick line continuation
python transformer_from_scratch.py --train --sdpa `
--train-size 2000 --val-size 300 --test-size 300 `
--batch-size 4 --max-len 128 --epochs 3 `
--n-layers 3 --n-heads 6 --lr 0.0002TERMINAL / Optional architecture exportRequires Graphviz executable to render
python -m pip install torchview graphviz
python transformer_from_scratch.py --smoke-test --visualizeEVALUATION & REPRODUCIBILITY
Measure more than training loss.
The implementation separates training, validation and test data. It selects the best checkpoint by validation accuracy, reloads that checkpoint and reports held-out test accuracy.
Training mechanics
Cross-entropy classification loss is optimized with AdamW. Gradient norms are clipped and a seed is set for repeatability. Training and validation losses and accuracies are printed after each epoch.
- OptimizerAdamW
- LossCross-entropy
- Gradient clippingMax norm 1.0
- Best checkpoint criterionValidation accuracy
- Test evaluationAfter checkpoint reload
Validation status & scope
The project README reports a completed offline smoke test and saved checkpoint. It does not provide a standardized external benchmark or independently reproducible performance record in this website.
- Offline synthetic exerciseDocumented as completed
- Yelp benchmark scoreNot asserted
- GPU throughput / VRAMHardware-dependent
- Production deploymentNot claimed
- Model outputBinary sentiment logits
Reproducibility note: The checkpoint saves the trained model state and configuration metadata, not the full frozen MiniLM embedding matrix. To reproduce real-data inference, load the matching tokenizer and embedding weights identified in the metadata.
READ THE SOURCE
Every layer is inspectable.
Review the code, execution instructions, dependencies, and source documentation on GitHub. The wiki navigation is linked, but its separate content was not independently retrieved for this page.
Source repository
Explore the project structure, revision history and complete implementation.
View repository ↗PyTorch implementationAttention projections, encoder layers, pooling, data loading and training loop.
Read Python source ↗README & setupFull installation instructions, CLI flags, experiment presets and troubleshooting.
Open README ↗Python dependenciesPackage requirements for the smoke test and real-data training modes.
View requirements ↗GitHub WikiProject wiki navigation and any published architecture or reference pages.
Open wiki ↗AI engineering portfolioExplore related work in retrieval, agent workflows, model training and inference.
Explore portfolio ↗