18 architecture families, side by side.
zigllm implements 18 transformer families not to run all of them fast, but to make their differences legible. Read them as a comparative study: the same primitives from Layer 3 and Layer 4, rearranged. Notes below summarise what sets each apart.
GPT lineage
The original decoder-only stack and its early descendants — where learned positional embeddings and classic pre-/post-norm placement come from.
GPT-2
Learned absolute positions, LayerNorm, tied embeddings — the reference decoder-only model.
GPT-J
Parallel attention + FFN and rotary embeddings on a GPT-style backbone.
GPT-NeoX
Scaled GPT-J-style design with rotary positions; the Pythia family lives here.
LLaMA family
The modern default: RMSNorm, RoPE, SwiGLU FFNs, and grouped-query attention. Most current open models are variations on this theme.
LLaMA
Pre-norm RMSNorm, RoPE, SwiGLU — the template the rest of the family refines.
Mistral
LLaMA-like with sliding-window attention and grouped-query attention.
Qwen
LLaMA-family backbone with its own tokenizer and bias/attention tweaks.
Phi
Small, data-efficient models on a compact decoder-only design.
Norm & scaling variants
Families that move the norms around or scale the embeddings — good for seeing how small architectural choices change the forward pass.
Gemma
Embedding scaling and additional normalization placement distinguish it from LLaMA.
Falcon
Parallel attention/FFN and multi-query attention for throughput.
BLOOM
ALiBi positional bias instead of rotary embeddings; multilingual training.
StarCoder
Code-focused decoder with multi-query attention and a large context.
Beyond the standard decoder
Where the architecture stops being "just a decoder" — encoders, state-space models, sparse experts, and multiple modalities.
BERT
Encoder-only, bidirectional attention — a contrast that makes the decoder design clearer.
Mamba
A state-space model: linear-time sequence mixing instead of quadratic attention.
MoE
Mixture-of-experts routing — only a subset of FFN experts fire per token.
Multimodal
Vision/other encoders feeding a language backbone.