12 reports · 2014–2023

Architectures

The network designs that mattered: attention, convolutions, residual and graph models, sequence-to-sequence and their descendants.

Fig. 02 / Field notesA small attention experiment
An attention instrument: five query keys, a five-by-five weight matrix and five output faders. The selected query is learn.
Five query keys, one matrix, five weights.

Sequence to Sequence Learning with Neural Networks

Architectures · 7 min read · Beginner

The Sutskever sequence-to-sequence LSTM encoded sentences to a vector and decoded translations — the encoder-decoder pattern behind neural translation.

seq2seqlstmmachine-translation Read

ResNet: Deep Residual Learning for Image Recognition

Architectures · 7 min read · Beginner

An AI-generated review of ResNet, the 2015 architecture that used skip connections to train networks hundreds of layers deep and became the default backbone.

cnnresidual-connectionscomputer-vision Read

The Transformer: Attention Without Recurrence

Architectures · 5 min read · Beginner

The Transformer replaced recurrence with self-attention, enabling massively parallel training — the architecture behind GPT, BERT and modern LLMs.

transformersattentionself-attention Read

XLNet: Generalized Autoregressive Pretraining

Architectures · 5 min read · Intermediate

XLNet combines permutation language modeling with Transformer-XL recurrence, outperforming BERT on SQuAD, GLUE and RACE benchmarks.

xlnetpretrainingtransformers Read

RoPE: Rotary Position Embeddings

Architectures · 5 min read · Intermediate

Rotary position embeddings encode token order as rotations in embedding space, giving transformers relative positions that stretch to longer text.

position-embeddingsropetransformers Read

Causal Attention for Interpretable Multi-Step Reasoning

Architectures · 17 min read · Advanced

A synthetic proposal for causal attention: separating correlational co-occurrence from causal influence so transformers can reason over interventions.

causal-inferenceattentioninterpretability Read

Mamba: Selective State Spaces

Architectures · 5 min read · Advanced

Mamba replaces attention with selective state spaces that scale linearly in sequence length — matching transformers at several times the inference speed.

mambastate-space-modelslong-context Read

Videos on Architectures

Narrated films that touch this topic, with chapters, captions and transcripts.

All videos