AI-generated review 7 min read Intermediate 1,084 words

LLaMA: Open and Efficient Foundation Language Models

LLaMA showed a 13B model trained on a trillion tokens could beat GPT-3 175B, reframing the scaling debate around compute-optimal token budgets.

large-language-models scaling-laws training open-models efficiency
Foundations & scaling · 2023

Primary source: LLaMA: Open and Efficient Foundation Language Models (arXiv:2302.13971)

Through 2022 the story of language models was a story about parameter counts. GPT-3 had 175 billion, PaLM had 540 billion, and the implicit assumption was that capability tracked size. Training budgets were spent on bigger models, and most of those models were closed.

LLaMA argued with the arithmetic. Given a fixed compute budget, the paper trained smaller models on far more tokens rather than larger models on fewer, and released four model sizes — 7B, 13B, 33B and 65B parameters — to the research community. The result was a 13B model that outperformed GPT-3 on most benchmarks and a 65B model competitive with systems an order of magnitude larger. Size stopped being the only lever.

PARAMETERS VS TOKENS TRAINED (SCHEMATIC) model size (log scale, schematic) tokens trained GPT-3 175B · ~300B tokens PaLM 540B · ~780B tokens LLaMA-13B 13B · 1T tokens LLaMA-65B 65B · 1.4T tokens More tokens at a smaller size bought more capability per unit of compute.

Key takeaways

  • Four sizes were trained: 7B, 13B, 33B and 65B parameters, on roughly one to 1.4 trillion tokens of public data.
  • LLaMA-13B outperformed GPT-3 (175B) on most benchmarks despite being more than ten times smaller.
  • LLaMA-65B was reported as competitive with Chinchilla-70B and PaLM-540B on the same benchmark suite.
  • The architecture is a largely standard decoder-only Transformer with three deliberate changes: pre-normalisation with RMSNorm, the SwiGLU activation, and rotary positional embeddings.
  • The models were trained on publicly available corpora and released under a non-commercial research licence, which made them widely used but not unrestricted open source.

The compute-optimal argument

The paper's premise comes from scaling research rather than from architecture. Earlier scaling laws related loss to model size, dataset size and compute, but the practical reading of them had been to grow models while keeping data roughly fixed. Later work on compute-optimal training argued that for a given compute budget, most large models were badly under-trained on tokens.

LLaMA takes that conclusion literally. Instead of asking how large a model can be trained, it asks how many tokens can be squeezed through a budget, then sets the number of parameters accordingly. The headline comparison — a 13B model beating a 175B one — is the visible consequence of spending the same compute on many more tokens. It also changes what "big" means operationally: inference cost scales with parameters, so a model that is smaller and better trained is cheaper to serve at equal quality, which is precisely the trade-off that made the family practical for researchers.

Architecture: three changes from the GPT-3 recipe

LLaMA is a decoder-only Transformer, and the paper is explicit that most of it follows established practice. Three modifications are worth naming because they became defaults in later open models.

Normalisation is applied before each sub-layer rather than after, using RMSNorm, which drops the mean-centring term of LayerNorm and is cheaper. The feed-forward block uses SwiGLU, a gated variant that consistently improved quality in earlier comparisons. And position information is carried by rotary positional embeddings, which rotate query and key vectors rather than adding positional vectors, and which had been shown to behave better for long sequences.

The training setup is AdamW with a cosine learning-rate schedule, weight decay, gradient clipping, and a context length of 2048 tokens. Bias terms are removed from the linear layers. None of these choices is exotic on its own; their combination and the token budget are the contribution.

Evaluation and what it displaced

The reported results cover common-sense reasoning, closed-book question answering, reading comprehension, mathematical reasoning and code generation. The pattern across them is what mattered: on most tasks the smaller LLaMA models matched or exceeded far larger closed models, and the 65B model sat alongside the strongest publicly described systems of the time. Instruction following and dialogue were explicitly out of scope, which is why a wave of later work — instruction tuning, preference optimisation and adapters — attached itself to these base checkpoints.

That combination of a strong base model, a research-friendly release and a size that fits on accessible hardware is why LLaMA became an ecosystem rather than a single artifact. The follow-on work that fine-tuned it, quantised it and specialised it is arguably a larger contribution than the benchmark table.

Limitations and open questions

The paper flags several. Benchmark contamination is the sharpest: because the training corpora include web text and code repositories, some evaluation examples may have appeared in training, which inflates scores in ways that are difficult to quantify after the fact. The data pipeline relies on public corpora whose exact provenance and licensing are not fully resolvable, and the models inherit the biases present in that text.

The licence is a limitation in itself. "Open" here meant research access under a non-commercial agreement, not an open-source release, and the gap between the two caused sustained confusion. The models also ship without any safety tuning or alignment step, so they will follow harmful prompts; the paper presents them as base models for research, and treats instruction-following as future work. Finally, the results are a snapshot of a benchmark suite, not a claim about capabilities the benchmarks do not measure.

Further reading

References

  1. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971.
  2. Hoffmann, J., Borgeaud, S., Mensch, A., et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556.
  3. Kaplan, J., McCandlish, S., Henighan, T., et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.
  4. Shazeer, N. (2020). GLU Variants Improve Transformer. arXiv:2002.05202.
  5. Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864.

Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.

Cite this report
@misc{knowyourml-llama-open-efficient-language-models
  title        = {LLaMA: Open and Efficient Foundation Language Models},
  author       = {{Know Your ML}},
  year         = {2023},
  howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=llama-open-efficient-language-models}},
  note         = {AI-generated review of arXiv:2302.13971. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit — decision matrix, scorecards and trade-off analysis
Toolkit 7 decision tools Model-agnostic

Machine Learning Decision Toolkit

Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.

$24 View on Gumroad

References

Frequently asked questions

What made LLaMA different from previous large language models?

LLaMA was trained on far more tokens than models of comparable size, following compute-optimal scaling research. That let a 13-billion-parameter model outperform GPT-3 at 175 billion parameters on most benchmarks, and made strong language modelling feasible on a single research group's compute budget.

Was LLaMA actually open source?

No. The weights were released to the research community under a non-commercial research licence, with access granted on request rather than as an unrestricted open-source release. The distinction mattered legally even though the weights circulated widely.

How large were the LLaMA models and how much data did they see?

Four sizes were trained — 7B, 13B, 33B and 65B parameters — on between roughly one trillion and 1.4 trillion tokens drawn from public corpora including CommonCrawl, C4, GitHub, Wikipedia, books, arXiv and Stack Exchange.

How to Remain Valuable When Intelligence Becomes Cheap — book cover
eBook 224 pages

How to Remain Valuable When Intelligence Becomes Cheap

This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?

Read the eBook