Knowledge Distillation: Teacher-Student Compression
Knowledge distillation compresses big teacher models into small fast students using soft targets — the key to efficient edge AI.
Search across AI research reports and briefs. Results load instantly — no JavaScript required.
9 results for “efficiency”
Knowledge distillation compresses big teacher models into small fast students using soft targets — the key to efficient edge AI.
QLoRA fine-tunes 65B models on one 48GB GPU by combining 4-bit quantization with LoRA adapters — nearly matching full-precision tuning.
Mixtral routes each token to 2 of 8 experts, matching LLaMA-2 70B quality with 6x faster inference — open sparse MoE that actually works.
Mamba replaces attention with selective state spaces that scale linearly in sequence length — matching transformers at several times the inference speed.
LLaMA showed a 13B model trained on a trillion tokens could beat GPT-3 175B, reframing the scaling debate around compute-optimal token budgets.
The Switch Transformer scaled to a trillion parameters with simplified mixture-of-experts routing, activating one expert per token for efficient sparsity.
FlashAttention tiles exact attention in fast on-chip memory, cutting memory from quadratic to linear and making long-context transformers practical.
LoRA adapts large models by training small low-rank matrices while freezing the base weights — fine-tuning quality at a fraction of the cost.
A synthetic proposal for energy-aware MoE routing that trades a little accuracy for large power savings in sparse expert models.