NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
An AI-generated review of NeRF, the 2020 method that represents a 3D scene as a neural field and renders photorealistic novel views from sparse input photos.
Primary source: NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (arXiv:2003.08934)
Classical view synthesis — generating a new photo of a scene from a different camera angle — required either a 3D reconstruction with texture mapping or heavy interpolation between many input views. NeRF, introduced by Mildenhall et al. in 2020, replaced the scene model itself with a neural network: a neural radiance field mapping 3D coordinates to color and density. From just a few dozen photos, NeRF renders photorealistic novel views — including view-dependent effects like specular highlights — and launched the modern neural rendering field.
The core idea
Represent the scene as a function F: (x, y, z, θ, φ) → (RGB, σ), where (x,y,z) is a 3D point, (θ,φ) the viewing direction, RGB the emitted color, and σ the volume density. This function is an MLP. To render a pixel:
- Cast a ray through the pixel into the scene.
- Sample points along the ray and query the MLP at each.
- Composite the sampled colors using volume rendering (alpha compositing weighted by density), producing the pixel's color.
The MLP is optimized to match the input photos: for each training image, render it and minimize the difference from the observed pixels. After training, the network is the scene, and any novel view is rendered by casting rays through the new camera.
How it works
- Input conditioning: 5D input (position + viewing direction). Density σ depends only on position; color depends on both position and direction, which lets the model represent view-dependent effects (speculars, reflections).
- Positional encoding: coordinates are mapped through high-frequency sinusoidal features before entering the MLP; without this, the network can only represent low-frequency functions and renders blurry scenes. This is the paper's most-cited trick.
- Hierarchical sampling: two networks — a coarse one samples uniformly, then a fine one concentrates samples where the coarse model predicts high density — improving efficiency and detail.
- Training data: 20-100 posed photographs of a scene (posed via COLMAP); optimization takes hours on a single GPU.
- Results: photorealistic novel views on synthetic objects and real forward-facing scenes, clearly better than the previous best methods (LFF, SRN, LLFF) on PSNR and LPIPS.
Why it matters
- A new scene representation. NeRF replaced explicit geometry with an implicit function, showing that differentiable rendering through a network can do what mesh pipelines could not.
- Novel view synthesis at photorealistic quality from sparse, casually captured photos — previously impossible.
- The neural-fields paradigm. NeRF spawned an enormous family: dynamic NeRFs (video), editable variants, 3D Gaussian Splatting (a later, faster approach inspired by the same rendering formulation), and neural implicit surfaces (NeuS, Instant-NGP).
- Bridging ML and graphics. Differentiable volume rendering became a standard primitive, connecting computer vision and computer graphics research.
Limitations and open questions
NeRF training is slow (hours per scene) and per-scene: a new network is required for every new scene, with no generalization without further work (generalizable variants came later). Rendering is also slow — hundreds of MLP evaluations per pixel — and the method assumes static scenes with no occlusions handled explicitly. Handling dynamic scenes, reflections, and unbounded outdoor scenes required substantial extensions, later accelerated by hash-grid encodings (Instant-NGP) and Gaussian splatting.
Key takeaways
- NeRF = MLP mapping (position, direction) → (color, density), rendered via differentiable volume integration.
- High-frequency positional encoding is the key to sharp, detailed reconstructions.
- Photorealistic novel views from tens of photos launched neural rendering as a field.
- Its successors (Instant-NGP, 3D Gaussian Splatting, generalizable NeRFs) made the approach fast and practical.
Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.
Cite this report
@misc{knowyourml-nerf-representing-scenes-as-neural-radiance-fields
title = {NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis},
author = {{Know Your ML}},
year = {2020},
howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=nerf-representing-scenes-as-neural-radiance-fields}},
note = {AI-generated review of arXiv:2003.08934. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit
Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.
References
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (arXiv:2003.08934) — https://arxiv.org/abs/2003.08934
Beyond the paper
How to Remain Valuable When Intelligence Becomes Cheap
This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?