Neural Radiance Fields (NeRFs): The Next Frontier in Deep Learning

Neural Radiance Fields (NeRFs) have emerged as one of the most exciting topics in the realm of deep learning and computer graphics. Since their introduction in 2020, the field has witnessed an explosion of research, as evidenced by the numerous submissions to CVPR 2022. Time magazine even recognized a variation of NeRFs, known as Instant Neural Graphics Primitives, as one of the best inventions of 2022. But what exactly are NeRFs, and how do they work? This article aims to demystify the various terminologies associated with neural fields, explore their functionality, and discuss their applications.

What is a Neural Field?

The term "neural field" was popularized by Xie et al. and refers to a neural network that parameterizes a signal. While this signal often represents a single 3D scene or object, neural fields can also be used for various types of signals, including audio and images. Their most prominent applications lie in computer graphics, particularly in image synthesis and 3D reconstruction.

Neural fields can be applied in diverse areas such as generative modeling, 2D image processing, robotics, medical imaging, and audio parameterization. Typically, fully connected neural networks encode the properties of objects or scenes, with a single network trained to capture the characteristics of a specific scene. Unlike standard machine learning approaches, the goal here is to overfit the neural network to a particular scene, effectively embedding the scene’s information into the network’s weights.

Why Use Neural Fields?

Traditional methods for storing 3D scenes often rely on voxel grids or polygon meshes. Voxel grids can be expensive to store, while polygon meshes are limited to representing hard surfaces, making them unsuitable for applications like medical imaging. Neural fields, on the other hand, offer a compact and efficient representation of 3D objects or scenes. They are continuous and differentiable, allowing for arbitrary dimensions and resolutions. Moreover, they are domain-agnostic, meaning they can adapt to various tasks without being tied to specific input formats.

What Do Fields Stand For?

In physics, a field is a quantity defined across all spatial and/or temporal coordinates, represented as a mapping from a coordinate ( x ) to a quantity ( y ). This can include scalars, vectors, or tensors, such as gravitational or electromagnetic fields.

Steps to Train a Neural Field

The process of computing neural fields typically follows these steps:

1. Sample coordinates of a scene.
2. Feed them into a neural network to produce field quantities.
3. Sample the field quantities from the desired reconstruction domain.
4. Map the reconstruction back to the sensor domain (e.g., 2D RGB images).
5. Calculate the reconstruction error and optimize the neural network.

Mathematically, this can be expressed as a neural field ( \Phi: X \rightarrow Y ) that maps world coordinates ( x{recon} \in X ) to field quantities ( y{recon} \in Y ). The sensor observation is also a neural field ( \Omega: S \rightarrow T ) that transforms sensor coordinates ( x{sens} \in S ) into measurements ( t{sens} \in T ). The forward mapping ( F: (X \rightarrow Y) \rightarrow (S \rightarrow T) ) is differentiable, allowing for optimization.

Neural Radiance Fields (NeRFs) for View Synthesis

NeRFs are a specific architecture within the broader category of neural fields, designed to tackle the problem of view synthesis. This task involves generating a 3D object or scene from a set of images taken from different angles, effectively achieving 3D reconstruction.

To understand NeRFs, one must grasp several computer graphics concepts, including volumetric rendering and ray casting. For those interested in a structured introduction to computer graphics, the Computer Graphics course by UC San Diego is highly recommended.

NeRFs Explained

NeRFs, as proposed by Mildenhall et al., accept a continuous 5D coordinate as input, which includes a spatial location ( (x, y, z) ) and a viewing direction ( (\theta, \phi) ). This input is fed into a multi-layer perceptron (MLP), which outputs the corresponding color intensities ( c = (r, g, b) ) and volume density ( \sigma ).

The volume density indicates how much radiance accumulates along a ray passing through a point in space, serving as a measure of that point’s contribution to the overall scene. This capability allows NeRFs to capture various lighting effects, such as reflections and transparencies, making them superior to traditional voxel grids or meshes.

Training NeRFs

Training NeRFs poses a challenge because the target density and color are unknown. A differentiable method is required to map these outputs back to 2D images, which are then compared with ground truth images to formulate a rendering loss for network optimization.

Volume rendering is employed to map the neural field output back to 2D images. During this process, rays are emitted at each pixel, and samples are taken at different timesteps—a technique known as ray marching. Each sample point possesses a spatial location, color, and volume density, which serve as inputs to the neural field.

To generate images, the rays are integrated, allowing for the color of each pixel to be computed. This integration can be expressed mathematically, incorporating the transmittance ( T(t) ), which measures how much a ray can penetrate the 3D space.

A Few More Notes on NeRFs

The landscape of NeRFs is rapidly evolving, with numerous variations and enhancements emerging. These typically fall into four categories:

1. Improving reconstruction by computing effective priors over 3D scenes and conditioning the neural fields.
2. Enhancing training and inference performance through hybrid representations that combine neural fields with discrete data structures.
3. Selecting better network architectures to eliminate spectral bias and efficiently compute derivatives and integrals.
4. Manipulating neural field representations.

Given the vast number of NeRF-related papers, it can be challenging to curate a comprehensive survey. However, resources like Dellaert et al. and Xie et al. provide valuable insights, along with a database of related papers and a Twitter account dedicated to neural fields.

Instant Neural Graphics Primitives with a Multiresolution Hash Encoding

One of the most significant advancements following NeRFs is the paper "Instant Neural Graphics Primitives" by Müller et al. from Nvidia. This work dramatically reduces training time from hours to mere seconds by introducing a novel input representation known as multiresolution hash encoding.

This encoding allows for smaller neural networks, minimizing the total floating-point operations required. The authors also propose specific GPU implementations for various tasks, further reducing computational complexity.

Multiresolution Hash Encoding

In this encoding scheme, both network parameters and encoding parameters (feature vectors) are trained. These vectors are organized into different resolution levels and stored at the vertices of a grid, with each grid corresponding to a different resolution.

For a specific location in a 2D image, the surrounding grids are identified, and indices are assigned to the vertices by hashing their coordinates. This allows for quick lookups of the corresponding trainable feature vectors. By linearly interpolating these vectors and concatenating them with other inputs, the final vector is produced and passed into the neural network.

This approach is fully differentiable, enabling the training of encoding parameters alongside the network. The benefits include improved final result quality, automatic levels of detail, and a task-agnostic encoding process.

Conclusion

NeRFs represent one of the most exciting applications of neural networks in recent years. The ability to render 3D models in seconds was unimaginable just a few years ago, and we are on the brink of seeing these architectures revolutionize industries such as gaming and simulation.

For those interested in experimenting with NeRFs, I recommend visiting the instant-ngp repository by Nvidia to create your own models. If you would like to see more articles on computer graphics, please let us know on our Discord server. Lastly, if you enjoy our blog posts, consider supporting us by purchasing our courses or books.

References

• Mildenhall, B., et al. "Neural Radiance Fields for View Synthesis."
• Müller, T., et al. "Instant Neural Graphics Primitives with a Multiresolution Hash Encoding."
• Xie, L., et al. "Neural Fields: A Survey."

Disclosure: Some links above may be affiliate links, and at no additional cost to you, we may earn a commission if you decide to make a purchase after clicking through.