6.4 Realistic Cameras

The thin lens model makes it possible to render images with blur due to depth of field, but it is a fairly rough approximation of actual camera lens systems, which are comprised of a series of multiple lens elements, each of which modifies the distribution of radiance passing through it. (Figure 6.15 shows a cross section of a 22-mm focal length wide-angle lens with eight elements.) Even basic cell phone cameras tend to have on the order of five individual lens elements, while DSLR lenses may have ten or more. In general, more complex lens systems with larger numbers of lens elements can create higher quality images than simpler lens systems.

This section discusses the implementation of RealisticCamera, which simulates the focusing of light through lens systems like the one in Figure 6.15 to render images like Figure 6.16. Its implementation is based on ray tracing, where the camera follows ray paths through the lens elements, accounting for refraction at the interfaces between media (air, different types of glass) with different indices of refraction, until the ray path either exits the optical system or until it is absorbed by the aperture stop or lens housing. Rays leaving the front lens element represent samples of the camera’s response profile and can be used with integrators that estimate the incident radiance along arbitrary rays, such as the SamplerIntegrator. The RealisticCamera implementation is in the files cameras/realistic.h and cameras/realistic.cpp.

In addition to the usual transformation to place the camera in the scene, the Film, and the shutter open and close times, the RealisticCamera constructor takes a filename for a lens system description file, the distance to the desired plane of focus, and a diameter for the aperture stop. The effect of the simpleWeighting parameter is described later, in Section 13.6.6, after preliminaries related to Monte Carlo integration in Chapter 13 and the radiometry of image formation in Section 6.4.7.

After loading the lens description file from disk, the constructor adjusts the spacing between the lenses and the film so that the plane of focus is at the desired depth, focusDistance, and then precomputes some information about which areas of the lens element closest to the film carry light from the scene to the film, as seen from various points on the film plane. After background material has been introduced, the fragments <<Compute lens–film distance for given focus distance>> and <<Compute exit pupil bounds at sampled points on the film>> will be defined in Sections 6.4.4 and 6.4.5, respectively.

6.4.1 Lens System Representation

A lens system is made from a series of lens elements, where each element is generally some form of glass. A lens system designer’s challenge is to design a series of elements that form high-quality images on a film or sensor subject to limitations of space (e.g., the thickness of mobile phone cameras is very limited in order to keep phones thin), cost, and ease of manufacture.

It’s easiest to manufacture lenses with cross sections that are spherical, and lens systems are generally symmetric around the optical axis, which is conventionally denoted by . We will assume both of these properties in the remainder of this section. Lens systems are defined using a coordinate system where the film is aligned with the plane and lenses are to the left of the film, along the axis.

Lens systems are commonly represented in terms of the series of interfaces between the individual lens elements (or air) rather than having an explicit representation of each element. Table 6.1 shows the quantities that define each interface. The last entry in the table defines the rightmost interface, which is shown in Figure 6.17: it’s a section of a sphere with radius equal to the curvature radius. The thickness of an element is the distance along  to the next element to the right (or to the film plane), and the index of refraction is for the medium to the right of the interface. The element’s extent above and below the axis is set by the aperture diameter.

The LensElementInterface structure represents a single lens element interface.

The fragment <<Load element data from lens description file>>, not included here, reads the lens elements and initializes the RealisticCamera::elementInterfaces array. See comments in the source code for details of the file format, which parallels the structure of Table 6.1, and see the directory scenes/lenses in the pbrt distribution for a number of example lens descriptions.

Two adjustments are made to the values read from the file: first, lens systems are traditionally described in units of millimeters, but pbrt assumes a scene measured in meters. Therefore, the fields other than the index of refraction are scaled by . Second, the element’s diameter is divided by two; the radius is a more convenient quantity to have at hand in the code to follow.

Once the element interface descriptions have been loaded, it’s useful to have a few values related to the lens system easily at hand. LensRearZ() and LensFrontZ() return the depths of the rear and front elements of the lens system, respectively. Note that the returned depths are in camera space, not lens space, and thus have positive values.

Finding the front element’s position requires summing all of the element thicknesses (see Figure 6.18). This value isn’t needed in any code that is in a performance-sensitive part of the system, so recomputing it when needed is fine. If performance of this method was a concern, it would be better to cache this value in the RealisticCamera.

RearElementRadius() returns the aperture radius of the rear element in meters.

6.4.2 Tracing Rays through Lenses

Given a ray starting from the film side of the lens system, TraceLensesFromFilm() computes intersections with each element in turn, terminating the ray and returning false if its path is blocked along the way through the lens system. Otherwise it returns true and initializes *rOut with the exiting ray in camera space. During traversal, elementZ tracks the intercept of the current lens element. Because the ray is starting from the film, the lenses are traversed in reverse order compared to how they are stored in elementInterfaces.

Because the camera points down the axis in pbrt’s camera space but lenses are along , the components of the origin and direction of the ray need to be negated. While this is a simple enough transformation that it could be applied directly, we prefer an explicit Transform to make the intent clear.

Recall from Figure 6.18 how the intercept of elements is computed: because we are visiting the elements from back-to-front, the element’s thickness must be subtracted from elementZ to compute its intercept before the element interaction is accounted for.

Given the element’s axis intercept, the next step is to compute the parametric value along the ray where it intersects the element interface (or the plane of the aperture stop). For the aperture stop, a ray–plane test (following Section 3.1.2) is used. For spherical interfaces, IntersectSphericalElement() performs this test and also returns the surface normal if an intersection is found; the normal will be needed for computing the refracted ray direction.

The IntersectSphericalElement() method is generally similar to Sphere::Intersect(), though it’s specialized for the fact that the element’s center is along the axis (and thus, the center’s and components are zero). The fragments <<Compute t0 and t1 for ray–element intersection>> and <<Compute surface normal of element at ray intersection point>> aren’t included in the text here due to their similarity with the Sphere::Intersect() implementation.

There is, however, a subtlety in choosing which intersection point to return: the closest intersection with isn’t necessarily on the element interface; see Figure 6.19. ^† For example, for a ray approaching from the scene and intersecting a concave lens (with negative curvature radius), the farther of the two intersections should be returned regardless of whether the closer one has . Fortunately, simple logic based on the ray direction and the curvature radius indicates which value to use.

Each lens element extends for some radius around the optical axis; if the intersection point with the element is outside this radius, then the ray will actually intersect the lens housing and terminate. In a similar fashion, if a ray intersects the aperture stop, it also terminates. Therefore, here we test the intersection point against the appropriate limit for the current element, either terminating the ray or updating its origin to the current intersection point if it survives.

If the current element is the aperture, the ray’s path isn’t affected by traveling through the element’s interface. For glass (or, forbid, plastic) lens elements, the ray’s direction changes at the interface as it goes from a medium with one index of refraction to one with another. (The ray may be passing from air to glass, from glass to air, or from glass with one index of refraction to a different type of glass with a different index of refraction.)

Section 8.2 discusses how a change in index of refraction at the boundary between two media changes the direction of a ray and the amount of radiance carried by the ray. (In this case, we can ignore the change of radiance, as it cancels out if the ray is in the same medium going into the lens system as it is when it exits—here, both are air.) The Refract() function is defined in Section 8.2.3; note that it expects that the incident direction will point away from the surface, so the ray direction is negated before being passed to it. This function returns false in the presence of total internal reflection, in which case the ray path terminates. Otherwise, the refracted direction is returned in w.

In general, some light passing through an interface like this is transmitted and some is reflected. Here we ignore reflection and assume perfect transmission. Though an approximation, it is a reasonable one: lenses are generally manufactured with coatings designed to reduce the reflection to around of the radiance carried by the ray. (However, modeling this small amount of reflection can be important for capturing effects like lens flare.)

If the ray has successfully made it out of the front lens element, it just needs to be transformed from lens space to camera space.

The TraceLensesFromScene() method is quite similar to TraceLensesFromFilm() and isn’t included here. The main differences are that it traverses the elements from front-to-back rather than back-to-front. Note that it assumes that the ray passed to it is already in camera space; the caller is responsible for performing the transformation if the ray is starting from world space. The returned ray is in camera space, leaving the rear lens element toward the film.

6.4.3 The Thick Lens Approximation

The thin lens approximation used in Section 6.2.3 was based on the simplifying assumption that the lens system had 0 thickness along the optical axis. The thick lens approximation of a lens system is slightly more accurate in that it accounts for the lens system’s extent. After introducing the basic concepts of the thick lenses here, we’ll use the thick lens approximation to determine how far to place the lens system from the film in order to focus at the desired focal depth in Section 6.4.4.

The thick lens approximation represents a lens system by two pairs of distances along the optical axis—the focal points and the depths of the principal planes; these are two of the cardinal points of a lens system. If rays parallel to the optical axis are traced through an ideal lens system, all of the rays will intersect the optical axis at the same point—this is the focal point. (In practice, real lens systems aren’t perfectly ideal and incident rays at different heights will intersect the optical axis along a small range of values—this is the spherical aberration.) Given a specific lens system, we can trace rays parallel to the optical axis through it from each side and compute their intersections with the axis to find the focal points. (See Figure 6.20.)

Each principal plane is found by extending the incident ray parallel to the optical axis and the ray leaving the lens until they intersect; the depth of the intersection gives the depth of the corresponding principal plane. Figure 6.20 shows a lens system with its focal points and and principal planes at values and . (As in Section 6.2.3, primed variables represent points on the film side of the lens system, and unprimed variables represent points in the scene being imaged.)

Given the ray leaving the lens, finding the focal point requires first computing the value where the ray’s and components are zero. If the entering ray was offset from the optical axis only along , then we’d like to find such that . Thus,

In a similar manner, to find the for the principal plane where the ray leaving the lens has the same height as the original ray, we have , and so

Once these two values have been computed, the ray equation can be used to find the coordinates of the corresponding points.

The ComputeCardinalPoints() method computes the depths of the focal point and the principal plane for the given rays. Note that it assumes that the rays are in camera space but returns values along the optical axis in lens space.

The ComputeThickLensApproximation() method computes both pairs of cardinal points for the lens system.

First, we must choose a height along the axis for the rays to be traced. It should be far enough from so that there is sufficient numeric precision to accurately compute where rays leaving the lens system intersect the axis, but not so high up the axis that it hits the aperture stop on the ray through the lens system. Here, we use a small fraction of the film’s diagonal extent; this works well unless the aperture stop is extremely small.

To construct the ray from the scene entering the lens system rScene, we offset a bit from the front of the lens. (Recall that the ray passed to TraceLensesFromScene() should be in camera space.)

An equivalent process starting from the film side of the lens system gives us the other two cardinal points.

6.4.4 Focusing

Lens systems can be focused at a given depth in the scene by moving the system in relation to the film so that a point at the desired focus depth images to a point on the film plane. The Gaussian lens equation, (6.3), gives us a relation that we can solve to focus a thick lens.

For thick lenses, the Gaussian lens equation relates distances from a point in the scene at and the point it focuses to by

For thin lenses, , and Equation (6.1) follows.

If we know the positions and of the principal planes and the focal length of the lens and would like to focus at some depth along the optical axis, then we need to determine how far to translate the system so that

The focus point on the film side should be at the film, so , and , the given focus depth. The only unknown is , and some algebraic manipulation gives us

(There are actually two solutions, but this one, which is the closer of the two, gives a small adjustment to the lens position and is thus the appropriate one.)

FocusThickLens() focuses the lens system using this approximation. After computing , it returns the offset along the axis from the film where the lens system should be placed.

Equation (6.4) gives the offset . The focal length of the lens is the distance between the cardinal points and . Note also that the negation of the focus distance is used for , since the optical axis points along negative .

We can now finally implement the fragment in the RealisticCamera constructor that focuses the lens system. (Recall that the thickness of the rearmost element interface is the distance from the interface to the film.)

6.4.5 The Exit Pupil

From a given point on the film plane, not all rays toward the rear lens element will successfully exit the lens system; some will be blocked by the aperture stop or will intersect the lens system enclosure. In turn, not all points on the rear lens element transmit radiance to the point on the film. The set of points on the rear element that do carry light through the lens system is called the exit pupil; its size and position vary across viewpoints on the film plane. (Analogously, the entrance pupil is the area over the front lens element where rays from a given point in the scene will reach the film.)

Figure 6.21 shows the exit pupil as seen from two points on the film plane with a wide angle lens. The exit pupil gets smaller for points toward the edges of the film. An implication of this shrinkage is vignetting.

When tracing rays starting from the film, we’d like to avoid tracing too many rays that don’t make it through the lens system; therefore, it’s worth limiting sampling to the exit pupil itself and a small area around it rather than, for example, wastefully sampling the entire area of the rear lens element.

Computing the exit pupil at each point on the film plane before tracing a ray would be prohibitively expensive; instead the RealisticCamera implementation precomputes exit pupil bounds along segments of a line on the film plane. Since we assumed that the lens system is radially symmetric around the optical axis, exit pupil bounds will also be radially symmetric, and bounds for arbitrary points on the film plane can be found by rotating these segment bounds appropriately (Figure 6.22). These bounds are then used to efficiently find exit pupil bounds for specific film sample positions.

One important subtlety to be aware of is that because the lens system is focused by translating it along the optical axis, the shape and position of the exit pupil change when the focus of the lens system is adjusted. Therefore, it’s critical that these bounds be computed after focusing. ^†

The BoundExitPupil() method computes a 2D bounding box of the exit pupil as seen from a point along a segment on the film plane. The bounding box is computed by attempting to trace rays through the lens system at a set of points on a plane tangent to the rear lens element. The bounding box of the rays that make it through the lens system gives an approximate bound on the exit pupil—see Figure 6.23.

The implementation samples the exit pupil fairly densely—at a total of points for each segment. We’ve found this sampling rate to provide good exit pupil bounds in practice.

The bounding box of the rear element in the plane perpendicular to it is not enough to be a conservative bound of the projection of the exit pupil on that plane; because the element is generally curved, rays that pass through the plane outside of that bound may themselves intersect the valid extent of the rear lens element. Rather than compute a precise bound, we’ll increase the bounds substantially. The result is that many of the samples taken to compute the exit pupil bound will be wasted; in practice, this is a minor price to pay, as these samples are generally quickly terminated during the lens ray-tracing phase.

The sample point on the film is found by linearly interpolating between the interval endpoints. The RadicalInverse() function that is used to compute the interpolation offsets for the sample point inside the exit pupil bounding box will be defined later, in Section 7.4.1. There, we will see that the sampling strategy implemented here corresponds to using Hammersley points in 3D; the resulting point set minimizes gaps in the coverage of the overall 3D domain, which in turn ensures an accurate exit pupil bound estimate.

Now we can construct a ray from pFilm to pRear and determine if it is within the exit pupil by seeing if it makes it out of the front of the lens system. If so, the exit pupil bounds are expanded to include this point. If the sampled point is already inside the exit pupil’s bounding box as computed so far, then we can skip the lens ray tracing step to save a bit of unnecessary work.

It may be that none of the sample rays makes it through the lens system; this case can legitimately happen with some very wide-angle lenses where the exit pupil vanishes at the edges of the film extent, for example. In this case, the bound doesn’t matter and BoundExitPupil() returns the bound that encompasses the entire rear lens element.

While one sample may have made it through the lens system and one of its neighboring samples didn’t, it may well be that another sample very close to the neighbor actually would have made it out. Therefore, the final bound is expanded by roughly the spacing between samples in each direction in order to account for this uncertainty.

Given the precomputed bounds stored in RealisticCamera::exitPupilBounds, the SampleExitPupil() method can fairly efficiently find the bounds on the exit pupil for a given point on the film plane. It then samples a point inside this bounding box for the ray from the film to pass through. In order to accurately model the radiometry of image formation, the following code will need to know the area of this bounding box, so it is returned via sampleBoundsArea.

Given the pupil’s bounding box, a point inside it is sampled via linear interpolation with the provided lensSample value, which is in .

Because the exit pupil bound was computed from a point on the film along the axis but the point pFilm is an arbitrary point on the film, the sample point in the exit pupil bound must be rotated by the same angle as pFilm makes with the axis.

6.4.6 Generating Rays

Now that we have the machinery to trace rays through lens systems and to sample points in the exit pupil bound from points on the film plane, transforming a CameraSample into a ray leaving the camera is fairly straightforward: we need to compute the sample’s position on the film plane and generate a ray from this point to the rear lens element, which is then traced through the lens system.

The CameraSample::pFilm value is with respect to the overall resolution of the image in pixels. Here, we’re operating with a physical model of a sensor, so we start by converting back to a sample in . Next, the corresponding point on the film is found by linearly interpolating with this sample value over its area.

SampleExitPupil() then gives us a point on the plane tangent to the rear lens element, which in turn lets us determine the ray’s direction. In turn, we can trace this ray through the lens system. If the ray is blocked by the aperture stop or otherwise doesn’t make it through the lens system, GenerateRay() returns a 0 weight. (Callers should be sure to check for this case.)

If the ray does successfully exit the lens system, then the usual details have to be handled to finish its initialization.

The fragment <<Return weighting for RealisticCamera ray>> will be defined later, in Section 13.6.6, after some necessary background from Monte Carlo integration has been introduced.

6.4.7 The Camera Measurement Equation

Given this more accurate simulation of the process of real image formation, it’s also worthwhile to more carefully define the radiometry of the measurement made by a film or a camera sensor. Rays from the exit pupil to the film carry radiance from the scene; as considered from a point on the film plane, there is thus a set of directions from which radiance is incident. The distribution of radiance leaving the exit pupil is affected by the amount of defocus blur seen by the point on the film—Figure 6.24 shows two renderings of the exit pupil’s radiance as seen from two points on the film.

Given the incident radiance function, we can define the irradiance at a point on the film plane. If we start with the definition of irradiance in terms of radiance, Equation (5.4), we can then convert from an integral over solid angle to an integral over area (in this case, an area of the plane tangent to the rear lens element that bounds the exit pupil, ) using Equation (5.6). This gives us the irradiance for a point on the film plane:

Figure 6.25 shows the geometry of the situation.

Because the film plane is perpendicular to the exit pupil plane, . We can further take advantage of the fact that the distance between and is equal to the axial distance from the film plane to the exit pupil (which we’ll denote here by ) divided by . Putting this all together, we have

For cameras where the extent of the film is relatively large with respect to the distance , the term can meaningfully reduce the incident irradiance—this factor also contributes to vignetting. Most modern digital cameras correct for this effect with preset correction factors that increase pixel values toward the edges of the sensor.

Integrating irradiance at a point on the film over the time that the shutter is open gives fluence, which is the radiometric unit for energy per area, .

Measuring fluence at a point captures the effect that the amount of energy received on the film plane is partially related to the length of time the camera shutter is open.

Photographic film (or CCD or CMOS sensors in digital cameras) actually measure radiant energy over a small area. ^† Taking Equation (6.6) and also integrating over sensor pixel area, , we have

the Joules arriving at a pixel.

In Section 13.2, we’ll see how Monte Carlo can be applied to estimate the values of these various integrals. Then in Section 13.6.6 we will define the fragment <<Return weighting for RealisticCamera ray>> in RealisticCamera::GenerateRay(); various approaches to computing the weight allow us to compute each of these quantities. Section 16.1.1 defines the importance function of a camera model, which characterizes its sensitivity to incident illumination arriving along different rays.