The Irishman, which is nominated for the Visual Effects category in Oscar 2020, is a reference for pushing the boundaries between human creativity and technology potential. Read below how a unique camera rig (RED Helium between two ARRI Minis) has manipulated aging in The Irishman. 

“Time machine” generated by Artificial Intelligence powered by NVIDIA 

From World War II to a nursing home in the 2000s, and every decade in between, Netflix’s The Irishman tells the tale of hitman Frank Sheeran through scenes from different times in his life. But all three leads in the film — Robert DeNiro, Al Pacino, and Joe Pesci — are in their 70s. A makeup department couldn’t realistically transform the actors back to their 20s and 30s. And director Martin Scorcese was against using the typical motion capture markers or other intrusive equipment that gets in the way of raw performances during filming. To meet this requirement, ILM (Industrial Light And Magic) has developed a new three-camera rig to capture the actors’ performances on set, using the director’s camera (RED Helium) flanked by two infrared cameras (ARRI Mini) to record 3D geometry and textures.

The team also developed software called ILM Facefinder that used AI to sift through thousands of images of the actors’ past performances. The tool located frames that matched the camera angle, framing, lighting, and expression of the scene being rendered, giving ILM artists a relevant reference to compare against every frame in the shot. These visual references were used to refine digital doubles created for each actor so that they could be transformed into the target age for each specific scene in the film. NVIDIA Quadro RTX GPUs were utilized to accelerate production. “AI and machine learning are becoming a part of everything we do in VFX,” said Pablo Helman, VFX supervisor on The Irishman at ILM. “Paired with the NVIDIA Quadro RTX GPUs powering our production pipeline, these technologies have us excited for what the next decade will bring.” As for the camera rig, read the description below.

Paired with the NVIDIA Quadro RTX GPUs powering our production pipeline, these technologies have us excited for what the next decade will bring

Pablo Helman, VFX supervisor on The Irishman at ILM

A dedicated rig: RED Helium surrounded by ARRI Minis

It’s important to emphasize that this AI-aging technology was created while the film is being shot. In order to get a precise execution of the aging process, the VFX team had to take into consideration many parameters, like performance capture and lighting acquisition, and most importantly, avoiding any interference with directing and acting. Thus a dedicated system had to be built. The rig was designed from a camera that was surrounded by two “witness” cameras, which are infrared. Every one of the cameras gave different points of view of specific performance. The crew called this rig a Three-Headed Monster.

Three-Headed Monster

As explained, the rig was designed by ILM in collaboration with cinematographer Rodrigo Prieto, ASC, AMC, and was composed of three-headed cameras. Each rig held a primary RED Helium camera along with two ARRI Minis as the “witness” cameras —  altered for infrared cinematography. The three camera views were fed into ILM’s proprietary Flux software, which turned them into 3D representations of the actors’ facial performances. The resulting performance data was applied to 3D models of each actor, creating a digital version of the performances captured on set that could be retargeted to younger likenesses of the actors’ faces ILM had prepared. This VFX rig was finessed through a series of trial and error and was created during the shooting.

The “witness” cameras have mounted either side of the RED DSMC2 director’s camera (fitted with Cooke Panchro and ARRI Ultra-speed lenses) to provide stereo views of the actor’s face

ARRI Minis as the Witness cameras

The “witness” cameras have mounted either side of the RED DSMC2 director’s camera (fitted with Cooke Panchro and ARRI Ultra-speed lenses) to provide stereo views of the actor’s face. Each witness camera was modified by having its infrared (IR) filter removed so that the sensor only recorded IR light. A ring of LEDs around the lens also threw infrared light onto the actors’ faces to neutralize unintended shadows, while remaining invisible to the production camera and the actors.

According to RED Digital Cinema, this three-camera set up itself had to be sufficiently lightweight to be carried on any kind of rig, crane, or head that Prieto or Scorsese wanted. There could be no restrictions. That included the ability to remove one of the witness cameras and fit it to the top of the RED camera if the shot dictated, for example, shooting up against a column. Even then, the three-camera rig proved too heavy for Steadicam, so this necessitated a two-camera version with the second witness camera positioned independently to complete the triangulation of the actor in 3D space. They also ensured that the width of the rig could pass through standard size doorframes.

Watch the detailed video below, which sheds some more light regarding the process.

Discussion: High-tech to allow filmmaking traditionally

That is the first time we’re witnessing this kind of technology to bypass the typical motion capture markers or other interfering methods that gets in the way of raw filmmaking and acting. Buiding this three-headed monster is one of the most out-of-the-box ideas I’ve ever seen in filmmaking, and the results are stunning. Go and watch The Irishman so you can appreciate this creative process, that the idea behind it is to utilize technology to allow pure and traditionally filmmaking.