The crown jewel of House of Moves is a new performance capture sound stage that features a 30'x50' full-body capture volume (stage), designed with a rig that contracts down to a 30'x30' volume size for tighter, detailed performance-capture requirements. The stage is capable of supporting up to 20 performers at once and can be outfitted with up to 270 Vicon motion-capture cameras. This stage is fully enclosed, with all supporting CPUs housed in a soundproof, glass-encased room so that clean dialogue can be recorded without any interfering background noise. In addition to the new stage, House of Moves also maintains a 40'x70' by 25'-high main stage, and it can manage clients working on both stages simultaneously, with or without first unit crews shooting alongside the motion-capture cameras. While Naughty Dog was filming actors Emily Rose and Nolan North for Uncharted 2 on the new sound stage last year, THQ had wrestlers performing moves for WWE Smackdown vs. RAW 2010 on the large stage.

According to Brian Rausch, vice president of production at Vicon House of Moves, videogames account for 60 to 70 percent of the company’s business. With the exception of movies like Avatar, A Christmas Carol, and The Lord of the Rings, most movies require a finite number of performance-capture days. Videogames tend to spend more time capturing the in-game action, as well as the cutscenes. Naughty Dog spent more than 50 days capturing actors’ performances for Uncharted 2.

“It’s safe to say that videogames are our breadwinner because we do hours and hours of capture for games over many weeks, compared to movies,” says Rausch.

“The cinematics that are happening in games like Uncharted 2 now are really amazing. The transition from cutscene to gameplay is seamless. Naughty Dog built this model with the first game and really took it to the next level with this sequel.”

In addition to the new performance capture studio, Rausch’s team created Velcro-less suits for the actors to wear while performing. This allows the actors to focus on their performances. The studio is set up to record voice along with facial-capture, full body-capture, and finger-capture simultaneously, although for most games and movies directors pick and choose what to focus on.

“We have real-time feedback for the actors and the director to see what’s being captured on screens throughout the studio,” Rausch says. “A lot of times, directors will focus more on the performance because that’s really what we’re trying to bring to life in the game. So, we have four different reference cameras that capture the action from different angles. We can hand off the video reference to an editor to cut the story together just like a live-action film. Then we have the audio crew and the motion-capture all together, and it’s all synched to time code so that when we actually deliver the package to the client, they can use all these different things to bring the game to life.”

According to Amy Hennig, creative director at Naughty Dog, Uncharted 2 would never have been possible to make without the level of performance capture afforded by House of Moves.

“The fact that we had the same actors doing the vocal performance and the motion-capture performance working together on stage over the course of this game’s creation allowed them to play off each others’ energies and each others’ ideas and improvise together,” says Hennig. “We were able to find all these happy accidents that we would never have discovered if we walked into a recording studio with a script and just recorded it with the actors never interacting, never getting to know each other. Performance-capture makes it much more fun for us, much more fun for them, and it makes the material much more organic and alive because it’s believable and authentic.”

Another aspect that brings the realism to life in new games like Uncharted 2 is the use of props, according to Taylor Kurosaki, the cinematics editor at Naughty Dog.  At House of Moves, Kurosaki’s team worked with physical wireframe sets that actors were able to perform on, such as a Jeep. The 3D model was loaded into the 3D space and then the physical props were calibrated to match up the real world action with the virtual environment. Actors were able to watch their avatars on a big screen as their actions and dialogue were captured simultaneously.

Far from the glitz of Hollywood, Quantic Dream has set up shop in the City of Lights to bring what the independent developer hopes will be the next evolution in performance-capture and interactive entertainment to PS3 gamers. The idea behind Heavy Rain started back in 2000, when David Cage, writer, director, and co-CEO of Quantic Dream, decided that performance-capture was the future of entertainment. The developer first created Indigo Prophecy for PlayStation 2 and Xbox, but as technology advanced, Cage was able to expand his studio and build an experience that will change depending on every choice one makes.

To call Heavy Rain epic is an understatement. To create the game, Quantic Dream built a performance-capture studio and employed a cast of 70 actors and stuntmen. They filmed a record 170 days of performance-capture to bring the film noir thriller to life. More than 30,000 animations were captured during this lengthy process, which required actors to physically become these characters on a prop-filled stage with doors, windows, tables, and other working props.

“We asked our actors to memorize every line of dialogue, which was the equivalent of three feature films,” says Cage. “The process was very grueling for them because while we had sets and props to work with and often other actors to work with, some scenes were shot minus some actors. The dedication we received from our cast in this game is unprecedented, which is why we spent a full year in casting these parts.”

While Heavy Rain’s performance-capture allowed Cage and his team to get the actors dialogue and acting together, as if performing in a stage production, the game also uses new facial-animation methods. Every actor was “cyber-scanned” and the graphic artists were able to create a full range of emotions in the game by using each actor’s real facial expressions. This technology also alleviated the “zombie eyes” look that haunted early Hollywood performance-capture movies like The Polar Express, which remains a prime example of the uncanny valley effect. (Uncanny valley refers to people’s penchant for being “disturbed” or even repulsed by attempts at making computer-generated people more lifelike.)

“The uncanny valley was a major challenge,” says Cage. “We realized how big a challenge it was when we [began] working on the casting demo. The actress was so powerful and we felt that our original demo only captured 50 percent of what she gave us. We just worked very hard since the demo to try to get at least 80 percent of that performance in the game. People always think about the technology, but that’s only part of the challenge. In addition to prototyping every day for the past three years working with actors to try new things, it’s also about art direction and casting. It’s not just about great facial animation, it’s about the camera and direction and actors. It’s on all different levels.”

To solve the uncanny valley dilemma, Cage and his team created a new technique whereby they motion-captured eyes from a real human being and then put that on each of the in-game characters. The end result is facial animation that looks real, rather than undead. It’s a large step forward in not just games, but other forms of entertainment, as well.

James Cameron first conceived the idea for his blockbuster 3D sci-fi hit, Avatar, 14 years ago. But the technology just didn’t exist to bring his vision to life. While much of the media attention for the visually stunning epic has focused on 3D technology, the film has been a breakthrough in pushing performance-capture to another level.

“There’d been performance-capture films before, and they hadn’t really broken through that barrier of having characters not look like zombies,”
says Cameron. “We knew that if we did that, we would just be wasting everybody’s time and money. We needed to get a better data set of what the actors were doing. We needed to have a better pipeline in order to transfer what the actors did through to their final characters.”

In addition to creating the 3D camera system that Avatar, TRON Legacy, and Resident Evil: Afterlife have all used, Cameron and his team came up with this concept of image-based facial performance-capture, which was based on a head rig. Instead of doing marker dots, and reading it with a motion-control system from a grid, they shot the actor's face from about three inches away with 100 percent close-ups the entire time. This allowed them to record all the information around the mouth, the tongue, the teeth, and the eyes. In addition, they filmed with reference cameras at the same time in high definition.

“This process gave us eye movement down to the tiniest quiver that the actor might make, and it also gave us all of the subtlety and nuance of what was happening around the mouth and across the whole face,” Cameron explains. “I think the face has 103 muscles. [Traditional] Motion capture just couldn’t do that. We used motion-capture to capture the body performance, but not the facial performance. That was a breakthrough.”

This technique allowed Cameron and his team to move back the motion-capture cameras, which enlarged their volume by about six times what had been used before. They were able to film scenes with galloping horses and have people flying on wires and performing big stunts never dreamed of before. Even something as simple as walking and talking along a forced path was opened up from 15 feet to 85 feet.

“We also introduced the idea of a virtual camera, so that I could operate as a director in the computer-generated world,” adds Cameron. “We worked in a proxy world; a computer-generated world that was real-time, meaning it looked like a videogame. A videogame is basically real-time CG. Everybody has it in their home. We actually used a game engine called Motion Builder to give us real-time renders of our world. It just didn’t look real. That wasn’t important because it allowed us to see the characters interacting in the environment, interacting with creatures, interacting with each other.”

Cameron worked with Peter Jackson’s WETA visual effects house to add the finishing touches to the performance-capture process. Cameron was able to compose the shots in any way he wanted with the basic data and CG models, and then WETA was able to render those models thousands of times over to bring a photorealistic world of Pandora to life.

“Instead of now rendering a frame of picture every thirtieth of a second, they’re rendering a frame of picture every thirty to ninety hours per processor...but then they put 40,000 processors all on a big render farm, or render wall, which made it possible to make the film photo-realistic,” said Cameron. “The important thing for me, as a director, was my interface in that CG world with this virtual camera. That didn’t exist before. So we got a facial performance-capture system, a virtual camera, and the fusion camera system for the live action. We got a whole bunch of new bits and pieces of technology that all kind of flew together to make this film possible.”

That technology also allowed the actors who brought the Na’vi creatures to life to see the world of Pandora that previously only lived in Cameron’s mind.

“Performance-capture made my imagination infinite,” said Zoe Saldana, who played Neytiri in Avatar. “Once you take everything out from around you and all you have is your mind’s work, this world was more alive than any book or movie. And I hadn’t even seen the final product. This was just going to Hawaii and doing research in the rain forest, and then coming back and seeing all these pictures and having extensive hours of conversation with Jim opening his mind to us. This world came more alive than any set I have been on, and it was hard to part with it.”

Just as Cameron’s technology on the 3D side continues to be employed on future films, the performance-capture technology should help others push this medium forward. The fact that Cameron collaborated with Ubisoft on the Avatar game also bridged the gap between performance-capture in games and movies. Convergence is ongoing across these mediums and performance capture is an evolving art form that is providing Hollywood and videogame creatives a new avenue with which to use trained actors to bring virtual beings to life in a more realistic and engaging way.