The challenges of Spider-man 3 were akin to making sand flow uphill. Literally.

Thanks to a close encounter with a particle accelerator, petty thug Flint Marko (Thomas Haden Church) is transformed into a living, sentient dune called Sandman. That's not easy for him - but it was tough as hell for the FX team at Sony Pictures Imageworks (SPI).

Sure CG particle systems have been around for years, but imagine combining that with actor-driven character animation and you begin to see the problems faced by SPI’s crew, headed by visual effects supervisor Scott Stokdyk. “It was a collaboration between our character animators and our effects animators, and we had to go back and forth between them,” says Stokdyk, who served as co-supervisor (with John Dykstra) on the earlier Spidey films.  “If there’s a theme of Spider-man 3, it’s the convergence of character work and effects work, and how tied those were together.”

Diehard Spider-man comics fans know that James Cameron’s liquid metal robot in Terminator 2 was inspired by Sandman illustrations, and The Mummy series offered a gigantic human visage as screaming sandstorm, so Stokdyk was relieved when he saw that director Sam Raimi’s Sandman storyboards were not at all like those previous movies. But something in his own background  - a particle beam created when the stones were assembled in The 5th Element - sent a familiar chill through Stokdyk: “That was just so crude compared to what we had to do [for Sandman,] and I knew enough to know how difficult a problem it was, because I knew the easy little cheats I had done many years ago with scale were so insufficient.”

Stokdyk’s 5th Element effect was a crude form of target-driven particle simulation – Sandman had to be the ultimate. Inspired by SIGGRAPH papers and Raimi’s imagination, SPI sought to put particles through as many different behaviors as sand is capable of – often at the same time. “It’s a huge collection of particles, but there’s so many different types of behaviors that the sand team broke those into their own separate systems. So a sand particle might go through a flowing simulation then a target-driven simulation, then it might go to gravity, into a piling simulation, and back into the flow simulation. I’ve got to hand it to the sand team: It’s one problem to figure out how to do this with a small number of particles, but to have it scale from thousands to millions of particles is something else, and to figure out how to render it…  It was incredibly complex and I still wonder how we got it all done.”

Stokdyk’s visual effects philosophy, embracing many off-the-shelf software tools, certainly helped: “I tend to be tool agnostic,” Stokdyk says, “We relied on Bonsai, our in-house compositing software for our 2D, and Houdini & Maya for our 3D work, and the openness of both those platforms has been critical to all our proprietary work. We occasionally used Shake and Flame, and RenderMan was the cornerstone of our rendering, but we had to push it to its limits.”

Ultimately, Sandman was a unique combination of hand-animated character and particle simulation, with limited use of facial capture to help convey his emotions. “Our animation director, Spencer Cook, relied on video footage for inspiration,” Stokdyk says. “Spencer’s philosophy is to shoot actors or his animators acting out scenes from different camera angles, then use that as really good reference to hand-do the animation. So our character animators would start with a basic interpretation of Thomas Haden Church, the actor playing Sandman, forming from a pile of sand on the ground plane, for example.”

After the physical animation of Sandman doing a particular action was completed, the trick was to form particles into his shape that looked convincingly like they’re motivating the action instead of the other way around. “That target-driven simulation was one of the plug-in pieces of our sand pipeline,” Stokdyk says. “We’d pick the moving animation as a target and a whole bunch of parameters would guide the particles into that.”

Only it wasn’t quite so straightforward: “Typically, we work in linear fashion: You animate an animal, it goes to fur, then you put that in a [photographic] plate and light it.  With Sandman, those processes were anything but linear, they were just iterative. The base shapes were driven by hand-animation, but then those shapes went into a simulation system, where they were used to create volumes, and a whole palette of different sand behaviors. The incredibly difficult thing about Sandman was it wasn’t just a hand-off from character animators to effects animators. They had to work really closely together, so if an effects animator needed a volume of sand to fall away from Sandman’s head, they would ask the character animator to animate the shape, then compensate in the body for that loss of shape. Then the effects animators would deal with those particle volumes in different ways - treat them as procedural surfaces or use cloth simulations for draping the surface to smooth out the shape - and add animated bounce, flow, and drip effects. So effects and character animation drove each other. We weren’t just trying to recreate natural phenomenon, we were creating an emotional, character-driven piece that had to be grounded in reality and natural movement.”

Except that sand doesn’t naturally form into complex human shapes. “Sand has certain characteristics that we’re all familiar with,” Stokdyk explains. “Sand dunes never have cliffs in them unless they’re wet. If it’s not wet, sand just cascades and avalanches down, creating these characteristic mounds that have this thirty-degree angle, which is called the angle of repose. So when we were putting particle effects on top of our character animation, we had to relax the surface into those characteristic shapes.”

Nowhere was this more challenging than in the superb Sandman creation sequence, wherein the disintegrated particles of Flint Marko pull themselves together through sheer force of will to give rise to a man made of sand. Stokdyk wisely chose to put off tackling the earlier sequence till later in the schedule. “We started the first sequence early, but we put the execution off till we could learn from the bank truck sequence, then apply everything we knew to the birth of Sandman sequence. Spencer Cook studied various sculptures, then previs-ed and blocked out key poses, and E.J. Krisor, an amazing artist, drew beautiful black and white illustrations of what those key poses might look like in sand, which gave our sand effects team a target to go towards.”

A terrifying thing happened as the birth sequence evolved: Sam Raimi kept consolidating cuts till what was conceived as a sequence of 20 short shots was distilled into a single whopping two-minute shot with the camera circling a mound of sand desperately trying to form itself into a body. The result: some of the most powerful character animation ever, evoking the sense of an entity trying to form out of a heap of particles. “That first shot was so hard—it was definitely the hardest shot I’ve ever worked on in anything I’ve ever done - because it was basically a long, slow shot, with nothing but computer-generated imagery to tell the story,” Stokdyk admits. “We couldn’t fall back on actors or other crutches; it’s all out there, which was a little scary at times.”

Imageworks animators and artists create the epic final battle between Spider-Man and Giant Sandman.
Rollover thumbnails to view full image
Photo credit for all images: Courtesy Sony Pictures Imageworks