Legend 3D Founder Barry Sandrew, Ph.D Talks 2D to 3D Conversion!

Michael here. I recently had the pleasure of speaking with Legend 3D Founder and President Barry B. Sandrew, Ph.D, regarding his company's work on 2D to 3D conversion. Before the interview I viewed a reel of 3D footage that Legend had done, and I must say that it blew me away.

The reel consisted of converted 3D footage from many of the blockbuster feature films from the past 10 years as well as commercials for Fanta, HP, and New Balance. In all cases the quality of the stereo was indistinguishable from the very best native 3D footage I have seen, Avatar included. The 3D was immersive, detailed, and continuous: completely gone were the flatness and "planes of depth" from G-Force, which In-Three and Imageworks converted. Instead, the 3D made me feel like I was part of the action.

Particularly impressive was a scene from Iron Man where he is being suited up: all of the intricate mechanical devices and metal facets, all in complex motion and in stereo, were a beautiful sight to behold. Also amazing were the scenes from "Alice In Wonderland", which Legend 3D recently wrapped. The practical scenes from other movies were just as artfully done. I could tell a great deal of thought was put into the stereospace of every shot. According to Sandrew, it is Legend 3D's methodology that makes it possible to deliver such complicated scenes in the highest quality 3D.

Read below for more on Legend 3D and the 2D to 3D conversion industry:

Michael: How did Legend 3D get started?

Barry Sandrew: I invented digital colorization back in 1986 and in the following seven years I produced just about all of Ted Turner's colorization work as well as titles from all the major studios. We built a huge studio to get the feature films converted from black and white to color. Believe it or not, at our peak we were up to seven movies a month for Turner. I left that company (American Film Technologies) in 1993, and founded several other startup companies. Then in the year 2000 I reinvented colorization, because advances in storage, computer power and networking just made everything possible that I had always imagined for colorization. So I developed a software process and pipeline that would semi-automate everything. We had a maquiladora down in Mexico back in the early '90s, but this time I went to India. I opened up a studio about ten years ago. It was one of the first visual effects studios in India. In fact, I opened our Patna, India studio before Imageworks and everybody else discovered India and we've had that studio operating continuously for a decade now. It’s a great studio. We started with about 250 people—but we're currently ramping up to become the largest visual effects studios in all of India. San Diego is where the bulk of the creative work happens. Let me go back to colorization for a second. Around 2005 - maybe a little bit earlier - my good friend Greg Passmore and I were talking about 3D, and we both immediately recognized that the Legend Films colorization software is perfectly suited for a 3D conversion process, because one of the basic aspects of all stereo conversion is the segmentation of everything in every frame into metadata. In my career I’ve produced close to 400 feature films on time and on budget… every one. That comes second nature to me and my team. We’ve come so far in our process that there aren’t any surprises for any feature film that hits our pipeline. We can colorize any movie so we felt that we could adapt the colorization process, pipeline and technology to stereo very easily. It wasn't quite as *easy* as we had hoped, but we did it. We also created a stereo process application that is very intuitive, and that we feel is the most advanced in the industry.

Michael: How does the 2D to 3D conversion process work? How is the other-eye view created?

Barry Sandrew: Most people just create a single new eye, one eye. We create two completely unique eyes from each 2D frame. This is a more natural way of creating 3D. There's various ways to convert a movie. One way, like the way IMAX does it, is to project rotoscoped objects onto volumetric CG primitives that have been created for most things in a scene. That's the way most of [the companies] are doing it. We do it quite differently. We use depth masks as well as several other sophisticated volumetric processes to mold a shot into dimension. We're actually molding in "z" -- depth masks -- and this helps to animate things more accurately and more efficiently than using CG elements, because in our process there is no tracking that takes a great deal of time and skill. We also have very little paint and compositing with our process. Normally, if I am moving something away from a background in stereo I'm exposing parts of the background that don't exist in the image. So what do you do with that? You have to either paint it or you have to somehow fill that in so that everything looks natural. One way of course is to use clean plates. But in a catalogue feature film like Harry Potter you may not have the clean plates available. So you have to resort to skilled painters, compositors to fill in data gaps in the images. Most of our gap filling is not done the traditional way. We actually do most of it algorithmically or automatically on render.

Michael: Based on the prior frame and the following frame?

Barry Sandrew: That's only one way of filling data and a very labor intensive way. We’ve developed incredible software tools for colorization that can automatically recognize objects based on pattern, luminance and chrominance. That software has been adapted to 3D conversion and is the basis for a multitude of different routines and tools that can fill in missing data automatically.

Michael: How is the team divided between San Diego and India?

Barry Sandrew: We have a very large team in San Diego. That's where all of the client interaction happens and where most of the creative is done; particularly movie breakdown, depth script, depth score, keyframing and stereo. In India they do the masking from frame to frame.

Michael: Could you explain what masking is?

Barry Sandrew: [Masking is] when you identify objects within a scene. You identify objects digitally as metadata, and those objects then can be manipulated in stereospace.

Michael: You told me you don't use rotoscoping?

Barry Sandrew: Our process of masking is much more organic than rotoscoping. It's a very sophisticated paint program that is 23 years in the making that uses pattern recognition and several other proprietary detection routines to move masks from frame to frame semi-automatically. We have a team of people in India making certain that every frame is as perfect as possible because sometimes computers make mistakes. So to get the highest quality we have to have a person monitoring the segmentation of every single frame. For instance, if I'm in a frame of a movie, and all of a sudden my hand comes up from behind a chair, the computer doesn't immediately know that's my hand, so the operator has to define it for the computer. Once it’s defined then the computer understands where that hand is in each frame and it takes over. That's pretty much the way we do it. Everybody else uses rotoscope which is a highly skilled, very time consuming, and very expensive way to mask. Another huge advantage of our colorization masking technology is that it was designed to be used by unskilled labor. So we can train someone in India in half a day to segment scenes. It only takes about two weeks for that same person to become proficient in the process.

Michael: One thing I have always wondered is, if you are converged on something...for example, with my right eye I can see the right side of this pen, and with my left eye I can see the left side. So those companies that create the other-eye view must have to make up information that doesn't exist. Let's say the original image is the left side. So to make the 3D image--to create the right eye image--they have to say, "what's on the right side of the pen?"

Barry Sandrew: Only if it's exposed. If it's not exposed there is no right side of the pen. It’s not necessary to create the entire geometry of every object. In our stereo program, we actually rebuild an entire scene using depth masks and proprietary volumetric rendering processes. We can place objects anywhere in a scene that we want. Then on output rendering the stereo program separates the 2D image we start with into two unique images; one for the left eye and one for the right eye. The secret sauce is really in the manner in which segmented portions of a frame are positioned, the pixel blending in stereo and in gap filling.

Michael: Could you go into how the stereo is rendered?

Barry Sandrew: There’s a great deal of proprietary information I can’t share on this but basically there is a difference in whether you are converting by making one new right eye from a 2D image or making an entire new left and right eye from the 2D image. Also, whether you're projecting rotoscoped objects onto CG elements or you're using depth masks, that's a big element. Those factors really differentiate the various companies and also dictate efficiency, speed and of course most important, quality.

Michael: What is your relationship to PassmoreLab? Are you part of PassmoreLab?

Barry Sandrew: No. Greg is actually part of Legend 3D and a major shareholder. He and I have been working together for fifteen years, and PassmoreLab has always been my software engineering team. For colorization, for instance, I used a lot of Greg's software engineers--their knowledge is based in physics and they are exceptional programmers. So I basically brought them in and project managed the coding of my colorization patents under Greg. And when the 3D idea surfaced, it was a natural. Greg is extremely knowledgeable in 3D, and he regularly shoots natively in 3D. Much of this started when he wanted to find a way to do 2D to 3D conversion because he does a lot of documentaries that are uniquely difficult and often dangerous. One of his primary interests is spelunking or cave exploring, and when he goes down in caves the big stereo rigs can’t be used. Sometimes he has to scuba dive in caves with cameras. So the best solution is to shoot in 2D and then convert. So that's one of the reasons he was interested in the process. Today, his stereo work continues to be focused on very interesting documentaries. So together we collaborated to build this process that's able to convert feature films from 2D to 3D within an efficient and time tested production pipeline.

Michael: Where do you see the market for 2D to 3D conversion going over the next two to five years, in terms of the demand?

Barry Sandrew: Demand I believe is going to accelerate. It is going to get bigger as time goes on, provided quality 3D is the norm. There is some pretty bad stuff out there that hurts your eyes and makes people sick and that definitely hurts this fledgling industry. I think that with catalogue titles... the various conversion companies--the legitimate ones, the ones that aren't coming out of the woodwork are going to be busy for a long, long time. All of us. There's just so many catalogue titles that would be amazing in stereo.

Michael: What's your wish list?

Barry Sandrew: Oh, They come natural. In fact many of the prime candidates you just viewed as demos in our 3D theater. Just think about it. What's your favorite movie? They just roll of your tongue. Movies like Star Wars, LOTR, The Matrix, 300, Jurassic Park, Titanic are perfect for stereo conversion. And I guarantee it’s going to happen eventually. There are new feature films that are going to be converted, and catalogue titles that are going to be converted. The new feature films are going to be converted primarily because a lot of directors don't feel comfortable in 3D. Also, it's cheaper to convert 2D to 3D than shoot natively in 3D. What we offer directors is an opportunity to direct a film in the medium in which they feel most comfortable--2D typically. And then we work with the creative director and convert his/her footage from 2D to 3D.

Michael: What do you think are the pros and cons of shooting in native 3D verses converting, as the technology stands right now?

Barry Sandrew: Well, the rigs are imperfect, awkward and very tricky to use. Everything in the two cameras have to be perfectly aligned and identical in every aspect. The director then has to spend the extra time to block and frame a shot for stereo rather than 2D. With 2D to 3D conversion, nothing really changes from a traditional 2D shoot. A stereographer is typically on the set advising the director and cinematographer but for the most part, the director can comfortably shoot the way they always have.

Michael: You can't be misaligned by more than a micron...

Barry Sandrew: Yeah. And if misalignment happens in a native shoot you’ll sometimes see highlights that look a bit strange. That's because the left and right lenses are off a tiny bit. There's a lot of these kind of issues with shooting that you don't get in conversion. In fact, I contend that conversion can look better and even more natural than a native stereo shoot. Certainly in the demo material you just screened there is no way you can tell it wasn’t originally created in 3D with a traditional shoot. When you shoot natively everything has to be staged precisely for 3D. If the convergence is off, if the interocular distance is off, you're going to have issues with the image. You can fix it in post, but that's another expense. One thing people can do when they screw up a stereo shot, is they can always convert it afterwards using just one eye. With conversion, you have a lot more creative freedom. Without exception, the directors I’ve introduced to the conversion process very much appreciate the creative freedom to create stereo after their 2D shoot is completed.

Michael: Are there any disadvantages to conversion? Is there any quality or attribute to a 3D shot that you can get by shooting natively that you can't get with conversion?

Barry Sandrew: I can't think of one, but I can think of a lot of advantages as I’ve already stated.

Michael: Where do you see other markets for 2D to 3D conversion going? We saw some very impressive ads on the Legend 3D reel. Have those played in front of 3D features yet?

Barry Sandrew: We produced the very first theatrical 3D ad. It was a Skittles commercial. The others have been contracted by ad agencies to be shown privately to new clients.

Michael: Currently the pre-theater advertising is done on a separate projector that is very low resolution.

Barry Sandrew: Yeah, but that's changing.

Michael: I wish it would change since you can see the pixels from the back row.

Barry Sandrew: It's terrible. There's no reason you can't get a DCP (Digital Cinema Package) of a commercial, just plug it in to the same server as the feature film and put it on the playlist to run prior to the main feature. Most of [the advertising agencies] are risk averse, and until Avatar came, they said, "We want to wait and see." As you might expect, after Avatar they are all clamoring to get their commercials on the screen at the same time.

Michael: Now the floodgates have opened.

Barry Sandrew: Exactly. Avatar was and is a game changer. And I think "Alice" is also going to be a game changer. Legend 3D did a substantial amount of the Alice conversion, and it's absolutely stunning. It really looks beautiful.

Michael: On "Alice", did Legend 3D share conversion duties pretty evenly with In-Three?

Barry Sandrew: No. In-Three did the bookends--before Alice falls in to the hole and when she comes out. Those were all practical shots. We did some of the most difficult effects shots in the film. When Alice falls into the round room. Most of it was practical and consequently we did not have clean plates to help out. On top of that, we came against brown/blonde walls, and Alice’s blonde hair that has almost the same texture, so we had to use our colorization masking technology to get her hair out of the wall so it could be converted seamlessly, looking perfectly natural in 3D. I think we did an incredible job. We also did much of the tea party, much of the castle with the queen of hearts, and much of the mushroom forest.

Michael: What types of shots are generally more difficult or time consuming to convert?

Barry Sandrew: That's kind of difficult to say. In Alice, the Mad Hatter walks on top of the tea party table which has a myriad of objects and utensils that get thrown about all over the place. That's a difficult thing for most conversion companies to do but it was second nature for us.

Michael: In terms of the cost of conversion, I don't know if you could tell me how your costs compare to those of your competitors, or whether you could tell me the cost per frame.

Barry Sandrew: We don't convert by frame or shot like a typical visual effects house. We typically quote a conversion based on the entire feature film. We don't consider ourselves a special effects house nor do we consider ourselves a post-production facility. We consider ourselves a creative studio. And as a creative studio we generally will not work on a feature film with another conversion company. Alice was an exception. We really want to have the director working directly with us to make their feature film a cohesive stereo experience.

Michael: Did you work intimately with Tim Burton on "Alice"'s stereo? Does Legend 3D have software similar to In-Three's that allows directors to place elements in various parts of the stereospace?

Barry Sandrew: As I mentioned earlier, our process places as much creative control on the film director as he/she wants. On Alice, we worked more closely with Imageworks, which was great. They're a good talented group, their quality is exceptional and they are very professional. Our separate processes complemented each other perfectly.

Michael: So in terms of the costs, you charge per movie?

Barry Sandrew: There are a multitude of factors that determine our costs, similar to putting together a movie budget. Our quotes are typically a fraction of all the other companies. Don’t get me wrong, you can get stereo done very cheaply, but you get what you pay for. Conversion is a very time consuming, labor intensive process no matter who does it. That's costly, especially when you've got highly skilled paint and compositing people that you have to bring in to do a major feature film. So if a conversion company low-balls a project, or tries to compete with a very low price, and a studio actually uses them, unfortunately they get what they pay for and they have no one to blame but themselves.

Michael: How do you see the field in terms of your competition? How many companies are at the upper echelon, and how many are springing out of the woodwork?

Barry Sandrew: There's just a few companies that I consider legitimate. What I mean by legitimate is that there's a handful of companies that can actually do an entire feature film. What it breaks down to is depth of experience. We’ve had artists in the company that have been working on evolving our patented process for a decade. Converting an entire feature film is a very complex and highly skilled task, you have to take it apart, break it down, dissect it into scenes, cuts, locations, characters, and then put a special effect on every single pixel in every frame of that feature film—approximately 180,000 frames--and then you're doubling it in the stereo process to 360,000 full resolution frames that encompass the separate left and right eyes. When you’re finished you have to put it all back together again and make it look better than the way you originally got it. That's a daunting task for most companies. Again, I've done close to 400 conversion from black and white to color that way. But getting back to your question, I think there are perhaps three or four of us who are legitimate. The nice thing is that we are all friendly competitors--it's not a cutthroat situation right now because we all see how much work there is out there, and we all know that none of us could tackle it all.

Michael: More 3D screens will be required to get all of that work out there. Do you think soon there will be enough screens to support to simultaneous 3D releases?

Barry Sandrew: Well, J.P Morgan, didn't they just arrange $700 million in financing [for 3D screens]? That's gonna do it. I think 3D screens are going to be ubiquitous internationally, and I think there's going to be more than enough screens to make this mainstream.

Michael: Back to your technical process--does creating two eye views require more work, and do you think it results in better 3D?

Barry Sandrew: It's not more work. It creates better 3D because, for one thing, the gap artifact in the background is spread across two images rather than one. It allows our gap fill algorithms to function more effectively.

Michael: What can you tell me about what Legend 3D is working on now?

Barry Sandrew: We are subject to a number of non-disclosure agreements which limit what I can say, but I can tell you that we've got multiple movies in progress from major studios and a number of other contracts about to be signed.

Michael: What do you think the timeline is for the arrival of in-theater stereo 3D advertising?

Barry Sandrew: It's starting to happen now. We've got commercials out there. We expect our Fanta commercial that we just completed will be shown in theaters.

Michael: Anything else you would like to say about Legend 3D or 2D to 3D conversion?

Barry Sandrew: I believe quality is definitely improving, and the reason it is going up is because the short list of companies that are involved in conversion is making certain the quality bar is high and stays that way. In fact, every day we’re getting better and our patented process is evolving. Our small group of competitors talk about it among ourselves. We all say, "the only thing that could hold back this industry is if cut-rate conversion companies screw up and produce subpar quality picture." [For the] companies that are legitimate and experienced in feature film production, the quality is high.

Michael: Thanks for taking the time to talk. It was great seeing the demo and learning about your process!

Visit Legend 3D's website at http://www.legend3d.com/