Q I read with interest your column on tracking and also the information on mocha, and wanted to give tracking a try. I shot some stuff and brought it into my system and was amazed at how it all locked up solid. In the past, I would have never thought it was possible without days of tedious adjustments and flipping back and forth between frames. With a little confidence, I decided to go back to a scene I had shot where I wanted to remove an object. It looked like I had some clean background to use as a replacement, but I could never get it to work. It wasn't even a matter of just drifting a bit; it wasn't even close. I've attached a small clip showing the results.
A Doesn't it always go that way? All the "tests" and demo footage work fine, but then when you go to use some actual footage, whatever software feat you try to accomplish—keying, tracking, distortion compensation—doesn't work.
While I'm not a tracking expert, by any means, and while I don't want to set up a "send me your footage and I'll figure it out" system, I was able to determine what the problem is in your clip. It's "Ze Plane! Ze Plane!" (My apologies to Tattoo.)
You've been experimenting with planar tracking. This means you're trying to track items that all exist on the same flat surface. The software then uses all of the details that you've selected in that plane.
In your case, the plane was a window. A window is a flat surface, but not everything that you see on the window exists on the window's surface. If a sign is plastered on the windowpane, it exists in that plane. But if the object you're tracking is a reflection, it doesn't exist on that plane.
If that doesn't make sense, try this simple experiment. Put a Post-it® note with writing on it on a window with some reflections. Focus on the Post-it® so the writing is sharp. Now widen out and look at the reflections. Are they sharp? Depending on your depth of field, probably not.
The reflections don't exist on the plane of the window, only the note does. The reflections exist in the world that's being reflected. This is why eye doctors' exam rooms don't need to be 20 feet long to do eye tests using a Snellen chart for 20/20 vision. If you have a 10-foot distance and a mirror, you can make that distance 20 feet. The chart doesn't exist in the plane of the mirror; it exists in the back of the room.
Getting back to your tracking problem, since the object you were tracking doesn't exist in the plane of the window, it's also most likely off-axis from the camera (unless it's a reflection of the camera itself). This means that any motion caused by camera movement won't move like everything else in the scene. In most cases, the tracked object will move in a reverse direction and will confuse the tracker. Depending on the scene, one way around this may be to track the window frame for the data and to exclude all of the reflections on the pane of glass.
Tracking is an amazing science, but it's not an exact one. Don't forget the planes.