Real-Time Physics Simulation Forum

I'm using a ceiling-mounted Kinect camera to acquire 3d data and translating it into a btHeightfieldTerrainShape by updating its data every frame. The idea is to use the heightfield terrain to represent (roughly) human figures in a space and allow them to interact with other rigid body elements in the 3d simulation.

It works, sort of, but in the following video you can see that it is glitchy. The green bar rotates about a hinge constraint in its center, kind of like a propeller, and the idea is that as you walk in circles in the room, it will move with the person. Sometimes the heightfield will move it, but not reliably--especially if the contact is from the side.

Is this the best way to translate real-time 3d data into an object in bullet? Am I just doing something wrong? Anyone else used the kinect with bullet?

Here's a screen recording... I initially set it up and spin it by hand, then enter the space around 0:27
http://www.youtube.com/watch?v=SaT7UCP09Ww

btHeightfieldTerrainShape simulates a 2D heightfield terrain

A real-time dynamic 3d generation is an interesting approach, however if you wanted something a little more reliable and "cheap", you could create a "user" object in your simulation ahead of time (fixed, arbitrary shape), and just tie its transform to the kinect-tracked real-world object. We've experimented with this approach in our lab with some kinda neat results so far. The transform is still jittery (would need some sort of motion-tracking smoothing/damping algorithms), but at least the shape is consistent.