I've done some more digging. As the RB approaches the limit, it violates it by a small amount. At this point, the body's velocity drops near zero and it crawls back toward the limit in minute increments. As soon as the body crosses the limit again, velocity increases substantially and the body swings back toward the equilibrium point and then gets stuck again on the other side. Repeat ad infinitum.
My investigation led me to consider the value of m_rhs for the constraint. This value gets very low as soon as the constraint is violated. Worryingly, there's a comment at this point in the code reading "fix rhs" with no further explanation (see
http://code.google.com/p/bullet/source/browse/trunk/src/BulletDynamics/ConstraintSolver/btSequentialImpulseConstraintSolver.cpp#923).
Does any of this sound familiar to anyone?
Thanks
David