This works great within a limited range of joint orientations.
Well, the major issue here is probably that the 6DOF Constraint animations are not "quaternion based" but "Euler axis based". Is that the reason of your "limited" range of joint orientations ?
Unfortunately, once a joint exceeds +/-90 degrees, the constraint goes haywire. I finally looked at the source for the btGeneric6DofConstraint and found that the y-axis is limited to +/-PI/2, explaining the strange behavior.
For now I am planning to rotate the joint space such that the y-axis maps to each joint's twist axis (which should never exceed +/-90 degrees). Hopefully the x and z axis limit of 180 degrees will avoid this problem... If not, I'm not looking forward to rolling my own custom constraint code.
To the Bullet developers: is there any plan to increase the angular limit range for future Bullet revisions? Or, to create a quaternion based "orientation-matching" constraint? This would be extremely useful for me, and I imagine, for others in the future.
A quaternion-based hinge constraint would be very useful indeed (and it would require just one motor instead of the 3 in a 6DOF constraint): basically, as far as I can understand it, this "quaternion hinge" should be able to change its "hinge axis" based on a "starting" orientation quaternion and a "target" quaternion (without doing any "slerp" rotation, just setting up a motor for it); if somebody knows how to implement it, he's welcome (but I guess it would still be difficult to add "Euler angles limits" to such an implementation: so there are some advantages in sticking to 6DOF constraints after all...).
As far as the 360° rotation around the Y axis with the current implementation of the 6DOF constraint, it's possible and it's working good. See this link: http://bulletphysics.org/Bullet/phpBB3/viewtopic.php?f=9&t=4457
(it seems to work with the X or Z axis free, since the Y axis must always be in [-SIMD_HALF_PI,SIMD_HALF_PI] ). I suggest you use a recent Bullet version for this fix (viewtopic.php?f=9&t=5046&hilit=spinning+like+crazy
PS. You may try the code I posted some time ago as a guideline: http://bulletphysics.org/Bullet/phpBB3/viewtopic.php?f=9&t=7676&p=27180&hilit=limits#p27180