This is a well-known instability. Light elements under high tension vibrates at high frequency - like tuning a guitar string. When the vibrational frequency becomes close to or smaller than the time-step size these high-frequency modes may develop into instabilities.
Increasing mass, element size and number of iterations help. Also decreasing the time-step size. Increasing the resistance to bending of the joint help as well (e.g. a relaxed lock-joint). But that should be done dynamically such that it increases under high tension and decreases again when the tension drops.
www.vrlab.umu.se/research/flexible/rbcve.pdfThere some other ways as well - but probably takes some labor to get it implement in Bullet.
I guess position-based dynamics steppers (that projects positions and orientations to maintain constraints) are less flawed by this instability. Anyone with experience on this, please comment!