The restitution/friction is approximate in the Bullet library, but you can make some changes that improves it:
How is the resultant coefficient of restitution between two objects calculated?
It is computed in Bullet\src\BulletCollision\CollisionDispatch\btManifoldResult.cpp, see calculateCombinedFriction and calculateCombinedRestitution.
You can change the source code, or override it, by setting the
bool MyContactCallback(btManifoldPoint& cp, const btCollisionObject* colObj0,int partId0,int index0,const btCollisionObject* colObj1,int partId1,int index1)
cp.m_combinedFriction = 0.f;
cp.m_combinedRestitution = 0.f;
gContactAddedCallback = MyContactCallback;
//for each body that you want the callback, enable it:
Even if I set the coefficient of restitution of both to 0.0, the ball does not stick to the plane as expected but bounces off of it.
Try enabling split impulse, it will prevent the added energy (velocity) from penetrations.
m_dynamicsWorld->getSolverInfo().m_splitImpulse = true;
Hope this helps,