I think that I found the "solution".
I hope there will be somebody who understand this and can explain why this is the situation:
So, the source of the bouncing is the internal edges of my concave mesh collision shape.
There is already a utility solution for this kind of problem in bullet, called "InternalEdgeUtility", and a demo sample called "InternalEdgeDemo".
I used this solution to my program, but this did not solve the bouncing in my situation.
This was because of the friction.
I use a big friction in my program, but the InternalEdgeDemo uses zero friction.
I configure the InternalEdgeDemo to a similar setup like my program. This setup is a bigger an longer, more dense grid, and a sphere collision shape on it, instead of those ones in the original InternalEdgeDemo.
If you download the modified InternalEdgeDemo from the attachment and compile and run it, press space to restart animation, press O for side view, and watch, that if you set zero friction in calculateCombinedFriction, then the sphere rolling smooth on the floor, and when you set for example .5 friction, then the sphere starts bouncing.
If you turn off "contact filtering" with key N, then the bouncing happens at zero friction, too. So the "internal-edge-anti-bouncing" solution works in bullet, but it seems, that it is accurate only at little friction values.
You can fire boxes with mouse, and with those bodies the situation is the same.
Now, the question is, that what is the connection between friction and "contact filtering", and why do I have to use small friction values ?
Last edited by openeye on Wed May 02, 2012 12:40 pm, edited 1 time in total.