So Bullet's GJK Implementation has a performance optimization when attempting to find the closest point to the origin on a Tetrahedron. Inside of btVoronoiSimplexSolver::closestPtPointTetrahedron( GJK attempts to find which triangles created from four points are relevant for checking for a closest point. It uses a function called pointOutsideOfPlane. If you have two parallel surfaces touching, this check can throw away the triangle that is actually CLOSEST to the "origin point", leaving the function with only the ability to find a point along the edge of the triangle that is nowhere near the touching object.
See illustration of this happening:
In this example, pointOutsideOfPlane discards the correct triangle from the check, causing the closest point to be found to be along the dotted line, causing weird collision behavior. This is very easy to reproduce with sliding blocks, or rolling cylinders on a wedge (anything that sets up for parallel surfaces easily).
The fact that this is possible leads me to believe that in the case of 2 triangles that are technically the same face (and technically normal) can end up with slightly different normals due to floating point errors, which would lead to an incorrect section of the face remaining as a possible contact, while the other is considered invalid. I was able to make the bug go away by forcing all pointOutsideOfPlane calls to return 1, which forces the algorithm to scan every triangle for shortest distance and this resolved the problem.
I'll try to find a fix to submit tomorrow, I've just spent 2 days trying to track this problem down and need some sleep. Just wanted to see if anyone has had this issue before?