Has anybody implemented occlusion culling using Bullet?
Not sure, but it is an interesting idea, and definately possible to use collision data for occlusion culling.
This is easy to implement as a series of plane tests. You could pass the 5 planes to GJK, but how well does GJK handle planes, since they are infinite, and what would you use as the supporting vertex? I could imagine using a large but finite quad as each plane but I can also imagine several issues with that approach.
Bullet implements infinite plane, by projecting the AABB of the other (moving) object onto the plane and creating (temp) triangles on-the-fly.
But I prefer your suggestion to use the 'support function':
Another idea I had is to use the batchedUnitVectorGetSupportingVertexWithoutMargin function, passing in each plane's normal, and then do standard plane tests against the results. This way, I could automatically support each shape that already works with GJK and not have to create custom code for each shape.
This is an excellent idea, and it should work very well. This should give you a test wether objects are completely overlapping the convex (truncated) frustum pyramid.
Any other ideas?
Perhaps another way would be to use AABB information (from sweep and prune, BVH/AABB tree and regular AABB's of objects) in combination with hardware occlusion query.
Sweep and prune allows you to perform front-to-back traversal, by extending the SAP ray cast (see links below) into a SAP box cast. When this box 'expands' properly during the traversal through sweep and prune space, you can conservatively approximate overlaps of any frustum pyramid too. This idea is probably too sketchy, so an implementation and whitepaper would be more suitable.
Thanks for sharing these ideas!
First publication/mention of raycast through sweep and prune
Unfinished draft paper for ray/box cast through sweep and prune