Well, it seems that many people make the same mistake (reported here too): http://bulletphysics.org/Bullet/phpBB3/viewtopic.php?f=9&t=5025&start=0&hilit=collision+flags
So it's better to point out that:
1) if you use a collision mask equal to zero, the object won't collide with anything.
2) As Alkane said, the collision happens only if both the two collision mask equations are satisfied.
3) Additionally I would add that the new collision filter groups should be compatible with the ones that Bullet uses by default:
///optional filtering to cull potential collisions
DefaultFilter = 1,
StaticFilter = 2,
KinematicFilter = 4,
DebrisFilter = 8,
SensorTrigger = 16,
CharacterFilter = 32,
AllFilter = -1 //all bits sets: DefaultFilter | StaticFilter | KinematicFilter | DebrisFilter | SensorTrigger
This means that for example when you add an object to the world without arguments (and probably when raycasting without arguments), Bullet uses the group DefaultFilter and the mask AllFilter that has all bit sets; if you use a NoCollision group you should create a new AllFilter mask that does not include it and ALWAYS add objects to the world / perform raycasts with the new collision group/flags. Also I would start making new flags past "32".
4) Another thing that should be useful to know is that the collision group/mask is passed as a "short int".
A lot of time ago I posted this message:http://bulletphysics.org/Bullet/phpBB3/viewtopic.php?f=9&t=964&p=7525&hilit=collision+flags#p7525
in which I created new flags (at that time I started at "32" because the CharacterFilter was not present), and subclassed the Bullet world so that I could use the addRigidBody method without arguments and still use a NoCollision flag (in my experience a NoCollision flag is not very useful, but for example, if I have a ghost oject frustum or some kind of sensor object, maybe I might like the AllFilter flag not to include it by default: I suggest using this approach only if the NO_CONTACT_RESPONSE method is not good for you...).
Hope this is useful to cut down the number of future posts about problems with collision flags