my center of the compound shape, is not the same as the center of mass of the object.
AFAIK, Bullet does not know and does not try to calculate the center of mass of the compound shape you pass to it: it just takes the point (0,0,0) in that shape and makes it the center of mass of your shape.
If you need to move it from there, you can shift all the child shapes inside the compound shape, or you can just create a new compound shape and add as a child shape the old compound shape shifted by an offset.
When I create the object I have to pass a motionstate, with a worldtransform. Is the center of my compoundshape positioned at that point, or is the objects center of mass positioned at that point?
The object center of mass [(0,0,0) inside the compound shape] is positioned at that point.
I create a motionstate for the child object (I feel like I wouldn't have to do this, cause the local position/rotation of the child shape should never change right?)
Maybe I'm missing something here... Motion states can be applied to btCollisionObjects (or more frequently to btRigidBodies): the children of a compound shape are btCollisionShapes (unless you extend the btCompoundShape class...); I suppose you're trying to inglobe existing rigid bodies into an existing btCompoundShape.
Or does nothing happen and do I have a problem cause my worldtransform of the compound is not at the center of mass of the object?
The latter as far as I know. But you can shift the center of mass inside the btCompoundShape manually and then update the local inertia.
There's a demo posted by Erwin you can download from here that can be useful: http://bulletphysics.org/Bullet/phpBB3/viewtopic.php?f=9&t=3303&hilit=shift+demo