weird rayTest bug

Post Reply
MadExecutioner
Posts: 2
Joined: Tue Nov 04, 2014 7:57 pm

weird rayTest bug

Post by MadExecutioner »

Hi,

I am trying to implement picking of objects by following this tutorial. Currently I have two objects. One of them is at the origin, the other is moved 3 units along the x-axis. Picking the one at the origin works, but for some reason I can only pick the right object by clicking on an area where the ray goes through both objects. Here is a picture of it: http://i.imgur.com/nfrDwIb.png

I can easily select the monkey, but I can only select the table by clicking the red area. Oh, and everything works fine if I create the table at (3, 0, 0) instead of moving it with my setPosition method, so I guess there lies the problem. Here my code of that method:

Code: Select all

void MeshObject::setPosition(float posX, float posY, float posZ) {
	position = glm::vec3(posX, posY, posZ);

	btTransform transform;
	rigidBody->getMotionState()->getWorldTransform(transform);
	transform.setOrigin(btVector3(posX, posY, posZ));
	rigidBody->getMotionState()->setWorldTransform(transform);

	btTransform transform2 = rigidBody->getCenterOfMassTransform();
	transform2.setOrigin(btVector3(posX, posY, posZ));
	rigidBody->setCenterOfMassTransform(transform2);
}
I don't really understand if I am supposed to move the motionState or the rigid body, but it doesn't work correctly either way.

Not sure if it helps, but here is the ray casting code (basically the same as in the tutorial):

Code: Select all

	glm::vec4 rayStartNDC(
		((float)mouseX/(float)screenWidth  - 0.5f) * 2.0f,
		((float)mouseY/(float)screenHeight - 0.5f) * 2.0f,
		-1.0, // The near plane maps to z = -1 in Normalized Device Coordinates
		1.0f
	);
	glm::vec4 rayEndNDC(
		((float)mouseX/(float)screenWidth  - 0.5f) * 2.0f,
		((float)mouseY/(float)screenHeight - 0.5f) * 2.0f,
		0.0,
		1.0f
	);

	glm::mat4 M = glm::inverse(projectionMatrix * camera.getViewMatrix());
	glm::vec4 rayStartWorld = M * rayStartNDC;
	rayStartWorld /= rayStartWorld.w;
	glm::vec4 rayEndWorld = M * rayEndNDC;
	rayEndWorld /= rayEndWorld.w;

	glm::vec3 rayDirectionWorld(rayEndWorld - rayStartWorld);
	rayDirectionWorld = glm::normalize(rayDirectionWorld);

	rayDirectionWorld = rayDirectionWorld * 1000.0f;

	btVector3 start(rayStartWorld.x, rayStartWorld.y, rayStartWorld.z);
	btVector3 direction(rayDirectionWorld.x, rayDirectionWorld.y, rayDirectionWorld.z);

	btCollisionWorld::ClosestRayResultCallback rayCallback(start, direction);
	dynamicsWorld->rayTest(start, direction, rayCallback);

	if (rayCallback.hasHit()) {
		printf("hit!\n");
		((MeshObject*)rayCallback.m_collisionObject->getUserPointer())->toggleSelection();
	}
	else {
		printf("hit nothing\n");
	}
MadExecutioner
Posts: 2
Joined: Tue Nov 04, 2014 7:57 pm

Re: weird rayTest bug

Post by MadExecutioner »

Okay, I solved it by calling dynamicsWorld->stepSimulation. I am guessing that the spatial partitioning tree wasn't updated so bullet didn't do the narrowphase collision test unless the ray hit the old tree node.
Post Reply