Synchronizing Client Server Physics

Post Reply
User avatar
KKlouzal
Posts: 65
Joined: Thu Mar 06, 2014 5:56 am
Location: USA - Arizona

Synchronizing Client Server Physics

Post by KKlouzal »

In a client server environment the server would run physics and periodically update clients with new object positions and rotations. Adding in some interpolation would remove most of the jerkiness resulting from net lag, however clients with moderate to high pings would see a delay before actions occurred.
A remedy for this issue would be to have clients also running physics so they could do predictions on actions they take which modify the physics environment receiving frequent updates from the server which has the final say on what an objects position and rotation are. Again adding interpolation and smoothing when prediction errors occurred would remove jerkiness in this situation.

I figure there are 4 values which need to be sent from the server during updates
Position, Rotation, Velocity, and Angular Velocity

I've done a lot of research on this topic and I've came across multiple posts where people say "It's not a good idea to manually set positions and rotations of rigid bodies". Is this truly the case and does anyone see any faults in my logic?

Thank you all very much for your time.
User avatar
KKlouzal
Posts: 65
Joined: Thu Mar 06, 2014 5:56 am
Location: USA - Arizona

Re: Synchronizing Client Server Physics

Post by KKlouzal »

8 years later, much learning, abandoned projects, some success, many failures. Here we are once again. I have gotten client/server physics running, but with some issues which I hope others can give me guidance on. Let me start with a video..
https://youtu.be/NzFPlARgWZ0

I have a simulation running on both the client and the server, the server sends updates to the client about rigid bodies once every 333ms. You can see from the video that things like to bounce around when they collide. The client simulation is fighting the updates from the server simulation.

Here is what gets sent over from the server to clients:

Code: Select all

void MeshSceneNode::Tick(std::chrono::time_point<std::chrono::steady_clock> CurTime)
{
    if (LastUpdate + std::chrono::milliseconds(333) < CurTime)
    {
        btTransform Trans = _RigidBody->getWorldTransform();
        btVector3 Origin = Trans.getOrigin();
        btVector3 LinearVelocity = _RigidBody->getLinearVelocity();
        btVector3 AngularVelocity = _RigidBody->getAngularVelocity();
        
        for (auto& Client : WorldEngine::NetCode::ConnectedClients)
        {
            KNet::NetPacket_Send* Pkt = Client.first->GetFreePacket((uint8_t)WorldEngine::NetCode::OPID::Update_SceneNode);
            if (Pkt)
            {
                Pkt->write<uintmax_t>(GetNodeID());         //  SceneNode ID
                Pkt->write<float>(Origin.x());              //  Position - X
                Pkt->write<float>(Origin.y());              //  Position - Y
                Pkt->write<float>(Origin.z());              //  Position - Z
                Pkt->write<float>(Origin.w());              //  Position - W
                Pkt->write<float>(Trans.getRotation().getX());            //  Rotation - X
                Pkt->write<float>(Trans.getRotation().getY());            //  Rotation - Y
                Pkt->write<float>(Trans.getRotation().getZ());            //  Rotation - Z
                Pkt->write<float>(Trans.getRotation().getW());            //  Rotation - W
                Pkt->write<float>(LinearVelocity.x());      //  LinearVelocity - X
                Pkt->write<float>(LinearVelocity.y());      //  LinearVelocity - Y
                Pkt->write<float>(LinearVelocity.z());      //  LinearVelocity - Z
                Pkt->write<float>(LinearVelocity.w());      //  LinearVelocity - W
                Pkt->write<float>(AngularVelocity.x());     //  AngularVelocity - X
                Pkt->write<float>(AngularVelocity.y());     //  AngularVelocity - Y
                Pkt->write<float>(AngularVelocity.z());     //  AngularVelocity - Z
                Pkt->write<float>(AngularVelocity.w());     //  AngularVelocity - W
                //
                //
                //  TODO: This will work for now.. But if clients connect in on a different NetPoint then this will not suffice..
                WorldEngine::NetCode::Point->SendPacket(Pkt);
            }
        }
        LastUpdate = CurTime;
    }
}
We're sending the Origin, Rotation, Linear Velocity, and Angular Velocity. Perhaps this is not the best way to grab these values? Perhaps I'm grabbing the wrong values or there are more values that I need to grab aside from these?

On the client side, we update the values immediately when we receive them:

Code: Select all

					//
					//	Update SceneNode
					else if (OperationID == WorldEngine::NetCode::OPID::Update_SceneNode)
					{
						uintmax_t NodeID;
						_Packet->read<uintmax_t>(NodeID);
						TriangleMeshSceneNode* Node = static_cast<TriangleMeshSceneNode*>(WorldEngine::SceneGraph::SceneNodes[NodeID]);
						//
						if (Node)
						{
							btTransform Trans = Node->GetWorldTransform();;
							//
							btVector3 Origin;
							_Packet->read<float>(Origin.m_floats[0]);
							_Packet->read<float>(Origin.m_floats[1]);
							_Packet->read<float>(Origin.m_floats[2]);
							_Packet->read<float>(Origin.m_floats[3]);
							Trans.setOrigin(Origin);
							//
							float RotX, RotY, RotZ, RotW;
							_Packet->read<float>(RotX);
							_Packet->read<float>(RotY);
							_Packet->read<float>(RotZ);
							_Packet->read<float>(RotW);
							Trans.setRotation(btQuaternion(RotX, RotY, RotZ, RotW));
							//
							btVector3 LinearVelocity;
							_Packet->read<float>(LinearVelocity.m_floats[0]);
							_Packet->read<float>(LinearVelocity.m_floats[1]);
							_Packet->read<float>(LinearVelocity.m_floats[2]);
							_Packet->read<float>(LinearVelocity.m_floats[3]);
							btVector3 AngularVelocity;
							_Packet->read<float>(AngularVelocity.m_floats[0]);
							_Packet->read<float>(AngularVelocity.m_floats[1]);
							_Packet->read<float>(AngularVelocity.m_floats[2]);
							_Packet->read<float>(AngularVelocity.m_floats[3]);
							//
							Node->NetUpdate(Trans);
							//Node->NetUpdate(Origin, Rotation, LinearVelocity, AngularVelocity);
						}
					}
The NetUpdate functions from the end of the previous code block are as follows:

Code: Select all

	void NetUpdate(btVector3 Origin, btVector3 Rotation, btVector3 LinearVelocity, btVector3 AngularVelocity)
	{
		btTransform Trans = _RigidBody->getWorldTransform();
		Trans.setOrigin(Origin);
		Trans.setRotation(btQuaternion(Rotation.x(), Rotation.y(), Rotation.z()));
		_RigidBody->setWorldTransform(Trans);
		_RigidBody->setLinearVelocity(LinearVelocity);
		_RigidBody->setAngularVelocity(AngularVelocity);

		bNeedsUpdate[0] = true;
		bNeedsUpdate[1] = true;
		bNeedsUpdate[2] = true;
	}
	void NetUpdate(btTransform Trans)
	{
		_RigidBody->setWorldTransform(Trans);
		bNeedsUpdate[0] = true;
		bNeedsUpdate[1] = true;
		bNeedsUpdate[2] = true;
	}
So you can see, we directly update the rigid body via its setWorldTransform() function. If I could cache this updated btTransform inside the object, and use it with the btMotionState then I would prefer to do it there, but if I'm correct, the motion state is only for receiving updates when a rigid body moves, not the other way around? Nonetheless, perhaps I'm updating these values incorrectly, or at the wrong time?

The other thought is that I'm simply snapping to the updated values, and it might be better to smoothly interpolate into the updated values? I don't think this accounts for how jumpy/explodey the rigid bodies are when they come in contact..

And maybe none of my ideas are going in the right direction, someone else might know of a better way to synchronize things?
User avatar
KKlouzal
Posts: 65
Joined: Thu Mar 06, 2014 5:56 am
Location: USA - Arizona

Re: Synchronizing Client Server Physics

Post by KKlouzal »

Some other forum posts I found about this same topic:
viewtopic.php?p=33699
viewtopic.php?t=11075
viewtopic.php?t=2370
viewtopic.php?t=9023
viewtopic.php?t=3002
viewtopic.php?t=6231
viewtopic.php?t=3435

Basically, others looking for solutions to the same problem, some arguing determinism being the main issue. The problem is:
#1 - Authoritative updates from the server, while sent in the present, once received by the client are effectively updates about the past. (however long it takes the server to write the packet, send it, travel across the network/internet, be read by the client, and finally applied to the simulation). A local network could be 5ms, across the country could be 120ms, across the world 300ms+. How to account for this is the primary problem, and can't be avoided, but can be mitigated.

#2 - Determinism, your client/server simulations are going to drift apart faster as determinism goes down. This too can't be avoided. Bullet could be designed in such a way that it is 100% deterministic, however, running the same simulation on a different subset of hardware and you will get different results.

#1 and #2 compound each other. So, the techniques used to hide or mitigate this is extremely important.

I feel like there was a #3 but lost my train of thought... Anyways... I'm here asking for what techniques I can use to synchronize the client/server simulation..
User avatar
KKlouzal
Posts: 65
Joined: Thu Mar 06, 2014 5:56 am
Location: USA - Arizona

Re: Synchronizing Client Server Physics

Post by KKlouzal »

So some progress.. This video demonstrates two clients:
https://youtu.be/47fuT13mAjw
I tried to line each client up as close as possible, each client does appear to be synchronized. Rigid bodies are instantly snapped/teleported to it's new location when receiving an update from the server, so some blending/interpolation here would go a long way to 'smooth things out'. Not sure how this will look with higher latencies.

Nonetheless here is the changed code:

Client receives update packet from server:

Code: Select all

					else if (OperationID == WorldEngine::NetCode::OPID::Update_SceneNode)
					{
						uintmax_t NodeID;
						_Packet->read<uintmax_t>(NodeID);
						TriangleMeshSceneNode* Node = static_cast<TriangleMeshSceneNode*>(WorldEngine::SceneGraph::SceneNodes[NodeID]);
						//
						if (Node)
						{
							//
							//	Only update if this packet UniqueID is greater than the most recent update
							if (Node->Net_ShouldUpdate(_Packet->GetUID()))
							{
								btTransform Trans;// = Node->GetWorldTransform();
								Trans.setIdentity();
								//
								btVector3 Origin;
								_Packet->read<float>(Origin.m_floats[0]);
								_Packet->read<float>(Origin.m_floats[1]);
								_Packet->read<float>(Origin.m_floats[2]);
								_Packet->read<float>(Origin.m_floats[3]);
								Trans.setOrigin(Origin);
								//
								float RotX, RotY, RotZ, RotW;
								_Packet->read<float>(RotX);
								_Packet->read<float>(RotY);
								_Packet->read<float>(RotZ);
								_Packet->read<float>(RotW);
								Trans.setRotation(btQuaternion(RotX, RotY, RotZ, RotW));
								//
								btVector3 LinearVelocity;
								_Packet->read<float>(LinearVelocity.m_floats[0]);
								_Packet->read<float>(LinearVelocity.m_floats[1]);
								_Packet->read<float>(LinearVelocity.m_floats[2]);
								_Packet->read<float>(LinearVelocity.m_floats[3]);
								btVector3 AngularVelocity;
								_Packet->read<float>(AngularVelocity.m_floats[0]);
								_Packet->read<float>(AngularVelocity.m_floats[1]);
								_Packet->read<float>(AngularVelocity.m_floats[2]);
								_Packet->read<float>(AngularVelocity.m_floats[3]);
								//
								Node->NetUpdate(Trans, LinearVelocity, AngularVelocity);
							}
						}
					}
Node->NetUpdate( Trans, LinearVelocity, AngularVelocity);

Code: Select all

	void NetUpdate(btTransform Trans, btVector3 LinearVelocity, btVector3 AngularVelocity)
	{
		_RigidBody->activate(true);
		_RigidBody->setWorldTransform(Trans);
		_RigidBody->setLinearVelocity(LinearVelocity);
		_RigidBody->setAngularVelocity(AngularVelocity);
		_RigidBody->clearForces();
		bNeedsUpdate[0] = true;
		bNeedsUpdate[1] = true;
		bNeedsUpdate[2] = true;
	}
The last code block is where the updates are actually applied.

I tried using _RigidBody->GetMotionState()->setWorldTransform() but the scene nodes did not move.
There are some functions like ->SetInterpolatedWorldTransform(), these don't appear to do anything either..?

I'll post an update when I get smoothing/interpolation added.
Also, if bodies are sleeping I will slow down the rate of updates, as well as bodies farther away from the client will update to the client at a slower rate..

If anyone has more ideas/advice on how to more effectively synchronize clients to the server then I am all ears!!
User avatar
KKlouzal
Posts: 65
Joined: Thu Mar 06, 2014 5:56 am
Location: USA - Arizona

Re: Synchronizing Client Server Physics

Post by KKlouzal »

Well I got the simulation to stop exploding. Was an issue with my net library sending garbage/corrupted packets. So now it's just down to smoothing/interpolation.
https://youtu.be/gdiuOsqK8yQ

So I'm curious about these functions:

Code: Select all

		_RigidBody->setInterpolationAngularVelocity();
		_RigidBody->setInterpolationLinearVelocity();
		_RigidBody->setInterpolationWorldTransform();
What do these do exactly? Using them seem to do nothing.
User avatar
drleviathan
Posts: 849
Joined: Tue Sep 30, 2014 6:03 pm
Location: San Francisco

Re: Synchronizing Client Server Physics

Post by drleviathan »

The code comments offer some hints:

Code: Select all

   ·///m_interpolationWorldTransform is used for CCD and interpolation
   ·///it can be either previous or future (predicted) transform
   ·btTransform m_interpolationWorldTransform;
   ·//those two are experimental: just added for bullet time effect, so you can still apply impulses (directly modifying velocities)
   ·//without destroying the continuous interpolated motion (which uses this interpolation velocities)
   ·btVector3 m_interpolationLinearVelocity; 
   ·btVector3 m_interpolationAngularVelocity;
How they are used also supplies some info: btCollisionBody::getInterpolationWorldTransform() is used in btDiscreteDynamicsWorld::synchronizeSingleMotionState() to feed the MotionState the interpolated transform which includes any remainder time between fixedSubSteps. It is also used in btDiscreteDynamicsWorld::internalSingleStepSimulation() before calling btDiscreteDynamicsWorld::createPredictiveContacts().

At one time I was working on a distributed physics simulation where clients would perform local simulation within a limited area but could see moving objects outside of it because other clients would be performing simulation over there and would send updates to the server which would broadcast to other clients. The client would represent other-simulated objects in the physics engine, but would move them kinematically. To reduce drift the KinematicMotionState would use the same integration logic as Bullet, to supply interpolated transforms between updates. When two clients got close and their local athoritative simulations overlapped there would be a negotiation logic as to who was authoritative for the simulation of various objects, arbitrated by the server. Needless to say the overlapped objects would drift in the multiple simulations, would be corrected when authoritative updates arrived through the server, and this would cause problems. The visual snaps could be reduced by interpolating the render object toward the physical one, however interpolating the physical body toward its updated position (rather than just teleporting it) was more difficult and we never got around to actually tackling that problem in a way that could avoid or significantly reduce interpenetration.
User avatar
KKlouzal
Posts: 65
Joined: Thu Mar 06, 2014 5:56 am
Location: USA - Arizona

Re: Synchronizing Client Server Physics

Post by KKlouzal »

Yeah my thought process is going towards something similar, server would be authoritative, sending updates at a dramatically slower rate than clients. Clients would be semi-authoritative in the sense that they were responsible for broadcasting out updates about objects they had ownership on.
The issue is indeed interpenetration. Outside updates coming into the client cause rigid bodies to penetrate and physics does not like this..

Even for a rigid body that has stopped motion and gone to sleep, if the position of the rigid body matches exactly on the client and server, if this rigid body receives an update, it causes it to 'twitch slightly'. Not sure how else to describe it. Will need to check this I suppose, if a rigid body position/location is within 1% of the servers then don't apply the update maybe.
Post Reply