Networking Code Gets Complicated

Disclaimer: You'll only find this interesting if you're curious about network game programming.  Otherwise you might as well skip this one.  

So I noticed a couple months ago that there was a problem with the positioning of ships far away from the player.  Their position was not updating properly, or not at all, until the player gets within distance of about 2/3 the way to the edge of the minimap to them.  Then they suddenly snap to the correct position.  Now that is the distance as seen on the server, not the client.  The biggest problem on the client is when ships have moved far away from their starting position the player could approach what they believe to be a nearby ship and get right onto them, but that ship is actually very far away and the client is just not getting an update.  

So this was confusing to me, because this was not always the case.  When I originally wrote and tested the network API this appeared to be working fine.  I've also seen it working fine in Broadside previously.  But I have made some updates to the network API since then, and changes to the ships themselves, so I was not sure when this broke or why.  I did though know it had something to do with being subscribed or unsubscribed.  

Object Subscriptions

So to reduce network traffic and support larger numbers of networked objects I created a system built into the network API I call Subscriptions.  The way it works is periodically at a set interval the zone server checks range from all connected players to all other networked objects.  Any objects within that range the player's connection gets subscribed to, and beyond that range they are unsubscribed.  Networked objects are anything the server controls and syncs with the clients.  Ships and cannonballs are the best examples, but there's actually even more stuff.  

Currently I'm running the zone servers at a frame rate of 25 frames per second, and sending position updates for ships specifically every 1/25 seconds, or about every server frame.  That is the rate for subscribed clients.  If the client is unsubscribed to the object I was sending an update once every 2 seconds.  This significantly reduces the amount of data the server needs to send to each client, but shouldn't look that odd since those ships getting slow updates are far away.  

There's more to subscriptions than just that too.  Such as syncing specific variables only to subscribers like how much flooding a ship has, how many crew it has.  When a client gets subscribed all these "SyncVars" are automatically synced all at once to the newly subscribed client, so there's no difference really if it is information the client only needs when they are close to the object (those 2 specifically are just used for displaying above the ship at close range).  

Investigation

So I determined the positioning problem was only affecting objects the player was unsubscribed to, primarily ships.  What is going on with the unsubscribed ships?  Looking at the code I didn't pick up on anything obvious.  I first tried reducing the unsubscribed update frequency to once a second.  No change.  I found a bug that in certain circumstances could cause a network object which snapped to a new position on the client to be moved back to a previous position, which may appear that it didn't actually move.  That's the bug I fixed for 0.7.12 mentioned in the dev log.  That turned out not to be the cause though.  

I investigated if sending messages targeting unsubscribed clients in general was broken (I don't use this in many places in the code, so it was possible), but no dice.  The network API uses a channel system, where each channel has different settings.  Settings include whether to send with encryption, to send reliably, to hold short messages for a short time to allow for other messages on the same channel to be combined, etc.  The JCGTransformSync component that handles syncing positions actually is the only thing which uses channels 2 and 3.  Channel 2 is used for subscribed clients, and channel 3 is used for unsubscribed.  Almost everything else in Broadside is either on channel 0 or channel 1.  Ah ha!  Something might be set up wrong with channel 3!  

Nope, nothing wrong with channel 3.  

I looked at a lot more areas of code, more investigating, and today I finally found the issue as I was walking through all the code involved.  

The Cause

So the issue ended up being in JCGTransformSync itself.  Even though I send position updates 25 times per second to subscribers and now 1 per second to unsubscribed clients, I only do so if the object has moved since the last time an update was sent.  I send the update and save those values.  The next time I try to send an update I check if the object has moved more than a certain amount (very small amount) and only if so do I send the update.  This means that a large fleet, even if you are subscribed, actually generates next to no network traffic if they are not moving.  

The problem was I send the update to all subscribed clients, save the new position sent.  Then I try to send to all unsubscribed clients and first compare the current position against the last sent position.  Well it is comparing against the position just sent to the subscribed clients a moment ago, which is exactly the same as the position now obviously.  So no unsubscribed updates are sent to anyone.  

This bug though has been there since I originally wrote JCGTransformSync though, why didn't I see it back then?  I've only noticed the issue for a few months at most.  

Ahh, the bug has existed since Broadside 0.6.0, since September 2018, but it only became really noticeable starting in 0.7.2 when I changed the update frequency from every 1/10 second to every 1/25 second.  You see at a 25 FPS frame rate when sending updates to subscribers every 1/10 second there will be some frames when I don't try to send an update to subscribers but I do try to send an update to the unsubscribed clients.  The objects are all really distant so I probably didn't notice they weren't happening as often as they should, but the positions wouldn't end up that wildly off as they have been.  Now that I'm sending subscriber updates every frame (even when there aren't actually any clients subscribed it still runs through the code as if it is going to but just doesn't send anything out) there never is a frame when attempting to send to the unsubscribed in which I didn't already attempt to send to the subscribed.  

So now I changed the code so whenever it tries to update the unsubscribed it ignores if the object has moved or not.  It just sends them.  Sending an update every second or two for each object isn't a lot of traffic, but I may revisit this later to track last sent subscribed and unsubscribed positions separately.  But for now this feature should be working.  

Adding Zones

So I've been slow with the current work.  Last month I finished the feature of the master server spinning up and down the various zone servers.  This now makes possible the ability to fill out additional zones, as previously any zones in the game had to be started by the master server and couldn't be spun down.  This put a hardware limitation on the number of zones which could be running at the same time (around 5 on the relatively low spec test server, even though the current live server can support more than 3x that many).

Now I've been adding additional zones, currently I've been adding the Netherlands, North Sea, Baltic Sea, the Shetlands, and some ocean west of Scotland and Ireland.  The plan is over the next few months to add all of Europe and go out into the Atlantic towards North America.  I ran into a problem though with how I was triggering a zone transfer for the player.  What I had been doing, and was working fine up until now, is manually placing trigger colliders onto the map.  When the player enters a trigger collider for an adjacent zone the zone server would then attempt to send the player to that zone.

The problem though was in placing those trigger colliders by hand.  This was easy around the UK since the land provides some easy reference points.  When getting into borders between zones that are out to sea though there are no such reference points.  After struggling with this for about a week, I decided to redesign the system.

Now in code each zone is a list of bounding boxes, with a north west latitude/longitude point and one to the south each.  Already the zone server recalculates the player's lat/long position every second or two.  Now this will be checked against all zone bounding boxes, and if the location correlates to another zone then the zone server will attempt to transfer the player.

This should eliminate problems placing the trigger colliders, which when misplaced can result in the player transferring to another zone then immediately getting transferred back because of bad placement.  This will also eliminate the overlapping area between zone servers where two players can at the edge of a zone actually occupy the same location while connected to two different zone servers.

So I'm optimistic I will have this all completed within the next day or two and pushed out to the live server after testing.  After that I'll start working on the French, Spanish, and Portuguese coasts, and then the Mediterranean.  This will make for a much more expansive game compared to now, letting the player travel for hours discovering new lands.  :)

Edit:
The new zone transfer system and new zones are now undergoing testing in 0.7.11 and it has been very successful.  After additional testing is done, and some relatively minor bugs with the new zones are resolved, the build will be pushed to the Perilous cluster. 

Some gameplay footage

Here's just a bit of raw gameplay combat.  Click below to play the video.  This is a fight between British and French NPC's outside of London I sailed into and joined in.  Nothing special.  

Sometime soon I'll post some better videos with commentary.  But just a taste here for now. 

Things coming along, zones can be started/shutdown as needed

In 0.7.10 I'm now finishing the feature where zone servers can be automatically started and stopped as needed.  A while back I had implemented automatically starting zone servers when a player travels from one zone to the next, but it was never used because when you first log in the login server and the login client scenes still assumed the needed zone server was online.  So up until now all implemented zones have always been started and kept running.  This has limited further implementation of more zones. 

Currently there are 5 zones implemented, covering the entire British Isles region.  My development test server is an 8 core FX CPU, which can support running around 6 zones at most at the same time.  5 works better though.  So I've been maxed out in what I can test before pushing to the live server.  The live server has 16 opteron cores which each core is better performing, so it can probably support around 16 or more zones, but I don't do initial testing on that server.  The live server cluster will eventually have more servers added as well.  I'm targeting roughly 50 concurrent zones when the game goes public, maybe more. 

But these new changes are the most important step for adding new surrounding zones.  So now, when the master server starts up it automatically starts expected high traffic zones only, which are the national capitals.  Currently that is just London (zone 0).  Then as needed other zones are spun up.  When a zone is idle with no players for 30 minutes, it reports this to the master server.  If the master server is in the normal auto mode, it then tells the zone server to spin down.  Later when a player needs that zone again the master server will spin it back up. 

So after this functionality is all tested, this will allow me to expand the game to further areas.  The plan is to do northern Europe, the Mediterranean, out into the Atlantic, and down the African coast, eventually getting to India.  That will be a pretty large area and cover over half of the nations.  Then tackle east Asia and the American east coast to finish adding all nations.  After that it will just be a process of finishing the entire world map. 

Broadsides now implemented

In 0.7.8 you can now fire a broadside either port or starboard using the buttons on the cannon tab in the bottom left.  When you fire a broadside you do not need to be looking down the sights of the cannons on that side of the ship.  You can even be in 3rd person camera mode when doing so. 

How this works is you have to go to Battle Sails.  After 6 seconds at Battle Sails the broadside activation buttons become available.  When you fire a broadside all cannons are fired parallel to the sea, straight left or straight right from the ship.  If you are aiming the cannons, your aim is ignored.  You cannot manually fire cannons on that side of the ship until the broadside has completed. 

Cannons are fired in highly rapid succession, much faster than manual fire.  If your ship has multiple gun decks on each side, ALL GUN DECKS ARE FIRED AT THE SAME TIME.  The ships with 3 gun decks on each side get the most benefit from this.  Even on the largest ships, all guns are typically fired in around 1.5 seconds.  Ships with fewer guns even faster. 

The biggest negative is since you cannot aim guns up/down, ships with guns mounted high above the water may shoot over small ships.  In that case you should continue firing cannons manually instead of using a broadside.  Firing a broadside is most effective against larger ships.  Large Frigates vs large Frigates, or between Ships of the Line. 

Faction Warfare

Now that the nationality system is largely in place, the AI ships' behavior is going to be changed for 0.7.6 to take advantage of this.  Up until now the AI attacks any players, and are all of the Pirate faction.  Starting in 0.7.6 the AI will instead find targets based on relations, including targeting other AI. 

As far as the Pirate faction, you won't notice any change, since Pirates are at war by default with all nations.  What will be interesting is AI will be implemented for other nations, not just pirates.  Most of the AI ships in high populated areas, such as the English Channel, will be replaced with ships relevant to the area, such as Britain and France.  More remote areas will continue to primarily have Pirates.  

Part of the fun will be that Britain and France will often be in conflict, as they were historically.  You'll get to watch these AI fight each other, participate in it yourself, if chased by Pirates you'll see them come to your rescue if you're in range.  Going to make for a much more lively world, where things are happening on their own that you can join in, rather than all the content revolving around the players.  

Flotillas and Taking Prizes

Flotilla Basics

The plan now is to add a pretty fun feature called Flotillas.  These will be AI ships owned by the player, which follow the player around.  Effectively these allow the player to lead their own mini fleet.  This can be a force multiplier for players who wish to play solo, as well as allow for much larger squadrons of players. 

The entire feature isn't planned out yet, but the idea now is that the number of flotilla ships a player can control will be based on their Command skill level.  Maxing out at around 3 AI ships. 

Taking Prizes

This feature will also be the key for capturing ships.  There has been a plan to add boarding enemy ships, but without a way to take control of the ship the only options from a successful boarding would be to take the cargo and release the ship or scuttle the ship.  I wasn't happy with this, because historically boarded ships were often taken as a prize. 

So the plan now will be that flotilla AI ships will need to be ships the player's character has sufficient skills to operate themselves.  Bonuses from the character's skills will not be applied to flotilla ships (so they will operate like all AI ships as if they have no skills).  A boarded ship taken as a prize will follow the same rules.  Once successfully boarded you will get several options:

• Take any/all cargo and fittings
• Release the ship
• Scuttle the ship
• Take as a prize

To take a ship as a prize requires the player's character to have skills to operate that ship.  All surviving crew will be pressed into your service.  I'm probably going to include an option to transfer crew at this point to/from the prize ship.  Any fittings which the player's character does not have skills for will be transferred to the ship's cargo.  For example, if you take a prize ship with guns you don't have skills for, they will be removed from the ship's fittings and end up in the cargo hold.  The prize ship then gets added to your flotilla.  You won't be able to take a prize ship if it would exceed your flotilla limit. 

More Flotilla Stuff

Flotillas will not allow the player to directly control the ships, but will allow some general orders, such as to follow you or attack enemies.  Otherwise the ships will generally run themselves like any other AI ship.  They will need to be fed with ammunition, and can be sunk, like any other ship.  When the player joins a squadron with other players, flotilla ships will also join the squadron. 

Flotillas can be used in a variety of ways.  Certain ship types are not recommended to sail solo.  The most obvious examples are Carracks which lack the firepower to defend themselves and often carry a large amount of valuable cargo, and Ships of the Line which are vulnerable to swarms of smaller vessels.  A flotilla makes these ships more practical for solo players. 

You can sail a Carrack full of cargo and have several Frigates or Corvettes in your flotilla as escort.  Or you sail one of the Frigates and let the AI play online trucker simulator with the Carrack.  Similarly you can sail your Ship of the Line with Frigate and Corvette escorts.  When capturing of cities is eventually added, I imagine flotillas full of Gunboats (such as Ironclads) will be popular for their defense. 

One additional benefit of a flotilla is when sailing solo and your ship is sunk, you get placed into a dinghy and must return to a city to grab a real ship.  Instead you will be transferred to another ship in your flotilla if available.  This helps you stay in the fight longer, and allows the possibility of scooping up your own loot. 

Well, I'm pretty excited about this feature, and expect to have it implemented within the next month or two after the nationality system.  I'm hoping this helps Broadside: Perilous Waters stand out against other similar games.