The integration of the new JCGNetwork API, completely replacing the Unet networking API, has been completed. Problems appeared with the design of the smooth syncing code regarding the turning of objects, so that has been disabled for now for a future redesign. Overall though the process was time consuming, stressful, but was relatively free of issues. I'm still looking for issues, significantly fewer bugs were found in the conversion and the new networking API.
So in the immediate term, I'm still looking for new issues that have appeared as a result of the switch, but then returning to working on new feature development. Specifically back to working on the market trading system, and AI ships. The market trading system is already 2/3 done. It is core to making money in the game, so very important. The AI control of ships is core to a lot of the fun in the game outside of player vs player combat, so is also very important.
Currently with market trading players can make purchases from cities, but the supply and demand system is not fully implemented, and players can not yet sell. You also cannot yet place your own persistent market orders (either buy or sell) yet. With AI, right now it will follow the closest player, but has not yet implemented more advanced behavior such as pulling alongside for a broadside, and does not yet know how to fire cannons. Once that part is implemented there will be a bit more fun to be had.
After those are implemented, then implementing the scooping of loot from item drops will be next, which ties in with both having fun fighting AI ships (which already produce item drops when destroyed, just as player ships already do), and ties in with the sale of that loot on the various city markets.
The next feature after that will be completing the randomized map spawn points feature, which will be the semi-random appears of things like AI spawn points, free loot drops, resources such as schools of fish, etc. This is about 1/3 implemented already. This was held back partly by Unet due to the inefficiency of spawning a high number of AI ships in a zone even at large distance to any players. All players would get the full set of updates for these AI ships. Unet did have a feature to handle this, the network proximity checker, but was not a good fit for the game's design. I believe this should be solved with JCGNetwork's subscription system.
At that point the game will be in a good place to finish implementation of the zone transfer system, where when the player reaches the edge of a zone they are automatically transferred to the adjacent zone server. This will then allow for very long distance travel, differences in the difficulty of AI between zones, and the leveraging of the market trade system for the player to make money in game. Then I can begin the work of building out new zones for the player to enter and explore, eventually covering the entire world. The focus initially will be on the British isles, followed by Europe as a whole, then Africa and on to India.
Further in the future will then be resource gathering, manufacturing, player skill training and skill trees, and player owned trading companies. At that point the game will be possibly in a state to start getting some public feedback through a private or even public beta.
Sail the world in this exciting MMO by Joe Censored Games.
Under active development, ETA late 2021
Internal development builds available on Steam by request for a Steam beta key. Note that many features are currently not implemented, buttons may exist but do not work, there are no tutorials available.
Tuesday, September 18, 2018
Monday, September 10, 2018
New Networking API Complete! - JCGNetwork
The new in house networking API has now been completed and tested, and I've started with the task of ripping out Unet and replacing it with the new API - JCGNetwork. I took some inspiration from the Unet HLAPI but no code, and really do everything a whole lot differently. I'm considering releasing it as its own product to help other developers.
JCGNetwork uses a layered transport on top of UDP, supporting multiple channels, channel settings, and flexible message sizes. All channels have packet ordering enforced, remove duplicates, and channels can be reliable or unreliable. JCGNetwork is multithreaded, where the actual sending/receiving occurs on a separate thread, while the processing of the message contents occurs on the main thread.
With Unet I was implementing encryption on a message by message basis, but I've integrated a fast and weak encryption as a channel option of JCGNetwork. The encryption is by no means hack proof, but will make casual reading or spoofing of packets more difficult. Additionally I've integrated both hardware and IP ban lists.
With Unet messages sizes, rate of message send, and message buffer size were all limited in their configuration. In fact, I suspect a bug in the fragmented message system in the Unet LLAPI was my primary problem I was hitting in the end. Even without that possible issue, Unet's fragmented message system supported only a small number of fragments, meaning that message size was always a concern outside of performance. Standard channels couldn't support more than a single packet of data (up to 1500 bytes), and fragmented channels supported only up to 64 fragments, but chopped them down to 500 bytes by default. Messages would also be held to wait for additional messages to combine with by default of 0.1 seconds, and was only globally configurable. Message buffers could overfill without any way of seeing what their status was, without any notification other than in the console log, and without any remedy other than possibly slowing down sends even though your code couldn't know to do so.
With JCGNetwork message fragmentation supports up to max int number of fragments for a single message (2 billion+), and occurs automatically without having to configure the channel for fragmentation. Not that you should try to send a 2 billion fragmented message, but it basically just gets the networking API out of the way as a blocker to sending large messages if needed. JCGNetwork supports small message combining with again a default of holding for up to 0.1 seconds, but this is per channel configurable. You can have a separate high performance channel with a 0 wait time that sends immediately. This is nice because when you need performance for things like close by objects updating their positions, the performance is there, and when the object is distant you can use a channel with a holding time for more efficient use of the network when speed isn't that important. JCGNetwork also places no arbitrary limits on outgoing or incoming buffer sizes either, using basic lists and queues that can again support up to max int number of items, so you'll run out of memory rather than run out of buffer size, but it should never get to that.
The high level networking uses a scheme similar in concept to the Unet HLAPI but functioning entirely differently. Rather than using a not very performance friendly automatic serialization/deserialization system, with JCGNetwork the developer manually writes all serialize/deserialize functions for all messages, RPCs, and sync variables. This takes a little longer to develop the game, but results in a more clear understanding of what is going on, more control, and should be higher performance.
The high level API of JCGNetwork still uses the concept of a Player Object, and uses a unified RPC like system in place of Command, ClientRpc, and TargetRpc of Unet. In JCGNetwork the server can always send RPCs to clients, but the client can only send an RPC to the server on an object owned by that client's connection. This though can be a weakness of Unet, so in JCGNetwork I've added the option on an object by object basis to allow unsafe client RPCs, where any client can send an RPC on that object even without being the owner. Similar to Unet, sync variables only go from server to client, but differently the server will send all variables instead of just what changed, and the server needs to manually set a bool to true when it wants to send those variables to the clients.
Unet had a network visibility system that would only instantiate network objects within a certain range. This had many limitations, such as basing on the physics system so needed a collider, and it may not be ideal to wait to instantiate an object until it is close. There were also a good number of reported issues with this system that never were resolved. JCGNetwork uses a completely different system called object subscriptions. Every few seconds all networked objects will check distance against all Player Objects and add or remove those players' connections from their subscribed lists. All connected clients will instantiate all networked objects, but unsubscribed connections will get fewer or even no updates on those objects, depending on the scripts run on those objects. This allows for things like frequent position updates for objects close by, but objects far in the distance are still seen but their position isn't updated as frequently because it doesn't really matter at that distance.
So the work is on integrating this new system. I expect the work to be completed within a few weeks. I'm really excited about it, so I'll probably put in a lot of extra time so I can see the results.
JCGNetwork uses a layered transport on top of UDP, supporting multiple channels, channel settings, and flexible message sizes. All channels have packet ordering enforced, remove duplicates, and channels can be reliable or unreliable. JCGNetwork is multithreaded, where the actual sending/receiving occurs on a separate thread, while the processing of the message contents occurs on the main thread.
With Unet I was implementing encryption on a message by message basis, but I've integrated a fast and weak encryption as a channel option of JCGNetwork. The encryption is by no means hack proof, but will make casual reading or spoofing of packets more difficult. Additionally I've integrated both hardware and IP ban lists.
With Unet messages sizes, rate of message send, and message buffer size were all limited in their configuration. In fact, I suspect a bug in the fragmented message system in the Unet LLAPI was my primary problem I was hitting in the end. Even without that possible issue, Unet's fragmented message system supported only a small number of fragments, meaning that message size was always a concern outside of performance. Standard channels couldn't support more than a single packet of data (up to 1500 bytes), and fragmented channels supported only up to 64 fragments, but chopped them down to 500 bytes by default. Messages would also be held to wait for additional messages to combine with by default of 0.1 seconds, and was only globally configurable. Message buffers could overfill without any way of seeing what their status was, without any notification other than in the console log, and without any remedy other than possibly slowing down sends even though your code couldn't know to do so.
With JCGNetwork message fragmentation supports up to max int number of fragments for a single message (2 billion+), and occurs automatically without having to configure the channel for fragmentation. Not that you should try to send a 2 billion fragmented message, but it basically just gets the networking API out of the way as a blocker to sending large messages if needed. JCGNetwork supports small message combining with again a default of holding for up to 0.1 seconds, but this is per channel configurable. You can have a separate high performance channel with a 0 wait time that sends immediately. This is nice because when you need performance for things like close by objects updating their positions, the performance is there, and when the object is distant you can use a channel with a holding time for more efficient use of the network when speed isn't that important. JCGNetwork also places no arbitrary limits on outgoing or incoming buffer sizes either, using basic lists and queues that can again support up to max int number of items, so you'll run out of memory rather than run out of buffer size, but it should never get to that.
The high level networking uses a scheme similar in concept to the Unet HLAPI but functioning entirely differently. Rather than using a not very performance friendly automatic serialization/deserialization system, with JCGNetwork the developer manually writes all serialize/deserialize functions for all messages, RPCs, and sync variables. This takes a little longer to develop the game, but results in a more clear understanding of what is going on, more control, and should be higher performance.
The high level API of JCGNetwork still uses the concept of a Player Object, and uses a unified RPC like system in place of Command, ClientRpc, and TargetRpc of Unet. In JCGNetwork the server can always send RPCs to clients, but the client can only send an RPC to the server on an object owned by that client's connection. This though can be a weakness of Unet, so in JCGNetwork I've added the option on an object by object basis to allow unsafe client RPCs, where any client can send an RPC on that object even without being the owner. Similar to Unet, sync variables only go from server to client, but differently the server will send all variables instead of just what changed, and the server needs to manually set a bool to true when it wants to send those variables to the clients.
Unet had a network visibility system that would only instantiate network objects within a certain range. This had many limitations, such as basing on the physics system so needed a collider, and it may not be ideal to wait to instantiate an object until it is close. There were also a good number of reported issues with this system that never were resolved. JCGNetwork uses a completely different system called object subscriptions. Every few seconds all networked objects will check distance against all Player Objects and add or remove those players' connections from their subscribed lists. All connected clients will instantiate all networked objects, but unsubscribed connections will get fewer or even no updates on those objects, depending on the scripts run on those objects. This allows for things like frequent position updates for objects close by, but objects far in the distance are still seen but their position isn't updated as frequently because it doesn't really matter at that distance.
So the work is on integrating this new system. I expect the work to be completed within a few weeks. I'm really excited about it, so I'll probably put in a lot of extra time so I can see the results.
Subscribe to:
Posts (Atom)