Awesomenauts - Dedicated Servers

Introduction

Dedicated servers have been one of the biggest issues discussed on the forums.  Its one of the key concepts holding back Awesomenauts*.  This page is dedicated (no pun intended) to list the forum topics and general ideas around dedicated servers.

Notable Quotes

wrote:

 

Client side hit detection can be and is done in various dedicated server implementations, and actually works better and more consistently. As well as hybrid schemes.

Furthermore there are superior schemes for lag compensation in authoritative servers (e.g. source engine) that have as good as the same effect with better cheat resiliency.

There are experiments showing the lag of a dedicated CounterStrike server can be far below the input and display lag of a standard PC. So that's really the benchmark of what's been achieved in practice and dedicated servers beat P2P implementations by an order of magnitude in the lag stakes.

Clearly Awesomenauts can certainly do better by switching away from P2P, whatever particular approach is used.

conzar wrote:

 Click here to expand...

There are 3 main problems with the current networking of the game.

1. The server is hosted locally on 1 of the client's machine
2. Server Hacks
3. No regions

Problem 1
The first problem is the biggest issue. Most Internet users have asymmetric bandwidth. IE, the download speeds are significantly faster than upload speeds. Assuming the game uses a traditional server/client model, all clients not local to the host (server) are at a significant disadvantage. This can be made very clear when someone from North America (NA) tries to connect to a game that I host in NZ. They will experience lag that is unplayable due to my location and poor upload speeds. I get about 8Mb/s download but only 800kb/s upload which doesn't take into account latency (ie the number of hops packets take from NA to NZ). There are other factors for poor performance for local hosts such as shared network access (where other users on their network are using bittorrent for instance).

Problem 2
Since the server runs locally, its possible for a player to hack the server. There are several methods for this such as injecting code and memory hacks. There is a thread dedicated to how to inform Ronimo of said cheaters.

Problem 3
Players located nearby should be able to guarantee a playable level of lag; however, if there is no division by region, than its possible for a player from NA to connect to a server in a region half way across the world which could mean unplayable lag for that player. Basically, there is no expectation that when a player chooses to "battle", that they will be connecting to others with similar lag.

The Fix
Dedicated servers eliminates the above problem by moving the server from one of the clients machine to a server(in the cloud) hopefully nearby (more about regions later). Given that most players have better download speeds, everyone will get about the same lag/latency. Dedicated servers also means that Ronimo has complete control over the server and can guarantee that hacks made easy by being located locally are not possible. I'm not saying that its impossible to cheat; however, it will be much harder. I also strongly suggest that Ronimo maintains control over the server and doesn't release it to the public.

The How
I suggest hosting the dedicated servers on a cloud service provider like Amazon EC2 (its the standard). Amazon provides a lot of tools that allows servers (running in VMs) to spin up automatically to accommodate high demand and spin back down in low demand times (for saving money). Amazon also has all of the regions covered (NA, EU, Oceania, etc). The Amazon region makes it convenient for the developers since they can easily switch regions on the client to point to the different amazon regions. Puppet can also be used to provide orchestration for your VMs.Foremen can be used for provisioning (there is an EC2 provider). Foremen integrates with puppet by default. Since the server code is run on the client (currently), I imagine the code is already separated and so creating a dedicated server shouldn't be too difficult. I would suggest packaging the server up as a deb using FPM (FPM makes packing easy). 

By making the server a package (note, not releasable to the public), its easier to track the server version and operating system tools are designed to handle such packages. You could also host your own Ubuntu repos (I suggest Ubuntu Server 12.04 TLS). By having say 2 repos, 1 production and 1 test you can test the latest version of the server by uploading to the test repo. Once you are happy with this version, its easy to move that package to the production repo where you roll out the changes to the production servers. 

Another way to do it is to just spin up new VMs that pull from the production repo (with the latest changes), those new VMs are tested. Once they are found to be working correctly, the global server can add the available server IPs and remove the old servers (making sure to transition users off the old servers). 

Funding
All of the above can and probably should be done to some extent. Ronimo will need a systems engineer if they don't already have one (if they don't have one, I would definitely be interested). Getting the work done is a minor problem compared to funding for the bandwidth and hosting costs. With Amazon, you basically pay for what you use. I have no idea how much bandwidth is used for a 20 - 30 minute game. Ronimo would need to run the numbers but I suspect that they need an alternative source of funding since the price of the game might not cover the additional operating costs. 

This brings me to offering a possible solution. I suggest keeping the "local" server option available especially for those that want LAN play. But for those that want to play competitively online, give them an option to either subscribe ($5 a month) or a yearly subscription ($65). Your economist will need to figure out the right price points but I'm sure its possible given other companies can do this with one off payments (Guild War 2, Starcraft 2, etc). 

Conclusion
Awesomenauts is an awesome game and I would like to make it even awesomer! If Ronimo would like my help to achieve the above (assuming they aren't already doing the above), than I would be more than happy to offer my services. I am currently a DevOps/Site Reliability Engineer/Systems Admin that has experience with the above deploying locally to libvirt/KVM servers. (Amazon uses Zen as their Hypervizor btw). I have also deployed a limited number of VMs on EC2 so there are a few things that I would need to work through (foreman with the ec2 provider is new to me). I think Awesomenauts is already a great E-Sports game and hope that it has a bright e-sports future!

 

Dedicated Server Forum Post

Note, Dedicated Servers are mentioned in many more posts but the posts listed below mention dedicated servers in the OP's thread.