Introduction to the 333networks MasterServer

Questions, remarks and technicalities go in the Master Server section.


  • How do you find remote locations in the digital world? How do you obtain data from a system of which you have no means to access the server address?

    Prelude: Online Gaming
    I started playing Unreal Tournament (UT) Demo online somewhere in the year 2004. This first person shooter could connect to remote servers, allowing people to play together. In the following years I learned how to make one of my websites display in-game statistics of several UT servers, with the UDP protocol and PHP. Often I wondered how my game could tell which IP addresses were hosting UT servers, without going through the world's list of IP addresses and possible game ports.

    These online multiplayer games are organized by external companies, that offer services to connect the game servers with the game players. In the autumn of 2008, all three service companies went down at the same time. For three days, gamers playing with the GameSpy (GameSpy Industries, Inc.) supported games were frustrated that not a single online server showed up in the Online Games list.

    Image

    Problem: Support ending
    When Unreal Tournament originally aired, there were three masterservers: unreal.epicgames.com, master0.gamespy.com and master.telefragged.com. These masterservers provide lists of all available UT servers to everybody who browses for games in the Unreal Browser (UBrowser), which displayed a list of all the available online games (see figure above).

    Since 2004, I hosted my own UT servers. As hobby, I wrote a website to show the in-game statistics, which contained the number of players, which particular level was being played and who were present in the server. In early 2010, the awareness of the problem was raised again when unreal.epicgames.com permanently stopped working. The other masterserver, master.telefragged.com, stopped sending the serverlist, but still collected the uplinks for a similar service from GameRanger. Effectively, thousands of gamers were depending on a single masterserver: master.gamespy.com

    When newer games were created, some of the masterservers shut down or did no longer support UT99. In January 2014, Gamespy announced the following: Gamespy ceasing all hosted services. All services, including the masterserver, would shut down on May 31, 2014. Everybody who bought an expensive game would be left hanging: no more support for online games.

    Various implementations of the masterserver concept were written by corporate factories and have been kept closed-source. After the announcement of the GameSpy shutdown, some companies tried to find an in-house solution, but not all different game developers succeeded in doing so.

    Solution: Providing an alternative
    I learned of an applet embedded in the executable for the Unreal Tournament game, the UCC MasterServer applet. This closed-sources applet allowed for a single-game masterserver to receive and send server information. In name of 333networks, my small game community, I started hosting this applet to provide a backup for the masterserver for a relatively small community. When in 2008 all services went offline for three days, my own masterserver rapidly became popular.

    In 2011 I had gathered enough information to make an imperical model of the masterserver and started designing my own software to fulfill the tasks that were observed in the GameSpy MasterServer:

    Concept
    Every game server sends a beacon to a central server, the masterserver. This masterserver maintains a list of all server addresses (in the format ip:port) and distributes the list for every individual game amongst its players.

    Image

    Above figure shows a schematic overview of how this process works.

    1. Every UT server sends a signal ("uplink") to a masterserver, every minute or so.
    2. The masterserver keeps a list of all these servers, storing the IP and query port for a short time. The IP and query port are essentially the unique address of the server.
    3. When you open your Multiplayer Games menu and select Online Games, you request this server list.
    4. The masterserver tests whether you are really a UT client (note: or any of the other supported games) and decides whether to send you the list with addresses.
    5. After sending the list, the task of the masterserver is complete.
    6. Your UT then performs then "queries" all addresses in this list. This query is a request to each and every server in the list, asking whether this server can please send you the information about itself, such as the name, how many players are currently there, what map they are playing, what type of game it is and several other variables that we discuss elsewhere.
    7. As player, you select one of the servers from the list and decide to double click it. You then join the game, and are completely unaware of everything that just happened behind the scenes.

    Requirements
    I established the following requirements for a masterserver:

    1. Listen for incoming UDP beacons (server heartbeat)
    2. Response to TCP requests (in-game MasterServer list request)
    3. Putting all received server heartbeats in a database, respond to TCP requests only with the servers of which the last heartbeat was received less than 5-10 minutes ago.

    The same script also queries all UT99 servers as 333networks is currently doing for the site. Though this is not essential for the functionality of the MasterServer, it provides an immediate platform for 333networks' server statistics.

    The rest of this document explains through which process I went to find and implement the different parts for the masterserver. The practical implementation can be found on MasterServer-333networks (git repository) and can be seen in action on master.333networks.com. This open-source masterserver has been in action since January 2014.

    Analysis of TCP requests
    First I started off with analyzing the MasterServer response with an Ethernet Sniffer such as WireShark.
    Image
    I figured out how exactly the MasterServer communicates with the UT Client (UBrowser). Who sends what? What does it send?

    Code: Select all
    UBrowser -> Master    Open TCP Connection
    Master -> UBrowser    \basic\secure\XDWAOR
    UBrowser -> Master    \gamename\ut\location\0\validate\3JOdDFU5\final
    UBrowser -> Master    \list\gamename\ut
    Master -> UBrowser    \ip\255.255.255.255:7778\ip\255.255.255.255:7778\...\final
    Master -> UBrowser    Close TCP Connection


    Analysis of UDP heartbeats
    To be found, Unreal Tournament servers send a so-called heartbeat to the MasterServer every once in a while. This heartbeat, as the name suggests, is just an uplink/beacon to let the MasterServer know that the server is still up and running, waiting for connections from clients.

    The setting in configuration file UnrealTournament.ini that is responsible for this (server only) is:
    Code: Select all
    ServerActors=IpServer.UdpServerUplink MasterServerAddress=333networks.com MasterServerPort=27900


    The MasterServer receives the UDP connection, and then processes it into the list that the UBrowser receives. Scanning what is sent back and forth with Wireshark as described in previous figure, indicates that the server only sends one type of signal:
    Code: Select all
    \heartbeat\7778\gamename\ut\


    As one can see, this information contains only the query port of the server, as well as the game name (ut). There is no IP address involved, nor is there anything else about the contents of the game/server. The UDP packet fortunately contains the information about the sender, such as the IP address. Extract this from the socket to determine the original IP address and query port of the server.

    After receiving the first heartbeat, the MasterServer responds with a "secure\validate" challenge to test whether the uplink sender is an actual UT client:

    Code: Select all
    \basic\secure\l8jfVy


    The UT server, if legit, responds with:
    Code: Select all
    \heartbeat\7778\gamename\ut\validate\XuXkWPKC\final\


    This confirms that our UT server is in fact a real Unreal Tournament (Demo) Server and is then added to the list presented in the TCP section above. Further heartbeats are being sent throughout gameplay, every minute or so. The MasterServer does not bother to verify the UT server anymore, as long as it keeps sending heartbeats to demonstrate that it is still ... in the game.

    Testing
    During the development, the masterserver script was tested with a scripted client. This client functioned with the existing GameSpy masterserver and would eventually function with my Perl implementation. The test script code is found below.

    Code: Select all
    ################################################################################
    ## Subroutine getMasterQuery
    ## Get a list of servers from the a MasterServer
    ## Returns an array of [ip:port]s.
    ################################################################################
    sub getMasterQuery {
      my @list = qw();
      my $buf;
      my %r;

      # define 69.10.30.248 28900 (gamespy masterserver)
      my $masteraddr = "69.10.30.248";
      my $masterport = 28900;
      my $hostaddr   = pack_sockaddr_in($masterport, inet_aton($masteraddr));
     
      print "Querying $masteraddr:$masterport\n";

      eval {
        #die on time-out
        local $SIG{ALRM} = sub {die "Failed to connect to masterserver.\n"};
       
        my $data = "";
            $buf = "";
        my  $val = "";

        # set timeout time = 5 seconds (4..0 = 5)
        # set socket and connect
        alarm 4;
          socket(UTSERVER, PF_INET, SOCK_STREAM, getprotobyname('tcp'));
          connect(UTSERVER, $hostaddr);
         
          # part 1: receive \basic\secure\key\
         
          while(length $data <= 21) {
            recv(UTSERVER, $data, 0xFFFF, 0);
            $buf .= $data;
          }
          print "> $buf\n";
          $buf =~ s/\([^\]+)\([^\]+)/$r{$1}=$2/eg;

          # verify correct input received
          if (exists $r{secure}) {
            $val = `./gsmsalg $r{secure}`;
          }
          else {
            die "Unexpected response. Exiting.\n"
          }
         
          # part 2: respond  \gamename\ut\location\0\validate\2/TYFMRc\final\
          my $send = "\gamename\ut\location\0\validate\$val\final\";
          send(UTSERVER, $send, 0);
          print "< $send\n";
         
          # part 3: invoke masterserver list with \list\gamename\ut\
          $send = '\list\gamename\ut\';
          send(UTSERVER, $send, 0);
          print "< $send\n";
         
          print "Receiving... ";
          #receive masterserver list
          while($data !~ /\final\/) {
            recv(UTSERVER, $data, 0xFFFF, 0);
            $buf .= $data;
          }
          shutdown(UTSERVER, 2);
        alarm 0;
      };
     
      # process received data
      # l(ocations, \label\ip:port\) split up in a(ddress) and p(ort)
      foreach my $l (split(/\/, $buf)){
        if ($l =~ /:/){
          my ($a,$q) = $l =~ /(.*):(.*)/;
          push(@list, ($a.':'.$q));
        }
      }

      print "Done.\nFound ". scalar @list ." servers.\n\n";
      return @list;
    }


    Several UT players tested with both UTDemo and UT Game of the Year edition (GOTY). We noticed that the response from the experimental Master was significantly faster than the regular Master. It can either be the case that 333networks does not have such have traffic as can be expected from a commercial MasterServer, or it can be that 333networks sends the serverlist as a whole. With the packetsniffer, I saw that the list is split up in numerous packages of 1448 bytes each. On a TCP connection, sending all-at-once or in smaller packages is a significant difference in time/speed.

    Image
    Image source: http://hooksutplace.freeforums.org/mast ... t2156.html


    Extra functions
    After a month of testing, the following problems occurred (and were fixed shortly after).

    1. A number of servers showed up as archived servers. When servers went down for more than 1 hour, they were considered outdated addresses and skipped in the query cycle. When they came back online, they were not moved back to the list of active servers. This was fixed in two ways. Method 1 updates the last-seen time in the database after the server sends out beacons again. Method 2 queries old servers periodically; here was the original bug that the counter was reset every 3 minutes (regular cycle time) instead of after it finished a cycle of all old servers in the database. The latter method may take up to a few hours before old servers are found online again, the former picks it up instantly.
    2. Player and map name lengths (and some other values) sometimes surpassed the 100 character limit that I put in place. Why you would want to have a 100+ character name leaves me baffled, but I fixed it anyway. I don't like database errors/warnings in my masterserver output. Long names are truncated.
    3. Long secure queries (crash queries) instantly get banned for life. In contrary to UT, long secure queries do not affect the masterserver (well, up until 1 073 741 824 characters at least), but we know the intention of sending long queries now, don't we?
    4. Furthermore some small code fixes, logging options, convenience functions and other things that nobody notices or finds in the source code unless pointed to it.

    Epilogue
    The masterserver software has been running for almost a year now. After the GameSpy service shut down in July 2014, several gaming communities adopted the 333networks masterserver. Because of its fast performance, people refer to it as the "Faster Master". Since July 2014, there have still been a lot of improvements to make the masterserver more efficient.Since September 2014, multiple 333networks-based masterservers are able to communicate with each other. This allows for several backup systems when one or more masterservers go offline in the future.

    The following servers now host 333networks-based masterservers:

    Further reading

    Happy communities

    MasterServer
    --Darkelarious
    Image
    master.333networks.com -- Master Server home
    forums.333networks.com -- Errorist Forums
    irc.synirc.net #333networks -- our IRC channel
    http://git.333networks.com -- Master Server Source code
    User avatar
    darkelarious
     
    Posts: 58
    Joined: 09 Jan 2015 23:12
    Location: Phobos Moon



  • Excellent Article, and very interesting and enlightening indeed! ;)
    = Hook =
    UT Forum: http://hooksutplace.freeforums.net
    UT99 Pro-Redeemer Server: CROSSBONES Missile Madness {CMM}
    UT99 Server IP: 68.232.181.236:7777
    MH Server: CROSSBONES Monster Hunt {CMH}
    MH Server IP: 68.232.183.18:7777
    User avatar
    Hook
     
    Posts: 45
    Joined: 11 Jan 2015 06:17
    Location: Minnesota USA



  • I agree, an interesting and comprehensive read. It's funny that I once wrote a small perl module for quering servers and you started the same but took it to a whole new level! Thank you also for sharing the code, I'm quite happy to find out I'm not the only one "messing around" with perl and UT.

    Two questions though:
    (1) Is it possible to run a master server without the webinterface (and the associated server queiring)?
    (2) Could you expand a bit on the possibility to query other masterservers and get IP's that way?
    Projects: https://github.com/cterveen (all regarded Alpha, Murphy's law applies)
    User avatar
    Rork
     
    Posts: 13
    Joined: 15 Jan 2015 14:14
    Location: Deventer, The Netherlands



  • Rork wrote:Two questions though:
    (1) Is it possible to run a master server without the webinterface (and the associated server queiring)?
    (2) Could you expand a bit on the possibility to query other masterservers and get IP's that way?


    (1) Yes, you can run it without a web interface. If you do not configure apache, the interface will not be accessed. It does however make it more work to gain insight in your masterserver. The sqlite version does not have the module to query servers, in the postgres version you can disable server querying by commenting that service in lib/Masterserver/Core/Core.pm. The next version (expected release somewhere in March/April) will have a config setting for it instead.
    Code: Select all
    # status info for UT servers (333networks site)
    $ae{ut_server_scheduler} = $self->ut_server_scheduler();


    (2) I might add another long-read about it here, but that will take me some time. There is information available in the configuration file (see configuration on git) and some documentation on the 333 wiki.

    P.S.: is there any chance that the module that you wrote is the same as http://wiki.beyondunreal.com/Legacy:Ser ... ry_In_Perl ?
    --Darkelarious
    Image
    master.333networks.com -- Master Server home
    forums.333networks.com -- Errorist Forums
    irc.synirc.net #333networks -- our IRC channel
    http://git.333networks.com -- Master Server Source code
    User avatar
    darkelarious
     
    Posts: 58
    Joined: 09 Jan 2015 23:12
    Location: Phobos Moon



  • Ah, thanks for your answers. So much to do, and so little time.

    No, that's not my code. Mine can be found here. I haven't used it in ages and remember it would hang sometimes.
    Projects: https://github.com/cterveen (all regarded Alpha, Murphy's law applies)
    User avatar
    Rork
     
    Posts: 13
    Joined: 15 Jan 2015 14:14
    Location: Deventer, The Netherlands



Return to Master Server




Information
  • Who is online
  • Users browsing this forum: No registered users and 1 guest
cron