The overworld is a new extremely large feature for ExVenture. Admins can “paint” an overhead map that players can wander around. Exits can be made to rooms in the “old world” for travel back and forth.
I was very happy to get this working, even if there is still a large TODO list.
Gossip
There was some conversation this week on the MUD Coders Slack that made me finally get around to a cross game chat network. I started the Gossip project for this. It’s a simple Phoenix app that has a cowboy websocket handler to get around not using Phoenix channels.
I went with this route so others didn’t have implement the Phoenix channels protocol in other languages. I also wanted something that was extremely stable, since Phoenix Channels has bumped versions fairly recently I don’t want to have my library bump and break any existing connected game.
So far its been a fun exercise and it’s currently working in a simple form of routing messages to connected applciations. I am in the middle of an ExVenture connect.
Mudlet Additions
When connecting via Mudlet, the package down auto downloads the world map. This should save a lot of wandering time for players to get a fully functional mapper.
I also got a patch into Mudlet based on some functionality I pulled out of the ExVenture package. This is available as part of Mudlet 3.10.
Mudlet has been working on Discord integration, and ExVenture is ready whenever that ships. I have basic status setting functionality set up at the moment.
Admin React
To sort of build myself up to what I thought I would need for the overworld, I started adding some nicer interfaces to the admin panel via React components. The first two I did were effects and quest scripts.
Effects are now done via a nice form that dumps to raw JSON. No longer do you need to edit that huge textarea by hand!
Quest scripts are now filled out by a nice form instead of of a text area and I also have a script tester right in the panel. You can see how the NPC would react without having to head in game.
Exits
Room exits got a massive overhaul in preparation for the overworld (and continued afterwards.) They are now one way, but the admin panel makes and destroys both directions. This drastically simplified the usage of exits inside of the code base.
The only tricky part was getting the reverse exit to be created for the overworld, but I will take one tricky spot over it being littered throughout the whole app.
As a bonus, this let me finally add in diagonal movement. You can go north east now.
Character Stats
I revamped the character stats. Movement points are gone, replaced with Endurance Points. They aren’t used yet, but I would really like to let skills use Skill Points or Endurance Points. I might also rename Skill Points sometime in the future to better reflect the split. I want Endurance Points to be more physical based, and Skill Points to be more “mind” based.
The basic attributes have also been reworked. The new six attributes are:
Strength
Agility
Intelligence
Awareness
Vitality
Willpower
They all go in pairs of two to boost the HP, SP, and EP. Strength and Agility boost HP, Intelligence and Awareness boost SP, and Vitality and Willpower boost EP. This feels a lot better than how it was previously and has at least 100% more thought put into what they are and how they’re named.
Web Client
The web client got a major face lift over the weekend. It now has a sidebar chat to track channel messages, so they don’t get lost in the flow. I also tweaked the bottom bar to feel better, paddings and margins are now similar, along with font and font size. Some panels also got squeezed and moved around. Hopefully the new client looks much nicer to you as it does to me.
Next Month
I hope to continue with the overworld and Gossip integration. These are both big features that will be fun to continue with in the coming months.
I am in the process of reading The Art of Game Design by Jesse Schell so I expect some changes to come from that. Namely I want to start thinking of how I want people to experience the game and start tweaking it to better reflect that. I expect this to be a slow process but highly worth the thought put into it.
This week I wrapped up the absolute basics of ExVenture’s largest new feature, the overworld.
This was a huge feature that’s barely been scratched.
Map Editor
First up was I wanted to make a nice way to edit the overworld in the browser. This was my first big React app. You can see it in action below:
I have a lot of ideas on how to make this nicer, but the editor works for now. Things I want to add are: a nicer color picker, which will display the color instead of just using the word; and some way to paint bucket large areas with symbols and colors.
I may also set up a terrain swatch to bundle together a symbol and color (and eventually cell attributes) into a nice picker.
Exit Editor
After the map is created, you need to hook it up with rooms so players can move in and out of the map. This is another react page that does live API updates.
Moving Around
Once exits are created, players will be able to move around freely between the room based zones and the overworld based zones.
Here is movement on MidMUD, going from the room based town of Milay to the countryside around it:
Backing Architecture
Overworld zones are set up as a pool of Sector processes. Each sector handles a portion of the grid of cells in a map, 10x10 right now. The calling or casting process determines which sector to talk to so this should scale well.
Only minor updates for the commands were needed to get the overworld working. Mostly the updates were stripping functionality and making sure they did less on the overworld. Commands like involving items in particular were broken on the overworld, because I didn’t want to implement item storage on the overworld quite yet.
TODOs
I’ve started a GitHub issue to track all the things that are still missing. The big ones right now are:
Items in the overworld
NPCs in the overworld
Say communication should go further than your current cell
Seeing other players on the map
I am looking forward to seeing the overworld slowly fill out now that the basics of being able to get to it and move around are done. It is a huge relief to have this part over.
Mudlet is a client that can be used to connect to MidMUD or any other MUD. They have a way for the server to push a client package down to the client. This lets Mudlet be customized for MidMUD and any ExVenture server.
Through the server push mechanism, you can also auto-update the package. This lets the package stay lock-step with the server as new things are added and changed.
The package now contains and auto-mapper, stat gauges, and a tabbed chat.
Telnet Connection Flow
To go together with the Mudlet additions, I worked on the telnet connection flow. Previously you needed to get a one time password and enter that into you chat.
I changed it up to work more like how Netflix authorizes a remote device. The telnet connection gets an ID and registers it as a session waiting to be authorized. You then click the authorize link the connection is giving you. On this page you can authorize the connection and be signed in when you switch back to your client.
This is so much more convenient than copy pasting a password and I think its just as secure.
Large Scale Metrics
I spent an afternoon a few weeks ago trying to figure out how well ExVenture will scale as more and more data is pushed in and thousands of processes spin up. ExVenture performed wonderfully in my testing. I only tested the number of NPCs performing movement.
I set up roughly 14,000 rooms with 20,000 NPC processes wandering around them. This was on a quad-core Macbook Pro. The 99 percentile for movement was 350 _micro_seconds. I am still in a slight shock about how much Erlang did not care about data size.
Movement stabilizing over timeMovement metrics
Multi-Node Bugs
As expected, turning ExVenture into a distributed app came with a significant number of bugs. Luckily I had some help on the ExVenture Discord in finding them. I set up Sentry to start recording new exceptions as they came. This lets me see what was going on and record them better than a log I never look at.
A summary of the bugs I found so far:
Channel communication was not spanning the cluster
Raft deadlock
Keeping players out of the game until it is online
Rebalance zones as nodes drop out
Current connected players were locking up
Lots of functions were not expecting bad data back
Strange transient data errors, I think due to processes not rebooting properly
A room can crash often enough to tank the entire supervision tree and kill a node
Session recovery exponential back off
Not all of these are fixed, namely the room tanking the entire tree, but most are. Getting in session recovery to be exponential is a big win. A player connected could generate thousands of exceptions by how I had it before.
Game Jam
The MUD Coders Guild had a Game Jam going on during the beginning of May. In the last moments I joined with MidMUD in the cheapest feature I could think of: a random name selection for players signing up.
This is a simple feature, but one that I am glad to get in. Picking a name has to be the hardest part of making a character, so simplifying this for new people is a good win.
Home Page Updates
The home page now includes a web chat client! This was fun to add since it was the prototypical phoenix application. Phoenix channels are finally being used “right” in ExVenture!
The admin UI also got an update. I had been watching [Refactoring UI][refactoring-ui] and became inspired to make the admin look less crappy. There are now flash messages on all actions, and the forms should look a lot better.
If you attach an email to your account, you can trigger a password reset.
If you try using a skill that exists but you don’t have, you get a nicer message for it
Get all items in a room at once
Delete rooms from the admin
Handle web client disconnects, by reconnecting
Send mail from the home page
Disable user accounts
Bug: Adding an item spawning did not trigger its timer
Bug: Session recovery did not start regen
Quests always show their progress as 100% after completion
Bug: several bugs in the telnet protocol surrounding IAC
Bug: web page redirects after signing in was slightly broken
NPCs can have delayed actions
NPCs can have mutliple actions in a single event, queued up
Channel chatter is recorded for replay when connecting to the web client
scan command to view the surrounding area
Who list tweaks
Commands can be marked admin only
Automatic balancing of NPCs, change their level and the stats boost to the minimum for that level
Continuous effect for stat boost
Skills can default target yourself
Target yourself via self
Hide yourself when looking at a room
Format chat messages by capitalizing and adding punctuation
Bug: NPCs targeted you before entering a room
GMCP heartbeat message
Lock movement when skills are cooling down
GMCP skill status message
Next Month
Next month I am going to continue with bug fixing while the app is in clustered mode. I have been getting some suggestions about what to tackle next so I will probably continue with that. I also wouldn’t mind getting to in game forums in the next month.
I might also start switching these to a faster update cycle if I continue to have this many changes.
From the last update of ExVenture, ExVenture can now be configured to run in a multiple node configuration. In this post I’ll show you the basics of how I did that.
ExVenture was heavily geared towards running as a single node to start out. I used Registry very heavily, which only works for a single node. I also have a few local cache processes that out of the gate wouldn’t work well when spanning multiple nodes.
My first thoughts where about splitting up the app into an umbrella app and figuring out how to boot the web on one node, the telnet connection on another, etc. I started talking about this in the MUD Coders Guild, and I got a question of “why?”
This was a great question and got me thinking about what else I could do.
I eventually settled on trying to get the same application booting on all nodes and a leader node starting the processes that can only exist once in the cluster, e.g. the world.
Clustering
First up was connecting up nodes in an automated fashion. Since my app was heavily geared towards being a single node, this should be fine. Each node wouldn’t talk to each other and the entire world would be spun up on each node.
This was extremely simple with libcluster. It was as simple as installing the hex package and adding this to my configuration files:
Once the world was clustered, I started on picking a leader. I had heard about Raft before, but never really looked into it.
If you are interested in clustering at all, I’d highly recommend giving the paper a read. It is very simple to follow along and understand. Which was the main point of creating Raft, a simple to understand consensus algorithm.
For ExVenture, I went with implementing my own Raft module because I only wanted the leader election part of Raft. I don’t need (at least yet!) the rest of Raft.
You can see that in the Raft module. There is a lot to this module that doesn’t need to be covered here, but it boils down to the group picks a leader and that leader calls the subscriptions that care about who was leader.
A leader is picked
Once a leader is picked, the Game.World.Master process uses pg2 to find all of the other Game.World.Master processes and sees what zones are alive in the cluster.
After finding out what nodes are alive it spins up the zones not online across the cluster, using a simple rebalacing algorithm.
Global process registry
I also switched to using the global process registry as part of this. I started looking at swarm but it seemed to be something different than what I was looking for.
I will most likely end up changing this in the future but switching {:via, Registry, {Game.NPC, id}} to {:global, {Game.NPC, id}} was an extremely simple change that worked. So I went with it and haven’t looked back.
Messages spanning the cluster
The game was now officially multi-node and you could play on either iex servers and see other players and NPCs on the other one.
There was only one final step and that was setting up pg2 groups for each of my caches and my communication layer.
This is very simple and can be done as follows in the init function and a slight change to casting to your GenServers.
With this up and running I was able to get MidMUD in a multi-node (3 world servers) set up for production. It has been working out pretty well so far and only once I found a large bug (of not being able to communicate across nodes in channels.)
I will post more updates as I continue enhancing the distributed nature of ExVenture.
You can see everything described here in these two pull requests, #37 and #39.
The last month of ExVenture had a lot of updates to clustering and some extra world details.
The documentation website is exventure.org. You can see the latest additions here on MidMUD, my running instance of ExVenture. There is also a public Trello board now.
The biggest new feature of ExVenture in the last month is the improved support for erlang node clustering. I started with adding libcluster to join nodes together. Then I had the world start spanning the cluster via a really simple leader election.
Next up was using pg2 to have the player registry and send cache updates across the cluster. The last step was using :global as the registration mechanism for world processes. I tried out swarm for this but it had some weird properties of restarting all of the processes when a single one died.
The most recent step was implementing the leader election of Raft. It’s mostly done, but it should handle rejoining nodes and adding nodes to the cluster. Once a leader is picked, I have a set of modules that get called on the winner node. On start up this will spin out the world across the cluster.
Early on in the month I added custom damage types and damage resistances. Each damage type has an opposing stat that reduces the damage. This step also added an “echo back” of the damage that was actually applied. A character calculates the effects to send over, sends them, and hears back what actually happened.
A new command was added, listen. This lets you tune into noises that are in the same room as you. This feature has lots of room to grow as more things make sound.
NPCs can change their status line and listen text via emotes now. As part of this they set a status key to eventually enable gating of events they do. Such as they only start combat if they are in a certain “mood” and can get out of it later on.
For next month I want to continue with the distributed part of ExVenture. I definitely need to get the world rebalacing as nodes fallout and re-join the cluster. I would like to continue with the other game additions from this month. Listening and the npc status engine are both very good foundations for a lot of extra features in the future.
Once I have more work in the clusting of ExVenture I want to do a deep dive blog post.
