Elixir OTP Supervision Tether

Posted on 09 Jul 2018 by Eric Oestrich

I got to try out a new OTP Supervision tree pattern in ExVenture, that I am going to call a tether.

Background

I started a new service, named Gossip, that ExVenture creates a persistent websocket connection to.

When this service dies or is not alive when the ExVenture server starts, the websocket connection will crash or refuse to start entirely. It crashes immediately and then cannot reconnect. This ripples up the supervision tree taking the entire application down in short order.

To get around this, I added a layer of supervision trees for just that socket process. My tree looks like this when fully booted:

- ExVenture
  - Gossip.Supverisor
    - Gossip.Monitor
    - Gossip.Supverisor.Tether
      - Gossip.Socket

The Process

The tether supervision starts with no childspecs. After boot the Monitor (or the cluster leader) will start the Socket inside the tether. When the socket process dies, it will eventually crash the tether supervisor causing it to restart.

When the tether supervisor restarts, it will restart with no children breaking the restart loop ("cutting the tether".)

Eventually the monitor process will try to restart the Gossip socket which may crash, causing the tether to crash and so on. Either way the crashing process was contained and the application stayed up.

Conclusion

The code for this was pretty simple, you can see it on GitHub in the gossip folder.

I learned of this at least part of this technique from Adam, over at the MUD Coders Guild. If you are interested in Elixir and multiplayer game programming, come check out the Slack and say Hi.

comments powered by Disqus
Eric Oestrich
I am:
All posts
Creative Commons License
This site's content is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise specified. Code on this site is licensed under the MIT License unless otherwise specified.