Library Source Code at the end of this article: Rayman git repo
Playground Source Code at the end of this article: Rayman Playground git repo

Preamble

This article is the first in a series of posts about creating a Hex package in Elixir that will implement a common network protocol. In this case, the protocol in question is MQTT. We will implement the MQTT protocol based on the MQTT 3.1.1 specifications, from the ground up, leveraging Elixir's pattern matching for easily processing binary data.

In this first series of articles, we are going to build a low-level implementation of the procol, without client or server behaviour, however, adding that on top of the abstraction this library will provide will be easy.

The intended audience for this series is anyone interested in creating Hex packages, as well as learning more about handling binary data with Elixir. Also, for programmers not familiar with Elixir and its ecosystem, this might give insight into some development practices and the possibilities Elixir and its package system.

Please Note that there is no productive use for re-implementing MQTT in pure Elixir, there are battle-tested solutions already available out there, mostly in Erlang. However, I think it is a great learning opportunity!

  • If you just want to learn about hex packages, this article by Krzysztof KempiƄski might help you faster
  • If you are looking for an implementation of the MQTT protocol in elixir/erlang, check out mqttex or erlang-mqtt
  • If you are looking for an MQTT broker in elixir/erlang, check out emqtt or vernemq

Focus

My goal is to publish at least one article in this series per week. There is no fixed schedule on when this series will end, considering the library built in the original scope of these articles could be extended by a full client and/or server implementation.

In this series, we will heavily focus on the following aspects:

  • Using generic protocol specifications as a implementation basis
  • Matching, handling and generating binary data in Elixir
  • Performing low-level (e.g. Bitwise) operations in Elixir

Also, we will cover the following, rather generic topics:

  • Defining proper specifications for our functions
  • Documenting and explaining our library functions
  • Creating and developing in a usable test project
  • Packaging and distributing our finished library

Let's get started!

There are different approaches to building an Elixir library. Some people like relying only on protocol specifications and documentations for building their implementation, but I would personally recommend to build the codebase for a library inside a project where it can be used right away, or create a live test project. For achieving this, we can use Hex' capability of using local dependencies. But before we dive into that, let's look at how Elixir libraries are structured.

While there are no fixed rules as to how Elixir libraries must be structured, it is undoubtedly the best choice to follow the community's best practices. This usually means that there will be one (or more) top level modules, exposing functions important to the user. Specific and internal functionality will be stored in separate modules within that namespace.

For our library, this could look something like that:

lib/rayman.ex                 # exposes encoding / decoding of packets
+ lib/rayman/mqtt_string.ex   # implements encoding / decoding of MQTT UTF-8 strings
+ lib/rayman/mqtt_packet.ex   # implements encoding / decoding of MQTT control packets
+ lib/rayman/mqtt_error.ex    # implements a common exception format for our library

But let's start small. We are going to generate two new applications, Rayman, which is our library, and RaymanPlayground, which is our playground for testing our implementation. Of course we will write tests for the actual library, but when implementing a network protocol, being able to test it in a live scenario greatly increases its understandability.

Rayman

We create our future library using mix new rayman. For now, we will not add any actual functionality to it but just expose the library's most important methods, decode_packet/1 and encode_packet/1.

For this, we edit lib/rayman.ex, remove all the default code and put in some placeholder functions:

Rayman Playground

For our test implementation, we are going to create a new supervised application (mix new --sup rayman_playground), so we can play around using libraries such as ranch as TCP acceptors later.

In this case, we strip the lib/rayman_playground/application.ex file of its default content and add a simple TCP acceptor using Erlang's gen_tcp. What we do here, when the application is started is:

  • Bind a TCP listener on port 1883, which is the dafault MQTT port. For more details on the option parameters, check out the specifications for :gen_tcp.listen/2
  • Await a TCP connection and accept the connecting client. :gen_tcp.accept/1 will block the execution until a client establishes a connection, since it's default timeout is :infity. Once connection is lost, we loop back to awaiting a new connection.
  • Read incoming data from that TCP connection via :gen_tcp.recv/2 and redirect it to our library placeholder function. After reading, we loop back to read more data from the connection when needed.

Since we are using the :rayman library already, let's add it as a local dependency in our mix.exs!

Setup Conclusion

With this all set up, we have a bare-bones library and an application to test it right away. All that's left to do now is getting an MQTT client for actually testing our library and understanding the MQTT protocol flows. Again, we could implement this without rayman_playground and a test client but it makes understanding the protocol flow a lot easier. This is especially true for protocols far more complex than MQTT. Also, with this approach, we will automatically be writing the server implementation while creating our parsing library.

I use MQTT.fx as client for testing, but you can pick any other client. For a few suggestions, there's a good listing on HiveMQ's Blog listing some awesome MQTT test clients.

To finish off this article, let's connect to our TCP listener and see what our dummy implementation of Rayman.decode_packet/1 is doing. Start the rayman_playground application via iex -S mix and let the MQTT client connect to localhost:1883. If everything is working fine, the output of the packets should be displayed by iex:

Interactive Elixir (1.6.5) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> <<16, 26, 0, 4, 77, 81, 84, 84, 4, 2, 0, 60, 0, 14, 77, 81, 84, 84, 95, 70, 88,
  95, 67, 108, 105, 101, 110, 116>>

And also just to demonstrate this, :gen_tcp will let you handle any kind of TCP connection. We could use wget localhost:1883 to send an HTTP request to our listener and would get the following output:

Interactive Elixir (1.6.5) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> "GET / HTTP/1.1\r\nUser-Agent: Wget/1.18 (darwin15.5.0)\r\nAccept: */*\r\nAccept-Encoding: identity\r\nHost: localhost:1883\r\nConnection: Keep-Alive\r\n\r\n"

The Specification

As mentioned in the introduction, we are going to implement MQTT based on the MQTT 3.1.1 specification. Let's take a quick look at how this document is structured. If you are unfamiliar with specifications like these and/or RFCs, I recommend you quickly read through RFC2119 (don't worry, it's just 2 pages) which explains the terminology that a lot of future RFCs and specifications have adapted.

The MQTT specification is structured very well. One very important thing to note is, that they have highlighted all conformance statements, which is a huge help for the actual implementation. A lot of protocol definitions are cluttered with explanations to caveats such as these conformance statements, however, since MQTT highlights these so well, we can implement the core functionality first and create a backlog of edge-case that we will have to test for and fix later on.

Since MQTT is supposed to be a very lightweight protocol, the packet overhead is kept incredibly small, by only allocating as much space for information as absolutely needed. For understanding how information is nested within the protocol, a basic understanding of what bits and bytes, as well as binary and hexadecimal numbers are, is required for following along with this series.

There are lots of resources out there, but here's a quick refresher before we get into the implementation.

10       The number 10 in decimal
0x0A     The number 10 in hexadecimal
0b1010   The number 10 in binary

In the case of 0b1010, this essentially is 1x8 + 0x4 + 1x2 + 0x1.
Since 8 has the biggest influence on the number's value, the left 1 is called the most significant bit (MSB)
On the other end, the last 0 on the right side is the least significant bit (LSB), since it can only change the value by 1

One byte consists of 8 bits. That's 2 nibbles (2 half-bytes, packs of 4 bits)
Binary is easily converted to hexadecimal, because each nibble can be represented by one character
0b0001 = 0x01
0b1111 = 0x0F

This only scratches the surface and there are lots of cases where it is not as simple as that. However, for getting started with this project, these basics should suffice.

Just a peek

From looking at the specification, this is my proposed roadmap for the actual protocol implementation

  • Implement UTF-8 string handling
  • Implement encoding / decoding of control packet types
  • Implement encoding / decoding of control packet flags
  • Implement encoding / decoding of packet length information
  • Implement encoding / decoding of packet payload
  • Implement conformance statements and edge cases

That's it for part 1! In part 2, we will dive into the actual fun and implement UTF-8 string encoding and decoding for the MQTT protocol.