There are lots of articles on testing in Elixir, and probably ten times as many for each Javascript frontend framework. But what if we want to test all of it together? - Be advised that end-to-end tests do not replace unit and integration tests on either the backend or frontend, however, I think they do have their place in a good test suite for the following reasons:

  • We can test for specific user workflows that caused issues in the past to ensure these stay fixed forever.
  • Tests are almost entirely reduced to user interaction, so given a working tests setup, tests can almost entirely be defined by QA without knowledge of the software's inner workings.
  • Sometimes we just forget to test little caveats when interacting with the API, this way, we catch them.
  • It looks so cool to watch cypress run, like, seriously!

What we are going to do

  • We use cypress for simulating user interaction on the frontend
  • We create a phoenix router plug that is only accessible in the :test env
  • We isolate our connections using Ecto's sandbox adapter and cypress hooks
  • We expose our factories with a simple JSON API

Please be advised that this article does not conclude with a ready-to-use project. A lot of what is described here heavily depends on your application architecture. This article is meant to show one possibly way on how to do end-to-end testing in a Phoenix application.

Preparing Phoenix

Making sure the webserver is running
For using this kind of tests, we need to make sure our backend is running even in test mode. Phoenix does not do this by default but we can fix that by changing the server configuration in config/test.exs.

If you have not dockerized your application, this might cause issues if you attempt to run the test suite more than once (e.g. in two different CI tasks). In that case you might want to create a separate environment like :fulltest and leave the normal :test environment alone.

Using an end-to-end plug
In our router lib/myapp_web/router.ex, we add a forward that is only used in our end-to-end test environment. This will forward all requests to /end-to-end/* to that plug.

For now, let's set up some scaffolding for this plug, which we will complete later on.

Isolating database access

(This is written for Ecto 2.2.8, I think the API is vastly different in Ecto 3.x)
When running tests within your Phoenix application, the test suite is already doing quite a few interesting things for you. All of the details are documented very well in the Ecto.Adapters.SQL.Sandbox module documentation. Essentially, what you need to know is, that when you run your Phoenix controller tests with mix test, each test is run in a separate process by default (this is what ex_unit does to run tests concurrently). Each test process then checks out a database connection, makes some changes and checks that connection back in when the test is done. This way, changes made within that test are never actually persisted to the database.

Ecto's sandbox pool is based on an ownership mechanism, which means that the process checking out a connection is the only one that can access it. You can however allow other processes to use a connection or run the sandbox in shared mode, which means all processes will have access to a connection. The downside of using shared mode is, that tests can no longer be run concurrently, which is not ideal but it's a tradeoff we are willing to take for this example. It is not impossible to do all of this in parallel, however, it would require some more advanced logic. (Your Phoenix tests will not be affected by this)

We create two functions in our end-to-end test plug, one for checking out a connection and setting it to shared mode and one for checking the connection back in. ownership_timeout is set to :infinity, which means the connection will remain checked out until we manually check it back in.

Checking connections in and out via API

You might have noticed that checkout_shared_db_conn/0 is of arity 0, while checkin_shared_db_conn/1 takes an argument but doesn't care about it. Weird. The reason for this is, that connections are bound to their owner. If the owner process exits, the connection is closed. If you want to check connections in and out via API, this is terrible, since each request will spawn a process that is discarded once the reply is sent.

We add two routes to our plug, which allow us to check-in or check-out database connections. For solving the ownership problem, we spawn an agent when checking out a database connection, that is only terminated once we check the connection back in. This way, the connection stays open for API requests.

So, checkin_shared_db_conn/1 needs to take one argument, since it is expected to be an Agent getter that should care about the Agent's state. We can now send a POST request to /end-to-end/db/checkout, make some changes and then send another POST request to /end-to-end/db/checkin, and the changes will be gone... Hooray!

Exposing our factory

When setting up tests, we usually need to insert some data before our application is in the desired test state. We use thoughtbot/ex_machina factories for that purpose. If you use another way of mocking your test data, you will have to come up with a solution on your own but for ex_machina, this is what we are using. There are careless atom-conversions in this code but since it is only used for testing, that should be fine.

Configuring Cypress

I trust you will be able to set up cypress on your own, their documentation is pretty amazing. Once you have cypress installed and running on your frontend, we need to make some changes to /cypress/support/index.js. The hooks we add here are global and will affect every test we write. Don't worry if the commands looks strange to you, we will define them in the next section.

Alright, now let's look at these commands. We will have to add them manually in /cypress/support/commands.js, using Cypress.Commands.add().

That's it! Our database will be reset when our test suite is started. A connection will be checked out before every test and checked back in after every test, much like with out Phoenix integration tests. Now we can write cypress tests completely independent from each other, each starting with a clean application state. Here's a simple example: