Using ParaTest for running tests faster

I work with Laravel and PHP apps daily and I want the test suites to run fast so I can get back to coding.

I rely on my continuous integration (CI) to run the whole suite, so I rarely run an entire suite locally. But I still get blocked by slow tests when trying to release a feature. This is even worse when trying to fix a bug - I'm sitting waiting for tests to finish so I can release to production and stop the errors 😬.

On occasion, I find myself wanting to run the entire suite locally, but tend to avoid it because it takes so long. So let's look at one option to make it fast 🏃💨.

ParaTest runs tests in parallel using multiple processes, and in many cases, is a drop in replacement for PHPUnit.

Before we begin, let's see how our suite runs with standard PHPUnit. Note: this is a very small project that already runs fast, but you get the point.

Almost 4 seconds for 306 tests. That's pretty good.

But let's make it better!

We can install ParaTest with composer require --dev brianium/paratest and then just run vendor/bin/paratest:

Hmm, that doesn't look right - there are errors everywhere. I said you just install and run. Job done.

Depending on your tests and environment, ParaTest can be a straight swap out for PHPUnit. So let's dive in to see what's going on here. The errors reported are more-or-less the same:

Tests\Feature\ExampleControllerTest::testPostExample
Illuminate\Database\QueryException: SQLSTATE[42P07]: Duplicate table: 7 ERROR:  relation "password_resets" already exists (Connection: pgsql, ...
...
Caused by
PDOException: SQLSTATE[42P07]: Duplicate table: 7 ERROR:  relation "password_resets" already exists

The test is trying to create the password_resets table and failing because it already exists. The test environment is using a PostgreSQL database that is being shared among all the tests running in parallel. Each test is trying to run the migrations and is running into concurrency problems as other tests are also trying to do the same. This isn't going to work in a parallel world.

To fix this we can switch our base TestCase from using the trait RefreshDatabase to DatabaseTransactions and run the migrations against the database before starting the test suite:

<?php
 
namespace Tests;
 
 use Faker\Generator;
-use Illuminate\Foundation\Testing\RefreshDatabase;  
+use Illuminate\Foundation\Testing\DatabaseTransactions;  
 use Illuminate\Foundation\Testing\TestCase as BaseTestCase;
 
 abstract class TestCase extends BaseTestCase
 {
-    use RefreshDatabase;  
+    use DatabaseTransactions;  

Now, the database will be fully migrated prior to starting tests and each test will run inside a transaction that is rolled back at the end, so it doesn't interfere with any other concurrent tests.

It's worth noting that this is specific to the test environment. Since it's running a shared database server alongside the tests, all the tests connect to the same one. It might not be that common, especially in a test environment. More than likely, you are running an SQLite in-memory database and that will have no issue with a drop-in replacement as each test will have it's own separate in-memory database and you can keep the RefreshDatabase trait.

After changing that trait in the base TestCase, a 1 minute job we can try running paratest again:

Aha. All green!

Studying the output, we see it:

• automatically chose to use 12 processes - one for each core (I'm running this on a Ryzen 5), and
• ran in 1.7 seconds

So that's more than 2x faster! That's pretty good for a 1 line change. I'll take that any day of the week.

There are some gotchas though

This is a pretty contrived example and a small Laravel project. The benefits we stand to gain from a test suite that already runs in a handful of seconds is pretty minimal.

In larger projects I've worked on, with more complicated tests and test environment, introducing ParaTest has been much more difficult and time consuming. However, the speed increase you get from running in parallel at a larger scale compounds quickly. I've seen test suites run in 10x less time from switching to ParaTest. And that is for every CI run forever more!

If you want to tackle this yourself, here is a strategy that has worked for me to help migrate:

1. Split your suite into multiple groups.
2. Put tests that work in parallel together in one group and run that with paratest --group paratest
3. Leave tests that need refactoring in another test that runs with phpunit --group default
4. Move tests over as you can refactor them

That way you start reaping the parallel rewards now.