There Are Only Integration Tests

      There is a false dichotomy in testing software: unit test and integration tests. One of the two, the unit test, is a false prophet. There are only integration tests. This begs the question, what are all those files containing the words “unit” and “test” in countless code bases? The good ones are integration tests, even if they go by another name. The bad ones are something else… One thing they are not is anything resembling a test.

      The concept behind this article came to me after performing a large service refactor. In total, I added 2000 lines of code and removed 5000 lines of code. I wanted to keep the service logic exactly the same, but change the code to provide a better platform going forward. So how did I manage to delete thousands of lines of code? Well, I had no choice; my hand was forced. All those lines of deleted code were unit test files that were testing obsolete types removed in the refactor. The purpose of testing is to ensure that nothing is broken by future changes, and now I had a great need for this guarantee. The unit tests, however, were completely useless! And then it struck me: the efficacy of unit tests is inversely proportional to the size of a change. When a large change lands, unit tests, in the best case, need to be heavily modified, and in the worst case are obsolete and need to be removed.

      As a practical example, I maintain a small dependency injection framework in Go called Dihedral. The entire framework is tested by this one, 25 line test file. I’ve changed the internals of Dihedral many times, and whenever a breaking change has been introduced, this test file has saved me, but most of the time it stays out of my way and lets me focus on the things that matter. Additionally, it’s easy to update. A user reported that non-pointer types could not be provided with an error during injection. After fixing the bug, adding a test case was only a few lines of code, and I’m confident this test will prevent the issue from being reintroduced in the future.

      Now let’s define what an integration test is. Wikipedia defines integration testing as:

The phase in software testing in which individual software modules are combined and tested as a group.

      This isn’t a good definition. What’s special about being “tested as a group?” Inherently, testing as a group doesn’t lead to a better test, so let’s drop that from the definition. The important bit is what comes after:

Integration testing is conducted to evaluate the compliance of a system or component with specified functional requirements.

      This part is much more useful. Integration tests aim to verify the “functional requirements,” that is the mapping between inputs and outputs. Integration tests don’t concern themselves with how those inputs are converted into the outputs. They only care about the final result. This is in stark contrast to most unit tests. One of the key staples of the unit test is the need for mocking. Mocking stems from the misguided belief that it is possible to replace a critical section of code, i.e. a call into a dependency, and still have something resembling a test. This belief is expressly false. Mocking does not replace one equivalent piece of code with another. Mocking is the codifying of one’s beliefs into a test file. Generally, one’s belief does not change between writing the code and writing the test, so this restatement is usually a waste of time. Writing a unit test with mocks is akin to saying “Yes, my code really is written that way,” which is a far cry from “Yes, my code actually works.”

      There are two important distinctions to make regarding unit testing and mocking. The first is that a unit test without mocks is really an integration test, because it only tests the “outs” for the given “ins”, and thus is a good thing to write. The second case is when mocks are provided as an alternate implementation of some well-defined piece of logic. For example, an in-memory DynamoDB implementation. In this case, the mocks don’t need to be verified explicitly and there is a contract in place to keep the mock implementation in-sync with the specification. If either or these conditions can be met, then a unit test may not be a waste of time, although it’s still likely that an integration test will provide the best bang for your buck.

      In closing, stop wasting time on unit tests, especially if mocks are required. They won’t help you when you need them, and they’ll slow down development. Write integration tests. They are amazing.