To Mock or Not to Mock; that is the Test(ion)

Pre-pandemic, I’m in the office and we’re having a heated conversation discussing testing and how things should work and there’s this point where one of my co-workers refers to a piece of code as a mock and I look him dead in the windows of his soul and say…Biiiiiiiii…don’t call it a mock.

The piece of code we were referring to was an in-memory abstraction of a client to which I was adamant about NOT calling it a mock, but an in-memory client. It’s not that he was wrong, an in-memory client CAN be a mock, but because it’s an implementation of the abstraction, you have to store implementation details in the in-memory client rather than the test. I was adamant about calling it (the piece of code) by its real name, so we wouldn’t expect something from it it couldn’t give. This article seeks to describe:

• How to identify the singular purpose of a mock (in the context of tests)
• How coding style can affect code coverage
• How modularized architectures can take advantage of mocks

A mock is a piece of code that can be used to abstract away certain functionality (especially for testing) that would otherwise be difficult to impossible to implement or re-create. For example you may find that it’s easier to run tests if you abstract away kafka into a mock rather than using actual infrastructure; a mock could save you time and complexity by not having to worry about implementation details, but this is a double edged sword. Depending on what you’re mocking, sometimes the implementation details are ALL that matters; they are best implemented in architectures that use some kind of OOP (Object Oriented Programming); whether that’s inheritance or composition, a mock REQUIRES that you have some kind of abstraction that can be replaced/injected.

For example: your metadata component could be implemented in-memory or using a database or something like kafka (just to keep it interesting); properly abstracted code can function without changing the contract/interface. Clearly there are some implementation details that make each a bit unique but that shouldn’t be a mock’s responsibility.

The best way to visualize the problem that mocks are supposed to solve is to look at it from the perspective of code coverage, the code within a mock shouldn’t increase code coverage directly but it should enable you to create situations that otherwise would be difficult to impossible to create. Imagine the difficulty of trying to cause a REST server to fail with a timeout as opposed to the much easier connection refused.

When tasked to develop a mock, a developer may attempt to codify the developers “idea” of an implementation with no dependencies (e.g., in memory). This is an anti-pattern, one that takes forever to notice. When developing a mock with a “working” implementation inside, it’s REALLY easy to implement bias in forms such as:

• This is how I think it works
• This is how it HAS to work in order for the tests to pass
• I must modify the functional code to match the implementation within my mock

A HARD lesson I learned is that tests confirm expected behavior; one or more tests will codifying how you expect your code to respond to known stimuli (input) and have an expected output. Sometimes, when you implement an anti-pattern like an in-memory implementation, you can split that expected behavior between the functional code itself and the mock. You should always ask yourself, “If I could use the real thing, why aren’t I?” Here is a short list (not comprehensive) of things I think that mocks (if properly implemented) are really good for:

• You want to test very specific errors that are difficult to re-create organically
• You have two consecutive operations using the same mock/abstraction and you need the first to succeed and a subsequent operation to fail
• You want to test client-side timeouts
• You want to generate specific output that otherwise would be impossible (this should be limited to negative cases)

You may be asking yourself, “Well, how do I create a mock that’s not all…anti-pattern-y?”

You create an implementation where you can “set” what the output is before you run the command and the mock simply outputs it. This ensures that the test maintains the singular responsibility of housing expectations for a given implementation. The test has the option to say that two implementations should have the same expectations or that “this” test is confirms expectation of a given situation (e.g., if the implementation succeeds, then immediately fails in a dependent operations).

This idea of a mock where you just set the outputs works really well in serialized tests, but has some very clear challenges when you need to run asynchronously; the start of the solution is to make the mock a singleton and create an instance per thread.

A more practical way to to know if you’re “doing it wrong” is to use code coverage as a metric. The code within a mock shouldn’t count towards your code coverage because you wouldn’t use it outside of testing. If your mock doesn’t increase your code coverage, then you have “failed the test”…so to speak: adding implementation details to your mock doesn’t increase your code coverage, it increases the complexity of your code base without a clear benefit.

Code coverage as a metric (by itself) is also an anti-pattern; certain coding styles are better suited for code coverage (but not readability) while others are suited for readability, but not coverage. For example in Go, we often check for an error and if an error is found, we immediately return which omits the remainder of the function from being able to be covered. Alternatively if we nested some of this decision making rather than immediately returning, we could artificially increase code coverage. Mocks assist in this in that it’s REALLY difficult to create a situation where an underlying client doesn’t return an error the first time it’s called, and then returns an error the second time it’s called (within a given function).

Modularized code, especially code that implements OOP, can be an incredible chore to test. Keeping code coverage in mind, mocks can simplify those integration tests by isolating code that you have to implement to the layers or components that the code being tested is aware of. This is easier said than done, but in a sufficiently complex code base, this is one of the most efficient solutions to avoid having to _actually_ run and configure every implementation/dependency. The easier said than done part has to do with NOT having leaky abstractions and properly defining borders between layers/components which in my opinion is the MOST difficult part about OOP.

Intelligently using mocks for integration tests allows you to effectively turn what would be an integration/functional test into a unit test. Unit tests are characterized by being able to test “one” thing such that a given input will ALWAYS return a known output (or more specifically expected behavior). Mocks allow you to fix the output of a given component such that it becomes a part of the control: the fixed input to test expected behavior. If a code base is properly abstracted, it should be safe to test against mocks and feel confident when you actually put things together they’ll work as expected. I’m NOT saying that you should forgo integration tests altogether with real components, dependencies and infrastructure, but what I am saying is that you can do significantly less of it. In addition, I’m making a strong assumption that the contracts used for your components don’t allow you to create nonsensical output (e.g., for Go you can trust that your functions won’t provide you a nil pointer and a non-nil error).

I think that making more robust tests is just as complex as writing the code you’re going to test; understanding the rules you can bend and the ones you can outright break can save you time, reduce complexity and increase the overall readability of your code as well as communication of expected behavior. Understanding how mocks, leaky abstractions and overall architecture can contribute to your ability (or inability) to test and how the anti-patterns above present themselves within your code can generally make your a better developer; even if its unreasonable to fix the code. That’s OK.