Testing The "Untestable"

As your software solution grows in maturity, the number of tests required to ensure full test coverage can grow exponentially. This is because each unit of code must be tested in isolation before its integration with other parts of the codebase is tested. As the number of tests grows, so too does the duration of a single test run. This is why unit tests are so crucial to an efficient development cycle, not only can they concisely locate bugs, they also provide immediate feedback. This rapid feedback allows developers to quickly test that new code behaves as expected without breaking any existing functionality.

Untestable units of code

When we talk about what a unit test is, we can reduce its purpose to a simple definition - given we provide specific input to a specific function, we should receive a specific output or return value. This logic seemingly breaks down in certain circumstances.

Fragile functions - testing a function that may fail due to external factors such as network issues or expired API keys
Nondeterministic functions - testing a function that does not produce a predictable result, such as functions that use random numbers
Mutator functions - testing a function that has no return value or that mutates a property or performs an action against some resource which can not be queried (such as a function which makes a POST request or writes to a log file).

‍

Mocking and Patching - testing the “untestable”

A mock object is a fake representation of a real object within your solution. Using a mock object instead of its real implementation allows us more control over the behavior of the code. If we take the fragile function scenario from above, a temporary network issue may result in intermittent failures in our tests that require external API calls. In order to establish more control, we can use a mock object to simulate our external services. Although using a mock will result in the external service never being invoked, we can test that the right methods are called the right number of times with the right arguments. Patching intercepts import statements, returning a mock instance that we can configure to establish more control over our test environment.

‍

Hands-on Python examples

Let's take a look at some real examples where mocking can help us test seemingly untestable (or at least difficult to test) functions.

A simple class that has mutator functions that write to log files and have no return values

Functions that call external services, in this case - writing to a MySQL database

Functions that use file IO (input/output)

‍

The Student class

This is a simple class, it has a constructor that takes in a name and a grade, and setters and getters which either modify or return a property value (respectively). Each function also logs the associated action for auditing purposes.

import logging class Student: def __init__(self, name, grade): self.name = name self.grade = grade self.logger = logging.getLogger(__name__) def get_name(self): self.logger.debug("Getting name") return self.name def get_grade(self): self.logger.debug("Getting grade") return self.grade def set_name(self, name): self.logger.info("Setting name") self.name = name def set_grade(self, grade): self.logger.info("Setting grade") self.grade = grade

So the question is how can we test this logic. The getters, also known as accessor methods, are straightforward enough to test, you simply call the getter function and assert that the returned value matches what you expect. The setters, also known as mutator methods, are also simple in theory, the issue is that after calling a setter, your test will also require a call to a corresponding getter function to confirm that it has worked. This means a flaw in the getter logic could potentially lead you to believe the setter logic is broken.

But what about the audit logs, where we have no setters or getters to leverage. This could be covered by integration tests - where we parse an actual log file to check the contents, however we are interested in immediate feedback, so instead we utilize mocking to unit test the logger logic.

@patch('logging.Logger.info') @patch('logging.Logger.debug') def test_student_logger(debug_mock, info_mock): student = Student("John", "A") student.set_name("Jane") student.set_grade("B") assert student.get_name() == "Jane" assert student.get_grade() == "B" assert debug_mock.call_count == 2 assert debug_mock.call_args_list == [ call("Getting name"), call("Getting grade") ] assert info_mock.call_count == 2 assert info_mock.call_args_list == [ call("Setting name"), call("Setting grade") ]

By mocking the debug and info logger, we can check how many times they have been invoked and the arguments they have been invoked with. The code shows that we have patched different parts of the logger, and performed some assertions around how many times they are called in the code and what arguments have been passed to them.

It is important to note that the order of your patches should be reversed, as they are applied bottom up. Check the official python docs to read more about this nested patch decorator behavior.

Working with Databases

Let's apply what we’ve learned to another common use case. Use the following MySQL commands to create a database called book_store and added some entries to the books table. The challenge is now how do we test that our python code interacts with this database correctly?

CREATE DATABASE book_store; CREATE TABLE book_store.books ( title VARCHAR(255) NOT NULL, author VARCHAR(255) NOT NULL, price DECIMAL(10,2) NOT NULL, PRIMARY KEY (title) ); INSERT INTO book_store.books (title, author, price) VALUES ('Harry Potter', 'JK Rowling', '10'); INSERT INTO book_store.books (title, author, price) VALUES ('Lord of the Rings', 'Tolkien', '20'); select * from book_store.books;

First, write the python code to read from, and write to the database. The following code assumes you have used root as the username and the password and are working with a database deployed to a local instance of MySQL:

import pymysql.cursors def read_from_db(title): connection = pymysql.connect( host='localhost', user='root', password='root', db='book_store',charset='utf8mb4', cursorclass=pymysql.cursors.DictCursor ) with connection.cursor() as cursor: sql = "SELECT `author` FROM `books` WHERE `title`=%s" cursor.execute(sql, (title)) result = cursor.fetchone() return result def write_to_db(title, author, price): connection = pymysql.connect(host='localhost', user='root', password='root', db='book_store',charset='utf8mb4', cursorclass=pymysql.cursors.DictCursor ) with connection.cursor() as cursor: sql = "INSERT INTO `books` (`title`, `author`, `price`) VALUES (%s, %s, %s)" cursor.execute(sql, (title, author, price)) connection.commit()

‍

Next come the tests. If we attempt to actually connect to the database in our test, we run the risk of the tests failing due to the database not being created, the table not being created, mysql not being installed locally, or the install being configured incorrectly. We also run the risk of really slow queries if the table size grows dramatically. We once again turn to mocking.

@patch('pymysql.connect') def test_read_from_db(mock_connect): expected_data = [{'author': 'Tolkien'}] mock_cursor = Mock() mock_cursor.fetchone.return_value = expected_data mock_connect.execute = Mock() mock_connect.return_value.cursor.return_value.__enter__.return_value = mock_cursor title = 'Lord of the Rings' actual_data = read_from_db(title) mock_cursor.execute.assert_called_once_with( "SELECT `author` FROM `books` WHERE `title`=%s", (title) ) assert actual_data == expected_data @patch('pymysql.connect') def test_write_to_db(mock_connect): mock_cursor = Mock() mock_connect.execute = Mock() mock_connect.return_value.cursor.return_value.__enter__.return_value = mock_cursor title = 'Programming Python' author = 'Mark Lutz' price = 40 write_to_db(title, author, price) mock_cursor.execute.assert_called_once_with( 'INSERT INTO `books` (`title`, `author`, `price`) VALUES (%s, %s, %s)', (title, author, price))

‍

You’ll notice that in these tests, we have tested logic which has no direct return value. We test that the call to the mutator (write_to_db) results in the execute function being invoked a specific number of times with very specific parameters. For example with our write to database logic, we can at least confirm that the insert query is called once with the correct insert query.

File IO

Finally we look at file IO, which historically has been notoriously difficult to unit test. The basic logic for basic file IO is trivial

def read_from_file(filename): with open(filename, "r") as file: return file.read() def write_to_file(filename, data): with open(filename, "w") as file: file.write(data)

To test this logic without actually interacting with the underlying file system, we use some out of the box mocking functions.

from unittest.mock import patch, Mock, call, mock_open @patch("builtins.open", new_callable=mock_open, read_data="data") def test_read_from_file(mock_f): contents = read_from_file("test.txt") mock_f.assert_called_with('test.txt', 'r') handle = mock_f() handle.read.assert_called_once() @patch("builtins.open", new_callable=mock_open) def test_write_to_file(mock_f): write_to_file("test.txt", "data") mock_f.assert_called_with('test.txt', 'w') handle = mock_f() handle.write.assert_called_once_with("data")

The built in open allow us to mock the read and write functions associated with file IO. This means we can easily test units of code that incorporate the file system. It is worth exploring the official python documentation to see what other hacks and tricks are available.

‍

Conclusion

These hands-on examples should highlight the simplicity of mocking to test code deemed difficult to test, specifically, mutating, indeterministic or fragile functions. Rather than leaving all the heavy lifting to integration tests, developers can mock calls using the patch decorator, and make sure that each call is invoked the correct amount of times using the correct arguments. The immediate feedback provided by unit tests can help developers produce better code quickly and provide all parties with a peace of mind due to increased code coverage.

‍

Snowmate - Your Whitebox Testing Mate

What if you had the option to continuously validate your code by adding tests that simulate actual data, significantly reduce the time of writing tests, and add coverage and validation to the code in real-time. If this interests you, sign up now.

‍