What we’re testing
In my last post I used pytest to write a unittest for code which wrote to sys.stdout
on Python2 and sys.stdout.buffer
on Python3. Here’s that code again:
# byte_writer.py import sys import six def write_bytes(some_bytes): # some_bytes must be a byte string if six.PY3: stdout = sys.stdout.buffer else: stdout = sys.stdout stdout.write(some_bytes)
and the unittest to test it:
# test_byte_writer.py import io import sys from byte_writer import write_bytes def test_write_byte(capfdbinary): write_bytes(b'\xff') out, err = capfdbinary.readouterr() assert out == b'\xff'
This all works great but as we all know, there’s always room for improvement.
Coverage
When working on code that’s intended to run on both Python2 and Python3, you often hear the advice that making sure that every line of your code is executed during automated testing is extremely helpful. This is, in part, because several classes of error that are easy to introduce in such code are easily picked up by tests which simply exercise the code. For instance:
- Functions which were renamed or moved will throw an error when code attempts to use them.
- Mixing of byte and text strings will throw an error as soon as the code that combines the strings runs on Python3.
These benefits are over and above the behaviours that your tests were specifically written to check for.
Once you accept that you want to strive for 100% coverage, though, the next question is how to get there. The Python coverage library can help. It integrates with your unittest framework to track which lines of code and which branches have been executed over the course of running your unittests. If you’re using pytest to write your unittests like I am, you will want to install the pytest-cov package to integrate coverage with pytest.
If you recall my previous post, I had to use a recent checkout of pytest to get my unittests working so we have to make sure not to overwrite that version when we install pytest-cov. It’s easiest to create a test-requirements.txt
file to make sure that we get the right versions together:
$ cat test-requirements.txt pytest-cov git+git://github.com/pytest-dev/pytest.git@6161bcff6e3f07359c94a7be52ad32ecb882214 $ pip install --user --upgrade -r test-requirements.txt $ pip3 install --user --upgrade -r test-requirements.txt
And then we can run pytest to get coverage information:
$ pytest --cov=byte_writer --cov-report=term-missing --cov-branch ======================= test session starts ======================== test_byte_writer.py . ---------- coverage: platform linux, python 3.5.4-final-0 ---------- Name Stmts Miss Branch BrPart Cover Missing ------------------------------------------------------------ byte_writer.py 7 1 2 1 78% 10, 7->10 ===================== 1 passed in 0.02 seconds =====================
Yay! The output shows that currently 78% of byte_writer.py is executed when the unittests are run. The Missing
column shows two types of missing things. The 10
means that line 10 is not executed. The 7->10
means that there’s a branch in the code where only one of its conditions was executed. Since the two missing pieces coincide, we probably only have to add a single test case that satisfies the second code branch path to reach 100% coverage. Right?
Conditions that depend on Python version
If we take a look at lines 7 through 10 in byte_writer.py
we can see that this might not be as easy as we first assumed:
if six.PY3: stdout = sys.stdout.buffer else: stdout = sys.stdout
Which code path executes here is dependent on the Python version the code runs under. Line 8 is executed when we run on Python3 and Line 10 is skipped. When run on Python2 the opposite happens. Line 10 is executed and Line 8 is skipped. We could mock out the value in six.PY3
to hit both code paths but since sys.stdout
only has a buffer attribute on Python3, we’d have to mock that out too. And that would lead us back into conflict with pytest capturing stdout as we we figured out in my previous post.
Taking a step back, it also doesn’t make sense that we’d test both code paths on the same run. Each code path should be executed when the environment is correct for it to run and we’d be better served by modifying the environment to trigger the correct code path. That way we also test that the code is detecting the environment correctly. For instance, if the conditional was triggered by the user’s encoding, we’d probably run the tests under different locale settings to check that each encoding went down the correct code path. In the case of a Python version check, we modify the environment by running the test suite under a different version of Python. So what we really want is to make sure that the correct branch is run when we run the test suite on Python3 and the other branch is run when we execute it under Python2. Since we already have to run the test suite under both Python2 and Python3 this has the net effect of testing all of the code.
So how do we achieve that?
Excluding lines from coverage
Coverage has the ability to match lines in your code to a string and then exclude those lines from the coverage report. This allows you to tell coverage that a branch of code will never be executed and thus it shouldn’t contribute to the list of unexecuted code. By default, the only string to match is “pragma: no cover
“. However, “pragma: no cover
” will unconditionally exclude the lines it matches which isn’t really what we want. We do want to test for coverage of the branch but only when we’re running on the correct Python version. Luckily, the matched lines are customizable and further, they can include environment variables. The combination of these two abilities means that we can add lines to exclude that are different when we run on Python2 and Python3. Here’s how to configure coverage to do what we need it to:
$ cat .coveragerc [report] exclude_lines= pragma: no cover pragma: no py${PYTEST_PYMAJVER} cover
This .coveragerc includes the standard matched line, pragma: no cover
and our addition, pragma: no py${PYTEST_PYMAJVER} cover
. The addition uses an environment variable, PYTEST_PYMAJVER
so that we can vary the string that’s matched when we invoke pytest.
Next we need to change the code in byte_writer.py
so that the special strings are present:
if six.PY3: # pragma: no py2 cover stdout = sys.stdout.buffer else: # pragma: no py3 cover stdout = sys.stdout
As you can see, we’ve added the special comments to the two conditional lines. Let’s run this manually and see if it works now:
$ PYTEST_PYMAJVER=3 pytest --cov=byte_writer --cov-report=term-missing --cov-branch ======================= test session starts ======================== test_byte_writer.py . ---------- coverage: platform linux, python 3.5.4-final-0 ---------- Name Stmts Miss Branch BrPart Cover Missing ------------------------------------------------------------ byte_writer.py 6 0 0 0 100% ==================== 1 passed in 0.02 seconds ===================== $ PYTEST_PYMAJVER=2 pytest-2 --cov=byte_writer --cov-report=term-missing --cov-branch ======================= test session starts ======================== test_byte_writer.py . --------- coverage: platform linux2, python 2.7.13-final-0 --------- Name Stmts Miss Branch BrPart Cover Missing ------------------------------------------------------------ byte_writer.py 5 0 0 0 100% ===================== 1 passed in 0.02 seconds =====================
Well, that seemed to work! Let’s run it once more with the wrong PYTEST_PYMAJVER
value to show that coverage is still recording information on the branches that are supposed to be used:
$ PYTEST_PYMAJVER=3 pytest-2 --cov=byte_writer --cov-report=term-missing --cov-branch ======================= test session starts ======================== test_byte_writer.py . -------- coverage: platform linux2, python 2.7.13-final-0 -------- Name Stmts Miss Branch BrPart Cover Missing ------------------------------------------------------------ byte_writer.py 6 1 0 0 83% 8 ===================== 1 passed in 0.02 seconds =====================
Yep. When we specify the wrong PYTEST_PYMAJVER
value, the coverage report shows that the missing line is included as an unexecuted line. So that seems to be working.
Setting PYTEST_PYMAJVER automatically
Just one more thing… it’s kind of a pain to have to set the PYTEST_PYMAJVER
variable with every test run, isn’t it? Wouldn’t it be better if pytest would automatically set that for you? After all, pytest knows which Python version it’s running under so it should be able to. I thought so too so I wrote pytest-env-info to do just that. When installed, pytest-env-info
will set PYTEST_VER
, PYTEST_PYVER
, and PYTEST_PYMAJVER
so that they are available to pytest_cov
and other, similar plugins which can use environment variables to configure themselves. It’s available on pypi so all you have to do to enable it is add it to your requirements so that pip will install it and then run pytest:
$ cat test-requirements.txt pytest-cov git+git://github.com/pytest-dev/pytest.git@6161bcff6e3f07359c94a7be52ad32ecb8822142 pytest-env-info $ pip3 install --user -r test-requirements.txt $ pytest --cov=byte_writer --cov-report=term-missing --cov-branch ======================== test session starts ========================= plugins: env-info-0.1.0, cov-2.5.1 test_byte_writer.py . ---------- coverage: platform linux2, python 2.7.14-final-0 ---------- Name Stmts Miss Branch BrPart Cover Missing ------------------------------------------------------------ byte_writer.py 5 0 0 0 100% ====================== 1 passed in 0.02 seconds ======================