Don’t let full coverage fool you
When playing XCOM 2, the survival and effectiveness of your squad is heavily influenced by the amount of cover available. Half-cover items provide some protection from enemy fire, but full-cover offers the best protection. Though critical to survival, full-cover doesn’t guarantee survival.
When testing your code, it’s common practice to examine code coverage. This is a metric that identifies what percentage of your code is executed by your tests. Higher coverage equates to more lines of code being put through the paces.
But just as full-cover doesn’t prevent your soldier from being pummeled in XCOM 2, having 100% code coverage doesn’t guarantee your code is bulletproof. Tests can execute lines of code, but the Engineer has to make sure the tests verify behavior.
Take this code sample of a square grid-based map, stored in a linear array.
class GameMap(object):
NUM_ROWS = 10
NUM_COLS = 10
def __init__(self):
# Initialize map to all 'grass'
self._map = ['grass'] * GameMap.NUM_ROWS * GameMap.NUM_COLS
def get_terrain(self, x, y):
# grass, road, mountain (impassable), brickwall (impassable)
return self._map[y * GameMap.NUM_ROWS + x]
def is_passable(self, x, y):
grid = self.get_terrain(x, y)
if 'mountain' not in grid:
return True
return False
The get_terrain() function returns the type of terrain at a given map point. To test this function, we only need x and y coordinates that are inside the map area.
import unittest
from main import GameMap
class TestGameMap(unittest.TestCase):
def test_get_grid_returns_value(self):
m = GameMap()
val = m.get_terrain(1, 1)
self.assertEqual(val, 'grass')
Since the get_terrain() function only has a single line, it is guaranteed to be executed by our test function (100% test coverage). Great! Except, this doesn’t test the full range of behavior. What happens if x or y are negative? What about x or y being outside of the map bounds? For both cases, the behavior is “raise an exception when this condition is true”. These are edge cases we should cover in our tests.
One of the benefits of testing your code is to provide real-usage documentation. A new user might assume the only behavior of get_terrain() is to return a terrain value. They might proceed to write their code, assuming invalid map coordinates are handled gracefully. With explicit tests that highlight these edge cases, the new engineer can write safer code.
Another example is when a function doesn’t properly handle return values. The get_terrain() function has 4 return values, 2 of which are ‘impassable’. The is_passable() function relies on these return values to do it’s job. Our tests below also offer 100% code coverage by touching every line of the function.
def test_is_passable(self):
m = GameMap()
m._map[0] = 'road'
val = m.is_passable(0, 0)
self.assertTrue(val)
def test_is_passable_fails_for_mountains(self):
m = GameMap()
m._map[0] = 'mountain'
val = m.is_passable(0, 0)
self.assertFalse(val)
Coverage is at 100% as well, but these tests don’t offer insight into a bug in the code. As it’s written, is_passable() will incorrectly identify ‘brickwall’ as passable. The missing test that checks for ‘brickwall’ will catch this error.
def test_is_passable_fails_for_brickwalls(self):
m = GameMap()
m._map[0] = 'brickwall'
val = m.is_passable(0, 0)
self.assertFalse(val)
This test will fail since ‘brickwall’ results in a True (passable) return value. Only ‘mountain’ terrain returns False (impassable).
Test coverage is a fantastic tool in your software engineering arsenal. Increasing the coverage percentage means more of your lines of code are executed. However, this should not be taken as the sole measure of quality.
