CIT 590 Unit Testing and TDD (Python)
Fall 2015 , David Matuszek

Tests and Testability

The most powerful new programming technique--TDD, or Test-Driven Development--results in better, more correct code, and practically guarantees a continuous rate of progress. However, TDD depends heavily on the ability to write and execute tests much more easily than in the past. Therefore, we first discuss the unit testing framework.

Unit Testing

The first unit testing framework, JUnit, was invented by Kent Beck and Erich Gamma in 1997, for testing Java programs. It was so successful that the framework has been re-implemented in every major programming language. Here we discuss the Python version, unittest.

Unit testing is testing individual "units," or functions, of a program. It does not have a lot to say about system integration--whether the various parts of a program fit together. That's a separate issue.

The goals of a unit testing framework are:

To make it easy to write tests. All a test needs to do is to say that, for this input, the function should give that result. The framework takes care of running the tests.
To make it easy to run tests. Usually this is done by clicking a single button or typing a single keystroke (F5 in IDLE). Ideally, you should be comfortable running tests after every change in the program, however minor.
To make it easy to tell if the tests passed. The framework takes care of reporting results; it either simply indicates that all tests passed, or it provides a detailed list of failures.
- The methods under test should not ask for any input or produce any output -- certainly no output that you need to look at. The testing framework should tell you all you need to know.
- Logic should be separate from input/output in any case.

Let's be honest: If it isn't easy, nobody is going to do it. Programmers are only human. The genius of the JUnit framework (and similar frameworks for just about every language) is that it does make it--well--a lot easier, anyway. Easy enough so that the next time you are too lazy to write good tests, and consequently spend hours debugging, you'll promise yourself to use TDD the next time.

For a first example, we will use the following code:

def add(x, y):
    return x * y

and save it in a file named arithmetic.py.

We write our tests as a set of functions. Tests are usually put in a separate file, and we'll do that here. We'll call the file testArithmetic.py. Since it is a different file, it will have to import arithmetic in order to access our add method. It will also have to import unittest, Python's unit test framework.

Finally, we need to put our test methods in a class, and tell unittest to run the methods in that class. Because they are in a class, we need to use the variable self in several places. We haven't discussed classes yet, so for the time being, just follow the pattern shown below. Here's the complete structure:

from arithmetic import *   # import everything from your module
import unittest  # This loads the testing methods and a main program

class TestArithmetic(unittest.TestCase):  # use any meaningful name

    ## Your test methods go here. Indent your
    ## methods, because they belong inside the class.
    
unittest.main()  # outside the class--this tells the framework to run

Here's an actual test:

def testAdd(self):
    self.assertEqual(4, add(2, 2))

and here are the results of the test:

.
----------------------------------------------------------------------
Ran 1 test in 0.009s

OK

(plus several other lines, which you can ignore)

You probably noticed that there is a mistake in the add method. I "accidentally" picked a pair of numbers that would cause the test to pass (there is one other pair). We can put additional tests within the same method--in fact, it's just a Python method, so we can put arbitrary Python code in it. For example,

def testAdd(self):
    self.assertEqual(4, add(2, 2))      # first test
    self.assertEqual(10, add(3, 7))     # second test
    self.assertEqual(90, add(-10, 100)) # third test

With this code, the "first test" will pass, but the "second test" will fail. The failure will halt execution of this test method (your other test methods will still run), so the "third test" will never be used. The result looks like:

F
======================================================================
FAIL: testAdd (__main__.TestArithmetic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/THUMBDRIVE/PythonPrograms/arithmeticTest.py", line 15, in testAdd
      self.assertEqual(10, arithmetic.add(3, 7))     # second test
      AssertionError: 10 != 21
----------------------------------------------------------------------
Ran 1 test in 0.027s

FAILED (failures=1)

(plus several other lines, which you can ignore)

Notice that the test shows us:

The line number where the error was detected (in testAdd, not in add itself),
The assertion that failed, and
What the actual result was, not just what was expected.

How many tests should you have?

Test at least one "typical" case (for example, 3+7==10 above). However, if this is all you do, you are following thehappy path where everything is sweetness and light, everyone loves everyone else, and mistakes never happen. In other words, this isn't good enough! Your object is writing tests is not to "prove" your program works, it's to try to find out where it doesn't!
Test every "extreme" case you can think of. For example, if you were to write and test a function to sort a list, some things you might consider are: What if the list contains some equal numbers? Do the first and last elements get moved to the correct position? Can you sort a 1-element list without getting an error? How about an empty list?

While you can put as many tests as you like into one test method, you shouldn't. Test methods (like all methods) should be short and single-purpose. If you are testing different aspects of a function, they should be in separate tests.

Here are the rules for writing test methods:

The name of a test method must start with the letters 'test', otherwise it will be ignored. This is so that you can write "helper" methods you can call from your tests, but are not directly called by the test framework.
Every test method must have exactly one parameter, self.
You must put 'self.' in front of every built-in assertion method you call.
The tests must be independent of one another, because they may be run in any order. Do not assume they will be executed in the order they occur in the program.

There are two ways you can call the application methods (the methods being tested):

You can simply import the module (file) with, for example, import arithmetic. To call a method from that module, you must prefix it with the module name, for example, arithmetic.add(2, 2).
As a shorthand, you can say from arithmetic import *. Then you can call the methods directly, as for instance, add(2, 2).

Here are some of the built-in test methods you can call. Each has an optional message parameter, to be printed if the test fails--the brackets, [ ], indicate an optional argument that can be omitted. In most cases, the message is superfluous; don't use it unless you really have something useful to add.

self.assertEqual(expectedResult, actualResult, [message]): Test that the two values are exactly equal.
self.assertNotEqual(firstValue secondValue,[message]): Test that the two values are different, and fail if they are equal.
self.assertAlmostEqual(expectedResult, actualResult, [places,[message]]): Test that the two numeric values are equal, after rounding to places decimal places (default is 7).
self.assertAlmostNotEqual(expectedResult, actualResult,[places,[message]]): Test that the two numeric values are equal, after rounding to places decimal places (default is 7).
self.assertTrue(booleanCondition,[message]): Test that the booleanCondition is true.
self.assertFalse(booleanCondition,[message]): Test that the booleanCondition is false.
self.assertRaises(exception, functionName, parameter, ..., parameter): Test that the function functionName, when called with the given (zero or more) parameters, raises the given exception. Note that, for this assertion method, there is no option for a message parameter.
self.fail([message]): Fail the test. This is most useful if you want to create your own tests, more complex that the above methods support.

Dealing with global variables

If you write a method setUp(self), the unittest framework will call setUp before each and every one of your tests.

Remember that the order in which your test methods run is not guaranteed. If one of your tests changes the values of global variables, those changes may or may not have been performed when some other test is run. In order to test successfully, you need to restore all global variables to their initial state before each test. Do this initialization in the setUp(self) method.

Better yet, don't use global variables. There is almost always a better way to do things.

Test Driven Development (TDD)

Test-driven development takes a simple but counter-intuitive approach: Test the code before you write it. In pseudo-Python, we can describe the process as:

  pick a method that doesn't depend on other, untested methods
  while the method isn't complete:
      write a test for the desired feature
      run all tests and make sure the new one fails
      while any test fails:
          add/fix just enough code to try to pass the tests
      refactor the code to make it cleaner

Except for very simple methods, you can usually write the method in stages. Try to take small steps. Write a test for the simplest case you can think of, then run it. The test should fail--if it doesn't, then either the method already handles that case, or you did something wrong in the test. This is a quick way to make sure you aren't wasting your time adding code to the method.

Next, write just enough code to make the test pass. The temptation may be strong to write more code than necessary, but resist. Don't write any code that you don't already have a test for. Make sure the new code passes, or debug it until it does.

At this point your method may not do everything it needs to, but it will be correct in the part that it does. Now is a good time to think briefly about how to refactor (make cleaner) the code. Is there a variable that could have a better name? Would it be more readable if you rearranged the parts of an if statement? Could you pull out a chunk of the code and make it a function? Did you repeat code? Refactoring is all about making the code better without changing what it does.

How TDD improves the quality of your code

A program that has been thoroughly tested:

Will have many fewer bugs.
Is testable. That is, it is possible to state exactly what the method does; what outputs it will produce for any given inputs.
Is likely to be single-purpose. Methods that do several different things are harder to write and harder to debug, but they are also harder to test. Moreover, a method that does more than one thing cannot be reused when you only want to do a subset of those things.
Is less dependent on context--what all the other methods do. If other methods have to be called to set up conditions for a given method to work, the method is harder to test in isolation, and harder to use. By writing tests, you are ensuring that the method can be used in at least two contexts: the test environment, and the application itself.
Uses fewer (or no) global variables. Global variables, though sometimes necessary, are undesirable for a number of reasons. Methods written to be easy to test are more likely to use parameters rather than global variables.
Clearly separates methods that do computations from those that do input/output. A method that does no input or output can be reused for other purposes. Input and output should be done by methods with that specific purpose. (It is possible to test methods that do input or output, but this requires different techniques.)
Is easier to modify in the future, with much less chance of unexpected consequences. This makes it much safer to refactor the code (restructure it to make it better, without changing what it does).

How TDD improves your programming experience

You will have a clearer understanding of just what each function does (thus avoiding the "Ready, fire! Aim." approach to programming).
You can trust any (tested) functions your new function calls. You don't have to keep going back and checking code you have already written.
If you need to change anything about your program, you won't be afraid to touch it, lest it break in mysterious ways.
You greatly reduce the risk of wasting hours chasing down mysterious bugs. (Oh, if we could only eliminate this entirely!)
You will have more confidence in your program (and in yourself).
Programming becomes more fun. (Writing code is a creative activity; debugging code is just a drag.)