Ian Lance Taylor has a nice post about DejaGNU, the test harness used for GCC and GDB. One of his main points—the one I found the most interesting—is that it was a mistake to try and test the actual user interaction with GDB, with a test script that essentially pretends to be a human typing commands and reading GDB’s responses. What should have been done instead is to create a programmatic interface. This would have been much easier to write test scripts for; in particular, parsing the replies would have been much easier and much less fragile. (Naturally, it’d still be prudent to have some tests for the user interaction, but for testing the bulk of GDB’s functionality a programmatic interface would be superior.)
This perfectly matches my experience writing tests for Simics, where we have both programmatic and interactive interfaces; testing the former is such a pleasure compared to testing the latter.
I guess the leasson to learn here is that testing gets so much easier if you have a programmatic interface to write your tests against that it’s even worth it to create such an interface if it didn’t already exist, just to be able to use it in tests. That might involve serious redesigning, but testing is that important!