Failure of a UnitTest implicates only one unit. You needn't look elsewhere to learn why it failed
s are not:
- Manually operated.
- Manually operated from memory without a script.
- Automated screen-driver tests that simulate user input (that's AcceptanceTests).
- Interactive. They run "no questions asked".
- Coupled. They run without dependencies except those native to the thing tested (see LargeScaleCppSoftwareDesign).
- Expensive. No new tools, civilians, or engineers required.
- Complicated. UT code is typically straightforward procedural code that simulates an event-driver.
That said, let's build a UnitTest
to test a class called eyeScream.
- Create a project called eyeScreamUT, in a sub-folder of the same name, below the home folder of eyeScream.
- See below "a good folder structure" to view a tree of the folders
- Inside that project, import eyeScream, construct an instance, and call a method. Now recompile.
- If you can't compile due to coupling, fix eyeScream so it's not coupled to whatever it should not be. This doesn't work. You need UnitTests before you start to ReFactor, otherwise you risk breaking the code.
- If you can't compile due to coherency, import whatever eyeScream needs.
- If eyeScream needs external things, like a database, remove the classes that supply this to eyeScream and replace these with local spoofs that return fixed data. Again, this may well be a non-trivial ReFactor, for which you'd need to have a UnitTest before you even started.
- Now add Assertion statements that prove eyeScream constructed, and it did all it was supposed to. If these Assertions pass they are silent (or optionally they state the item tested). But if they fail they halt the program (and optionally raise a Bug Enhancer on the problem statement).
- The last line of the test is an unconditional print statement that says "All tests passed".
- In the root of your project add a Makefile. Yes, a lowly UNIX-style Makefile. We need only just this kind of scripting host, because UnitTests need A> automated compilation dependencies, and B> automatic termination if a compile or test fails. Makefiles give both of those out-of-the-box, so don't go re-inventing any wheels here.
- Inside the Makefile, do not add any of the high-tech Prolog-style twizzlers found in, for example, the Makefiles that compile Linux. Just add a dependency called 'all', and after it write the command-line commands that raise your tool and compile with it. For VC++ that's like this:
msdev Project1.dsp /MAKE "Project1 - Win32 Debug"
msdev eyeScream\eyeScream.dsp /MAKE "eyeScream - Win32 Debug"
Notice I did not sweat over "exporting the MAK file from the project", or calling NMAKE or CL.EXE (or even changing the current folder). I got the same compiling environment as developers get inside the IDE, by calling the IDE itself in command-line mode.
- Inside the Makefile, add a dependency called 'test'. It fairly writes itself:
msdev eyeScream\eyeScreamUT\eyeScreamUT.dsp /MAKE "eyeScreamUT - Win32 Debug"
Observe we have no reason to compile UnitTest
s in Release mode. The hardest bugs to find are often those that only show up in Release mode; why on earth wouldn't you want your UnitTests to catch them?
Also observe the last line runs the test.
As you add more tests you just trot them out under the target 'test'. And you provide a shell short-cut to open a console and run the tests and "pause" after them so the line "All tests passed" can stimulate your pleasure centers. Then you order the crew to stimulate their pleasure centers every half hour or so.
That's all there is to it. We did it without buying more tools, or adding risk to the project, or hiring a "test engineer" (that's AcceptanceTest
s), or learning new languages. We got there by doing things we already know how to do, and obeying the definition of IUT:
- Integrated - written by the same coders, in the same language, on the same platform, as the bugs, with support from the objects to be tested, and tuned to run automatically and with dependencies correctly built.
- Unit - test one "component" (clump of classes) at a time. This permits (per LargeScaleCppSoftwareDesign) units to test in dependency order, so bugs in client objects are known to not arise in un-tested server objects.
- Test - call Assertions all over the place, and JUST CROAK if they fail. Don't log anything, or e-mail anyone, or "count" or "rate" the failures. If ONE FAILURE is enough to fail all tests, and if the test watchers are empowered to fix whatever broke, then don't bother testing anything else after the failure.
Hammering these simple concepts into some programming shops' brains will allow them to not hear "resource-burning overhead and complicated 4th-party tools" if you ask them with feigned innocence "do you UnitTest
I have today fooled around with ways to do the above a bit simpler using Visual C++ 6.0, (I don't know how to use and create makefiles under Windows). I will describe what I did here so that someone can tell me if I am missing the boat, and because it might be of use to others.
I have added another project called Tests to a Demo project so that there now are two projects in the Class View. Tests are the tests that need to be done on the classes in the Demo Project. A failure of a test causes Tests to write an error to a console (printf or cout). And the Tests executable returns a 1. You will see where this goes shortly.
I have made the Demo project Dependent on the Tests project and I have added the executable of the Test project as a pre-link step in the project settings of the Demo Project.
The result of all this is that all the output of the Tests executable is printed in the compile window and that the Demo project won't link if the Tests do not work.
I think this will basically be as functional as the process with the makefiles.
If someone wants to see the VC workspace, please leave a note on my WikiPage
and we can make a plan.
After some hacking I got CppUnit
integrated into the VC++ 6.0 IDE as well by using the above method. It works great and I am definitely converted to UnitTest
ing. I am still working on the powers that be to start implementing the rest of XP.
Projects fully involved with UnitTest
s should work from a completed framework such as TestingFramework
, etc. For Cpp, you can also use the test unit that's built into the BoostLibraries. And if you don't have Boost, you ought to get it anyway.
s permit fast & safe RefactorMercilessly
sessions, and are a natural complement to DesignByContract
s have to be
deterministic? (I.e. You set up a fixture, provoke some canned behaviour and then test the result. Must the behaviour itself be deterministic?) --
Speaking as one who has had to UnitTest
mail systems, you'll really hate yourself if the test only works most of the time. I kept coming up with excuses not to run those particular tests, because they were a) incredibly slow and b) failed about 1/6th the time because of subtle timing weirdness with the mail daemons. -- GrahamHughes
Their results have to be "testable", not "deterministic". If a function produces a pseudo-random number, you can do a few quick tests on a sample of results to see that it has a linear spread, a given range, etc. -- PhlIp
The results have to be reliable.
- If they say pass, they mean pass, and if they say fail, they mean fail. When testing a unit that interfaces with unreliable or unpredictable subsystems, you have to fake those subsystems to give predictability to your UnitTest
. -- WayneConrad
To discuss UnitTest
isolation factors, see StandardDefinitionOfUnitTest
To discuss unreliable subsystems, see ExtremeProgrammingChallengeNineteen
I realize this is probably my naivete with the msdev tool, but I'm confused how to do this one thing: "provide a shell short-cut to open a console and run the tests." What is a "shell short-cut"? I assume that it's some way to invoke nmake via batch file?
Install IE4 before you go to IE5, or get Win98 or Win2000, and you get the Quick Launch bar, typically between the Start button and the Task Bar. Drop the icon there, and it's generally always available.
- I believe it means creating a short-cut on your desktop that invokes said make command. Thus, just double-clicking on the short-cut launches the tests.
If you're sufficiently clever, you can use the Post Build Commands to do this for you. Then, every time you compile you will test.
So now, the question is how do structure the folder structure of the tests. I don't want to interminge code and tests because that distracts me from my code. I also don't want to hide my tests because that slows down the process.
Does this page yet contain a definition? There's a list of things "Integrated UnitTest
s are not", then a very platform-dependent example, and a lot of interesting discussion, but nothing else that looks like a definition.
Is the not-list supposed to be a sufficient definition?
Listen, at a new job, to the following whining...
"We can't do UnitTest
s. We don't have...
The "definition" is the minimum needed to JUST START WRITING THE TESTS. No framework, gurus, 4th-party tools, light methodology, or excess overhead.
- enough resources (people)
- a language that an existing xUnit framework adapts to
- enough time to write them
- a UnitTest guru who knows how to do coverage analysis
- Brand X's latest smarty-pants commercial UnitTest framework
- an XP culture with all the other trappings
- enough time to research all the Wiki pages about UnitTests"
I recently read about PairProgrammingCostsBenefits
, and this
discussion makes me yearn to hear about the UnitTestingCostsBenefits
. -- RobHarwood
More than a year on, I regret asking about the definition. At the time, the "discussion" above was helpful in that it spurred me to learn more about UnitTest
ing, XP and Wiki. Definitions are available at UnitTest
s and StandardDefinitionOfUnitTest
. I still don't think this page's name reflects its contents very well. -- DavidVincent