Functional Test Functional Test
FunctionalTests are programs or scripts configured to test that packages (groups of clusters of classes) meet external requirements and achieve goals, such as performance. They include screen-driving programs that test GUIs from without.
FunctionalTests are a key component of software engineering, and ExtremeProgramming leverages them to permit the OnsiteCustomer to understand and track a project's growth.
Specifying FunctionalTests is the responsibility of the customer, while UnitTests belong to the developers. On some teams the tests are written by the programmers, based on stories from the customer. Other teams use a separate team extracting functional test info from the customer and creating and running the actual tests. In either case, publish the results for all to see.
A useful measurement of ProjectVelocity is the increase in number of functional tests, and the improvement of the scores.
-- RonJeffries
Errors caught during functional testing should first be treated to a SupportingUnitTest, then fixed, then verified with a FunctionalTest.
Benefits of functional tests
AnnAnderson just told me of a project where her brother is a tester. They were supposed to have final code over two weeks ago, and just got it. They now have seven calendar days to "test the system" before shipping. (No, he doesn't work for MicrosoftCorporation.)
In ExtremeProgramming, you build a continually growing collection of FunctionalTests. You run them all the time. You graph them daily, showing total tests, green for correct, red for incorrect, yellow for not yet validated if you have that case (on C3 we do). The graph, on a larger time scale, should look like this over time: FunctionalTestGraph.
More and more tests, more and more correct, leading to great certainty that the program is working.
To make this work, the tests need to run automatically, and in finite time, such as overnight. To make them meaningful, you need a clean SystemIntegration every time you run the tests, such as daily.
We call this process ContinuousIntegrationRelentlessTesting.
-- RonJeffries
It is very important for the customer, that your application does all the things he wants. So I tried several ways to check it. But every time I had to sit in front of the computer and think about the application. FunctionalTests gave me the possibility to check the application automatically. Pressing the start button, going home, and on the next day, I know if I am on the right way or not.
Functional tests and user stories
How closely are FunctionalTests bound to UserStories? Do FunctionalTests accumulate completely independently, or is there some direct correspondence with UserStories? (e.g. "These three FunctionalTests correspond with this UserStory...") Is it the case that there's no need to list which FunctionalTests relate back to a particular UserStory?
FunctionalTests are bound intimately to UserStories. They are the means whereby the users convince themselves that the stories are correctly implemented.
Typically you will have several [to many] tests per story. We have the FTs in groups (suites) relating back to stories: some existing FT suites are: Third Pay Deductions; Field Car; Vacation Payoff; Reduced Hours; Foreign Nationals.
There aren't any public C3 FTs, but they look just like one would think: "An employee earning $1500 per biweek requests 3 periods' vacation pay. He is issued a check for $6000 gross in the first period, then nothing for the next three." This is all put down in some cryptic testing language, or even in code, of course.
Okay, so what have folk done about traceability between the originally agreed UserStories and the ultimate FTs that resulted out of them? Surely the functional tests are a little _more_ structured than the example above? -- ColinDavidMiller
Questions about functional tests
http://www.superlinksoftware.com/cgi-bin/jugwiki.pl?TestingGUIs
and discuss them in
http://groups.yahoo.com/group/java-gui-testing/ -- TomRoche
http://sourceforge.net/projects/abbot
and extending it to work with SWT (it already does AWT and Swing). One then has "straight" JUnit testcases, compatible with all the usual tools (BTW Eclipse has great JUnit support), that can interact with ones UIs. TFD, here I come :-) -- TomRoche
Name discussion: functional vs. acceptance test
I tried "acceptance test" as an alternative name for FunctionalTests for a couple of days and really liked it. It emphasizes the role of the tests over the way they are written, is more approachable for customers, and aligns with existing vocab better. Thoughts? -- KentBeck
I emphasize the role of functional tests as a medium of communication and supplement to stories in the dialog between development and customer. I once thought the name Story a bit dorky but now appreciate its simplicity. I wonder if functional tests might be replaced by the equally simple word, Facts. Someone might say, "I'll understand the story better once I see some facts.", or "how can I help you check these facts?"
If you like this idea you may want to change the name of UnitTests also. Unit test, you will recall, supplement code the same way facts supplement stories. Since UnitTests are made before the code, I suggests we call them Claims. One might hear, "Now that that is done, what will we claim next?" or "I see that the refactoring violates a few claims.".
But, isn't a functional test a claim also? -- MichaelFeathers
Could unit tests be a subset of functional tests? (see RenamingUnitTests)
How about constraints? -- AnonymousDonor
If I ask "what's a unit test?", most people will quickly say something about a programmer testing a smallish chunk of her own code. That's pretty close; the only XP wrinkle that isn't immediately obvious is that the tests come first.
If I ask "what's a functional test?", I doubt I'd very often get an answer that included "end-to-end" or something about being based on user tasks or user stories. It seems like what people call functional tests are not, indeed, end-to-end. Rather they're tests run by an independent tester beating up on some localized feature from the user interface. A set of functional tests might explore the options in a given dialog box, for example.
So, "functional test" probably does not convey the right impression to someone just hearing about XP. "Acceptance test" probably comes closer, because the connotation is that some direct user representative writes them. May have a greater chance of conveying end-to-endness. Maybe "user story tests"? Though that's missing the seemingly essential adjective-noun form.
-- BrianMarick
I believe the nomenclature has indeed been changed to AcceptanceTests. -- PhilGoodwin
The way I see unit tests, is that they are a little testbench for a piece of a program. I often compare this with electronic devices. I once got an old tape recorder (dating from 1967) with a single huge elecric scheme. Devices like this are not built anymore. Modern devices are built up from sub-assemblies, like "pre-amplifier", "motor-control", etc. These are the coarse parts the device is made of. They are separately testable and therefore able to be replaced by something better without the need to adapt the rest (which allows you to produce a complete line of tape recorders with all different specifications, but with a lot of the same parts and also allows you to repair a device by just replacing a part by a new, better one).
The same applies to software. The coarse parts can be tested separately, before they are built into the main program, if they are "Units". That is, modular. So if you want an alternative name for unit tests, I suggest "Part tests" or "module tests". However, "Unit tests" is fine by me. -- WillemBogaerts
See IsqWhyWeDontGetIt for why functional testing won't give you InternalSoftwareQuality.
This page mirrored in ExtremeProgrammingRoadmap as of April 29, 2006