Now called, in XP: AcceptanceTest
s are programs or scripts configured to test that packages (groups of clusters of classes) meet external requirements and achieve goals, such as performance. They include screen-driving programs that test GUIs from without.
s are a key component of software engineering, and ExtremeProgramming
leverages them to permit the OnsiteCustomer
to understand and track a project's growth.
functional test reads input from real system input files, pays one or more employees, exports results to real result files, then reads the result files back in and compares individual results to expected values. C3 has over 400 FunctionalTest
s is the responsibility of the customer, while UnitTest
s belong to the developers. On some teams the tests are written by the programmers, based on stories from the customer. Other teams use a separate team extracting functional test info from the customer and creating and running the actual tests. In either case, publish the results for all to see.
A useful measurement of ProjectVelocity
is the increase in number of functional tests, and the improvement of the scores.
Errors caught during functional testing should first be treated to a SupportingUnitTest
, then fixed, then verified with a FunctionalTest
Benefits of functional tests
just told me of a project where her brother is a tester. They were supposed to have final code over two weeks ago, and just got it. They now have seven calendar days to "test the system" before shipping. (No, he doesn't work for MicrosoftCorporation
, you build a continually growing collection of FunctionalTest
s. You run them all the time. You graph them daily
, showing total tests, green for correct, red for incorrect, yellow for not yet validated if you have that case (on C3 we do). The graph, on a larger time scale, should look like this over time: FunctionalTestGraph
More and more tests, more and more correct, leading to great certainty that the program is working.
To make this work, the tests need to run automatically, and in finite time, such as overnight. To make them meaningful, you need a clean SystemIntegration
every time you run the tests, such as daily.
We call this process ContinuousIntegrationRelentlessTesting
It is very important for the customer, that your application does all the things he wants.
So I tried several ways to check it. But every time I had to sit in front of the computer and think about the application.
s gave me the possibility to check the application automatically. Pressing the start button, going home, and on the next day, I know if I am on the right way or not.
Functional tests and user stories
How closely are FunctionalTest
s bound to UserStories
? Do FunctionalTest
s accumulate completely independently, or is there some direct correspondence with UserStories
? (e.g. "These three FunctionalTest
s correspond with this UserStory
...") Is it the case that there's no need to list which FunctionalTest
s relate back to a particular UserStory
s are bound intimately to UserStories
. They are the means whereby the users convince themselves that the stories are correctly implemented.
Typically you will have several [to many] tests per story. We have the FTs in groups (suites) relating back to stories: some existing FT suites are: Third Pay Deductions; Field Car; Vacation Payoff; Reduced Hours; Foreign Nationals.
There aren't any public C3 FTs, but they look just like one would think: "An employee earning $1500 per biweek requests 3 periods' vacation pay. He is issued a check for $6000 gross in the first period, then nothing for the next three." This is all put down in some cryptic testing language, or even in code, of course.
Okay, so what have folk done about traceability between the originally agreed UserStories
and the ultimate FTs that resulted out of them? Surely the functional tests are a little _more_ structured than the example above? -- ColinDavidMiller
Questions about functional tests
- How do people do functional tests of GUI based systems? Presumably with some sort of automated tool. Does anyone have experience with such things?
Yes, in Java. There are a number of Java GUI testing tools: see
and discuss them in
Setting up a Rails project for outside-in development you can use Cucumber, RSpec-rails, and Webrat
> sudo gem install cucumber rspec-rails webrat
You will then have to run the following generators that ship with the gems:
You mean any applicability to Java GUI FVT? (FVT == IBMish for "functional verification testing." I dunno why we always say "verification" -- what good is a test that doesn't "verify"?) I believe so: I
m not familiar with TestingFramework
, but it sounds like xUnit plus TestResult
. Several of the Java GUI testing tools listed above just enable JUnit testcases to "reach out and touch" (and interrogate) a GUI. My group at IBM is working with Abbot
and extending it to work with JavaSwt
(it already does JavaAwt
). One then has "straight" JUnit testcases, compatible with all the usual tools (BTW Eclipse has great JUnit support), that can interact with one
s UIs. TFD, here I come :-) -- TomRoche
- I'd like to know more about these FunctionalTests - are there public examples that interested people can read?
Name discussion: functional vs. acceptance test
I tried "acceptance test" as an alternative name for FunctionalTest
s for a couple of days and really liked it. It emphasizes the role of the tests over the way they are written, is more approachable for customers, and aligns with existing vocab better. Thoughts? -- KentBeck
I emphasize the role of functional tests as a medium of communication and supplement to stories in the dialog between development and customer. I once thought the name Story a bit dorky but now appreciate its simplicity. I wonder if functional tests might be replaced by the equally simple word, Facts. Someone might say, "I'll understand the story better once I see some facts
.", or "how can I help you check these facts
If you like this idea you may want to change the name of UnitTest
s also. Unit test, you will recall, supplement code the same way facts supplement stories. Since UnitTest
s are made before the code, I suggests we call them Claims. One might hear, "Now that that is done, what will we claim
next?" or "I see that the refactoring violates a few claims
But, isn't a functional test a claim also? -- MichaelFeathers
Could unit tests be a subset of functional tests? (see RenamingUnitTests)
How about constraints?
If I ask "what's a unit test?", most people will quickly say something about a programmer testing a smallish chunk of her own code. That's pretty close; the only XP wrinkle that isn't immediately obvious is that the tests come first.
If I ask "what's a functional test?", I doubt I'd very often get an answer that included "end-to-end" or something about being based on user tasks or user stories. It seems like what people call functional tests are not, indeed, end-to-end. Rather they're tests run by an independent tester beating up on some localized feature from the user interface. A set of functional tests might explore the options in a given dialog box, for example.
So, "functional test" probably does not convey the right impression to someone just hearing about XP. "Acceptance test" probably comes closer, because the connotation is that some direct user representative writes them. May have a greater chance of conveying end-to-endness. Maybe "user story tests"? Though that's missing the seemingly essential adjective-noun form.
I believe the nomenclature has indeed been changed to AcceptanceTest
s. -- PhilGoodwin
The way I see unit tests, is that they are a little testbench for a piece of a program. I often compare this with electronic devices. I once got an old tape recorder (dating from 1967) with a single huge elecric scheme. Devices like this are not built anymore. Modern devices are built up from sub-assemblies, like "pre-amplifier", "motor-control", etc. These are the coarse parts the device is made of. They are separately testable and therefore able to be replaced by something better without the need to adapt the rest (which allows you to produce a complete line of tape recorders with all different specifications, but with a lot of the same parts and also allows you to repair a device by just replacing a part by a new, better one).
The same applies to software. The coarse parts can be tested separately, before they are built into the main program, if they are "Units". That is, modular. So if you want an alternative name for unit tests, I suggest "Part tests" or "module tests". However, "Unit tests" is fine by me.