Design Versus Results Evidence

A contentious theme to many debates is about what kind of evidence is accepted or how much it is weighed. It can be roughly partitioned into "how" mechanisms and "why" consequents, either of which may be subject to measurement or precise qualification. Examples:

The How The Why The Evidence (under destruction) Other than possibly brief pro/con descriptions, please put discussion paragraphs below with a heading that corresponds to the bullet point.

(discussion here)

Why did somebody move CodeChangeImpactAnalysis to design? Amount of code that needs changing is a "result", not something you do at design time. Well, I suppose we could split it to test scenario CCIA before release and actual CCIA.

I believe the amount of code that needs changed to get a new capability or result is intrinsic to the design. A great number of design features, such as modularity, plugins, aspect-oriented programming for cross-cutting concerns, domain-specific languages and abstraction and OnceAndOnlyOnce so code-changes don't need to be duplicated, and even KeyLanguageFeatures and DesignPatterns to help do the RightThing - or a more flexible thing, such as VisitorPattern - the first time so you spend less time needing to duplicate code and less time needing to change code later to add new capability, are all aimed at (at least partially) reducing maintenance costs and total costs for code-changes. CCIA is something often considered greatly at design time; in many ways, it's among the reasons programming languages, libraries, and frameworks were designed and now exist.

Besides, CCIA is about stuff in the 'black box', observing properties over the code just as much as looking at safety, correctness, fault tolerance, or anything else. Customers (outside the aviation, medical, NASA, etc. industries) don't overtly care about these things.

Users observing results DO care about time-to-completion (feature requests / throughput - i.e. avg. latency between a request and an update that services it), which is affected by the cost of code-change and the cost of getting the code correct, but whether the reason it takes a long time to fix code is the lack of refactoring tools, the amount of code that needs to be changed, the difficulty in identifying which code needs changed, the difficulty in determining whether a change doesn't break things, or the cost of fixing actual breakage, is all quite irrelevant to them.

However, if you feel like splitting things, and you don't believe the 'latency' measure is sufficient, you can add it to runtime as well. You can also add design correctness vs. result detectable error, design safety vs. result detectable safety failure, etc. I would not object.

It seems we need to come with a clearer definition or description that separates the two. Result metrics focus on "why" while design metrics on "how". Result metrics are what one is trying to achieve for the "customer" when making or choosing design metrics. (The customer may be future code maintainers, not just users.) For example, some suggest that methods should be kept small. This is allegedly to make it easier to grok the code and system in parts. The "why" is grokkability for the developer/maintainer. Design metrics are "how" rules for how to satisfy the "why".

In that case, any developer is always his own customer. The future starts now... no, wait for it, NOW! Darn. Missed it again.

If you seek to divide the evidence along this dimension, I'll help you do so, but calling it 'Design Versus Results' evidence seems misleading. Both vision AND results are "why" issues, and 'design' is always part of 'vision' (you never have a 'vision' without knowing where it fits into some greater 'design'). The 'how' is everything between the two.

A better page-title might be: WhyVersusHowMetrics?

Perhaps our dichotomies are too simplistic of a classification system altogether. This page has gotten away from my original intention. People injected their own personal view of TheOneRightWay? into it, melding together stuff I was trying to keep apart. Thus [1], I'll try again:

PageAnchor: Customer

Try 2: Engineering Metrics Versus Customer Metrics - Engineering metrics are the "how to achieve" side and customer metrics are related to "what to achieve". Customer-side metrics focus on what the "end-user" ultimately wants. For example, a customer wants a car that's cheap to buy, cheap to maintain, uses minimal fuel, has plenty of room, and has decent power [along with a hundred unnamed metrics and qualities, such as being street-legal, not blowing up after a rear-end collision, being reliable, having good visibility, and tons of other things that are invariably desired but rarely named] {Many of these can be rolled up into "safety"}. The engineer may propose various ways to achieve this and even have their own metrics, such as power-to-weight ratio. Whether these match up to user-side metrics/desires or not depends. An engineer may claim that rotary engines are the best way to maximize these goals, but another engineer may have a different opinion. Ultimate user-centric tests such as test-driving and Consumer Reports Magazine will settle such issues.

For software development and maintenance issues, things get a little tricky because the end-user may also be the engineer. If they rotary engine (car) proponent above had to also serve as the builder or auto-mechanic on his/her own designs and not just be a "drawing board" engineer, he/she may fill a similar role as the software developer or maintainer.

It's true sometimes the customer focuses on engineering-side metrics. They may read an article on rotary engines and then decide that they "must have rotary engine" as one of their requirements. Similarly in software, one may convince the customer that "provably correct software" is the way to go because one can use math-like precision to guarantee results. However, if they later learn how expensive and time-consuming it may be (hypothetical only), they may back away from endorsing this one particular way of achieving accuracy. The typical customer of course wants accuracy, but they also want to keep the costs down. If formal methods can't achieve such, then forcing it down their throat is not the way to go. You cannot make it a customer-side-metric just because you like the technology. You have to show the customer how it maximizes *all* their weighted goals in terms of the customers goals themselves. If you want to use your own engineering metrics, you need to convince the customer that your metrics are a good stand-in or ruler for theirs. This is the way it is because they are paying you. If you give the customer what you think they want instead of what they want, you may found yourself with a different customer or none at all.

What I want to emphasize is the separation between the nuts-and-bolts and what the customer actually wants. Somebody who wants to get to the moon only cares whether there's a choice between solid or liquid rocket fuels if and only if it affects their weighed goals (safety, cost, accuracy, time, etc.).


Overall, I can agree with your sentiment as it is reworded here.

I would note, however, that as SoftwareEngineering becomes a real discipline, it becomes OUR responsibility - not that of the customer - to enforce certain parameters. An Architectural Engineer of a skyscraper would never be forgiven intentionally skimping on safety features such as state-of-the-art resistance to Earthquakes and fire simply because the customer, who wasn't planning on living in the building, wished to keep costs down. The same thing will become more and more true of software designers as the field matures. Not all software projects are safety or security or performance or correctness critical, but since these all are emergent properties (requiring pervasive implementation) that are desirable in almost all fields, it will likely become more and more the case that every software product needs these properties simply to be able to integrate with every other software product that might run or interact, even indirectly, with a secure/safe/correct system... which, in accordance with SixDegreesOfKevinBacon, will likely mean every product.

For the moment, we can get away with whining, pouting, throwing our hands in the air about how difficult everything is, etc.. RealProfessionalsGetSued. Real professionals are expected to know more about their field than the customers do. And, as a consequence, real professionals can't blame the customer for everything wrong with the product. But, at the moment, we aren't real professionals; we're just a bunch of hackers with a bad case of DisciplineEnvy, and (until we get our act together) that is how we should be viewed by big business.

We are indeed supposed to know more about our field than the customers, but this is where communications comes in. We need to describe the trade-offs. If they want it cheap or sooner instead of reliable; as things stand, that's their decision. Our responsibility is merely to communicate the trade-offs, not select them. (You can walk out if you want and give the job to somebody else, and I've done that before, but it hurts the wallet.) Product, civil, and building engineers have safety regulations and trial lawyers breathing down their backs as an incentive. It would be nice if we could get customers to sign off on risky decisions. The Ford Pinto and O-ring Shuttle engineers sort of were off the hook because documents found indicated that managers ignored engineer recommendations or concerns. But if people don't die from our software, no such conventions or regulatory body or paper trail will likely exist, at least not in the USA. The problem is that people usually don't die from software. Thus, SafetyGoldPlating (in a good sense) is not valued by customers. Sure, they grumble about a report with wrong numbers, but they also grumble equally about costly software contracts.

You often have the choice to pay up front or pay out the back. Many tradeoffs that exist today don't seem to be consequences of EssentialComplexity; they are consequences of AccidentalComplexity, myopic Simplicity, and other forms of PrematureComplexity. But I understand that you can't easily justify passing off costs to a small-business customer - e.g. there is no way you could quickly tweak an OpenSource RDBMS like MySQL then push the updates back out to the public repository just to have the features you want for a particular business venture. OTOH, that isn't necessarily the case for much bigger business ventures... nor is it the case for the OpenSource persons who are starved for money no matter which choices they make. Problem is, these 'up-front' costs are often huge and beyond easy reach without more capital and time than a typical hobbyist programmer possesses - the low-hanging fruit has already been picked, and the more software designed on any modern platform, the further the next platform would need to catch up just to match what has already been done. Inertia is a b*tch. But we'll get past it. Somehow. ComputerScience as a field is really only fifty years young.

And, besides, more people will start to "die from software" as more systems become reliant upon it, and thus we won't have that - what did you call it? - "problem" anymore. ^_^ As a note, military now uses software quite heavily for targeting, communications, etc. Medical now uses software quite heavily to track prescriptions, remote operations, etc. Aviation now uses software quite heavily for navigation, autopilot, collision detection and avoidance. Automobiles now use software more heavily for fuel-injection, maintenance, braking, wheel-power-distributions, etc. Traffic now uses software quite heavily for lights and control. Firefighters, Police, the list goes on. Fortunately for software, it's just one component so deaths can rarely be attributed solely to it. It would be nice, though, if people collected enough stats to add software-failure to the death counter (

Yes, and such software is usually much more expensive per feature. In many domains, management does not want to pay the "medical" rate for stuff. They want it cheap and nimble.

I believe it is more expensive per feature only because we haven't paid the big up-front costs to make it cheap per-feature for certain classes of features that are commonly desired from one project to another. As I was saying: you pay up front or you pay out the back.

Heavily-engineered systems tend to stay heavily-engineered and remain maintenance-heavy. Maybe you know how to do such systems right; but if so, that is not the norm.

Will you please define "heavily-engineered"? And would I be right to infer that you are assuming that obtaining such features as safety and provable correctness require "heavy engineering" and are necessarily maintenance-heavy? I find myself very skeptical of that pair of assumptions - I expect at least one of them is false no matter how you choose to define "heavily engineered". I do agree that codebase inertia is a rather fundamental property - a bit like 'mass' in physics - so that if something qualifies as "heavily engineered" it is unlikely to change.

It is an observational anecdote only. The counter-evidence is probably also anecdotal. "Heavily engineered" here are usually projects that involve critical line-of-business transactions (usually financial) or human safety. They have teams of at least 3 developers, formal version tracking, QA staff, and formal logged non-originator code reviews. They usually used compiled languages and strong typing. ("Heavily-engineered" could also mean lots of bell-and-whistle features in some contexts, but not meant that way here.)

This all very much reminds me of the dichotomy between assets and liabilities in financial accounting. For example, DesignByContract would be an asset, but for the project to be "balanced", its use be balanced by some kind of liability. In other words, for every piece of design, there must be a balancing, measurable result. --SamuelFalvo?

The equation, IIRC, is Assets = Liabilities + Owners Equity, or Owners Equity = Assets - Liabilities. There isn't any reason that Owners Equity couldn't be very, very positive. I'll admit I don't much respect the field of accounting and its voodoo arts for deciding values or costs of labor and services and everything else - any profession with competing models for hedging ought to do some serious reflection! But if you were going to place it, access to DesignByContract would be an asset - the liability being restricted choice of programming language (which might cost very little or a great deal depending on context). However, use of DesignByContract would be a liability - a statement of properties you are required to meet (and continue meeting), which is some metaphorical equivalent to services and such. I.e. Using DesignByContract voluntarily restricts your own behavior, and is thus a liability. The balancing asset would be the guarantee that those properties are implemented correctly in accordance with a code-proof - i.e. a guarantee of correctness, resistance to regression, etc. (Not that one can easily place dollar values on such restrictions or guarantees... but you can always hedge!)

I think you're confusing basic accounting with CreativeEconomics?. Economics, as a whole is the study of resource allocation. CreativeEconomics? is what happens when economists attempt to game the system for their benefit, trying to pass off their crimes against humanity as valid economics. At any rate, assuming no code ownership (which I find is increasingly prevalent in organizations today), owners equity will always be zero, so in my opinion, my original analogy still stands.

That being said, I like your precise example much better than mine! --SamuelFalvo?

There is always code-ownership, just not always exclusive code-ownership. Public domain = owned by everyone who manages to get themselves a copy. How well does accounting deal with open source?

{Easily. The asset value of OpenSource source code is zero. The (tangible & intangible) asset values of producing OpenSource software are derived from expertise, support or consulting contracts, associated publications, promotion, goodwill (in the classic sense), etc. The asset values of using OpenSource software derive from modifiability, reduced licensing costs, and whatever value using the software is to the organisation.}
What is 'correct' for a video-game?

What's wrong with "It plays the game the developers intended it to play."?

First, it doesn't touch on any service goals, such as entertaining those who play it. Second, which developers, among a group, get to answer that question?

Which question? The "Is the video game correct?" question or the "What is the game we are intending to build?"? The former would be answered the same way it would in other domains. They would pick a level formality (ranging from not answering to formal proofs) and do that. The latter requires an answer in any case, and would be done as it is now.

Correctness doesn't cover every goal for software in other domains, why require it to do so in gaming? I wouldn't consider the entertainment value of a game as part of its correctness. I don't find tic-tac-toe particularly entertaining anymore, but I would still consider software that plays it to be correct if it actually plays tic-tac-toe.

Correctness for a well-defined game, like Checkers or Chess or Tic-Tac-Toe, would be easy to determine. A Checkers game is correct if it puts the initial pieces in the right spots, allows you to move them along diagonals, allows you to leap other pieces, takes the pieces you leap off the board, kings your piece when it touches the end, otherwise supports and enforces rules, properly determines the winner and terminates the game when all is done, etc. The 'service' here is the automated provision of a particular game.

Correctness doesn't touch on reliability or maintainability or money acquired or any of those other features. However, it does relate to services and features - it is defined, after all, in terms of meeting service requirements. So, except where there are clear service requirements, there is no clear definition for 'correct'. Any software product that provides services in addition to its service-requirements is simply offering more than is required to be 'correct'.

The problem with most video-games is they possess no clear idea of the service they intend to provide. What is the service provided by 'The Sims'? How do you know when Harvest Moon is 'correct'? You are correct that if someone can (with authority) answer this question in a manner that lends itself to measurement or objective qualification, we could determine correctness. If it goes unanswered, however, we can only presume that the intended service is to impart some measure of entertainment upon players of a target audience, and, perhaps, to tell a story. For edutainment, one might also attempt to teach a skill. And, perhaps, creating then servicing an addiction by manufacturing then appealing to a lust for cheap rewards is an aim for certain games (e.g. World of Warcraft or Ecchi).

In any case, video-games are just one example of a field in which there is often no clear definition of 'correct'. I invite you to properly contest it in the general case, though, as it would be really neat to find a clear understanding of 'correct' for video-games and other things with vague requirements - something you could post in a gaming journal, perhaps.

I would define it just as was done for other domains, "a state in which software performs as required when operating within expected conditions". You appear to be including the step of finding out what those requirements are as part of this. While I don't want to downplay how important that step is, I don't include that as part of correctness. It's a prerequisite to be sure, but a separate issue.

Perhaps we're just viewing these from slightly different points of view. You're applying the definition of 'correct' as though it were universal and takes two arguments (requirements and product) and returns a logic-value (from MultiValuedLogic). I.e. correct :: Requirements -> Product -> Logic Value. I'm using the word 'correct' in a particular context, in this case 'video-games', implying that the product and requirements are available in the context. In this case, the Logic Value associated with 'correct' is simply not defined, or perhaps (a possibly better word) not resolved, until both the requirements and the product are resolved. As such, I say that 'correct' isn't well understood for video-games. I can't disagree with your view, but but I can say we have likely been talking past one another a bit.

Yes, the original definition I provided for 'correct' doesn't need to change at all, but it is still important to recognize that it isn't necessarily meaningful in a given context, and communicating this was my original intent.

RE: Understanding the reasons for X is not the same as understanding X... (supremely OffTopic... to be moved or deleted later)

[Well, back in the day, I got a ninety nine percent grade in accounting in high school and I hated it, so I dropped out of that class the next year instead of continuing. Understanding Monads is wonderful - but proving that it actually works with some supportive multi processor benchmarks/demos such as how Google has offered us information on how Sawzall works - would be far better than simply rambling on about how Monads does this and does that. In theory, and on some web page, Standard Pascal could be the greatest language in the world - until it is tested in the real world by some pragmatists like Pike, Kernighan, Richie, Thompson.. these people who had real world experience implementing compilers (and pages such as FutureOfProgrammingLanguages neither have real world pragmatic public tests yet - so be careful about what we predict on such pages. We might need a team as large as GNAT Ada to implement any of it. So RightBackAtYou - understanding programming languages is not about just having idealistic wiki pages about them.]

The above wasn't an attack on you, though you seem intent on making it into one. And I agree - an idealistic wiki page about programming languages (such as FutureOfProgrammingLanguages) does not mean an understanding of programming languages. OTOH, a bunch of pragmatic wiki pages about how to implement and analyze various aspects of languages and interactions - ways that have often been tested and proven already and exist in various languages around this world - do, if combined into one person, form the basis for a claim of understanding programming languages. And designing and implementing lots of LittleLanguages and compilers/interpreters helps, too.

[Already tested/proven and already exist in various languages - sounds a lot like 'reinventing and reusing old ideas and tweaking them for the future.]

That doesn't seem so relevant to this context. And tweaking is fine... it's the other half that is implied to not be so good: taking an old idea and thinking it your own. People who twitter "I am great! Look how high I am!" like little birds upon the shoulders of giants are amusingly arrogant.

[Please provide some exact URL's and places which I will change and update for you (I aim to please), to be less arrogant sounding. Also, if the URL's are clearly marked with humor, then don't bother providing that as a URL since that's clearly sarcasm and humor.]

Rather than start a fight here and direct you back through silly arguments involving wheels, tires, QompReactions, CapArrays, and the pages that started them - all without any real purpose or ultimate benefit to anyone - how about you simply tone down arrogance a notch or two on a more permanent basis? I mean, you can still be arrogant (LazinessImpatienceHubris), but you don't need to inflict it on everyone else.

RightBackAtYou - TopMind and others probably think you are very arrogant (self reflect).

As far as hypocrisy goes, I've made no claims of old ideas as my own... in fact, I search and find most ideas I think were original have already been done or are shortly put into progress by some company or another - sometimes the idea is just 'ripe', I guess (YouCantLearnSomethingUntilYouAlreadyAlmostKnowIt applied globally, much like various mathematical discoveries performed independently within years by different people across the world). It saddens me to know I could still get a patent many of these old ideas in a jiffy, just like Microsoft does. In any case, I suspect I've rarely come up with a fully original idea in my life. I just happen to think that coming up with ideas is not particularly valuable - it's making them work that is ultimately of consequence - only 1% inspiration, after all.

And TopMind has actually been quite reasonable in his more recent posts; I hope he continues to be so, because if he keeps that up, and holds to the promise he made in the BookStop page, and avoids reversing BurdenOfProof for any claims he makes, he'll earn my respect in a jiffy, perhaps even joining those whom I admire for their greater ability to maintain some semblance of intellectual argument in a heated discussion (such as DaveVoorhis and GunnarZarncke).

I prefer people who are intelligent, regardless of their arrogance - by the way. If one is arrogant that is okay. Most intelligent people, are fed up with the world and must be at least a little bit arrogant in order for their ideas to follow through (Dijkstra was claimed an Old Grumpy Critical arrogant man - so?). In the Fraiser Crane show, there was a quote "I prefer being called pompous and arrogant over..." which I'll have to look up. Your claim above is also very arrogant and hypocritical - since you are arrogantly criticizing me. Let's not start the NotNiceEnough page again.

It is very arrogant to claim that CapArray has no purpose.. and QompReactions was not created by me - but by arrogant critical people who hated the idea of Qomp. If they were humble, they would have said oh that is nice, passing by now.. Instead, arrogant people fired flames and created the QompReactions page (which, by the way, was very good - I don't wish for people to remain quiet - I prefer they speak up).

CapArray is a perfect example of an old idea that a young man thinks his own.

I clearly state that it has probably already been invented, but that I'd label the pattern anyway with a precise term called CapArray. Vectors and such vague terminology used in Java don't explain what the pattern or algorithm is. Might as well call them "triangles" or "funny arrays" if we're going to call them cute names like vectors (oh, how specific). Then you've got the scripting folks who use arrays in Ruby and such languages which don't label the algorithm as anything - instead they just call them associative arrays (without telling us - what kind of associative array? Assuming that we need the same algorithm, a hash list, for every program - which can clog a server up - since hashlists are overkill for many situations.) Then we've got arrogant people claiming that memalloc and realloc is good enough for me, no one needs this CapArray thing.. If CapArray is a duplicate page, then please point us to the pattern on this wiki which explains it - and we will merge it instead of moaning and complaining here on this page about how CapArray is so useless to him.

And then you said we're all jealous for not labeling it first. That was very amusingly arrogant. As is your entire rant here. Cute. And CapArray is pointless for reasons that have everything to do with runtime efficiency relative to far simpler solutions.

Far simpler solutions, such as using dangerous freehand Cee functions like memalloc and realloc.. Yes that often saves the day.

Hmmm? Is CapArray somehow avoiding any memalloc (or equivalent)? And, being a big fan of safety and correctness and concurrent programming, I'd never suggest use of freehand memalloc and realloc. The algorithm found in some Dobbs journal still isn't clearly patterned anywhere on the net (well please provide URL's if so, and we'll merge the info into the CapArray page if so). By the way: "Design versus Results" is about the vaguest, most useless, meaningless wiki page. It's possible it should be deleted too. As if people don't know the difference between results and design - how patronizing.

Not discussed anywhere on the net? Why don't you start here and follow the links.

Yes, how patronizing. We all know what a dynamic array is. Freepascal and Delphi have had them built in for years now. That is what triggers one to avoid using built in dynamic arrays - since a dynamic array algorithm can vary. Apparently, you still don't understand - nor will you probably ever - I'd suggest you'd just use realloc, setlength, and hash tables. Or use a hashlist - no one is stopping anyone from doing that. Or just use a dynamic array. Whatever a dynamic array is. Well it isn't static - so what's the algorithm behind the dynamic array (rhetorical question).

If you had bothered reading, the algorithm behind the dynamic array growth was on that page, in the section on 'geometric expansion and amortized costs': grow bigger => resize by multiplier (but it isn't clearly labelled as some pattern we can easily coin, which is what this wiki is all about (so... "geometric expansion" doesn't count, I suppose.)), copy from older array, properly discard older copy of array. It works pretty darn well until concurrency or versioning or backtracking or transactions are thrown in - and no worse than the CapArray under those or other circumstances. So, maybe I don't understand the value you see in it; maybe I never will. It is a replacement for regular arrays when you need to access a[1], a[2], a[3]. It's not so great for inserts - but many of my programs do not require random inserts (and if they do, they are sometimes very rare - and it wouldn't matter - this is where profiling comes in handy). Wikipedia explains a whole bunch of dynamic arrays without ever clearly labeling the cap dynamic array pattern. A CapArray is a form of dynamic array and is just one choice we have. Instead of calling it a CapArray, people resort to long sentences and phrases like "this array here uses a technique that has some initial length set and then later it automatically resizes unlike a typical dynamic array". Why not just call it precisely by its pattern instead of reinventing/re-explaining it each time - is my point. That is what this wiki is all about - labeling common patterns.

Some who discover the CapArray don't even quite understand the CapArray and just see it as some basic dynamic array, too. i.e. maybe it needs even clearer explaining of how and when I've used it and found it handy. Sadly, I see hundreds of freepascal and delphi programmers (and Cee programmers) reinventing the CapArray in their programs using SetLength? and ReAlloc? - which causes bugs because humans don't always index the array properly, don't allocate enough memory, forget to free memory, etc etc. That is why I labeled the pattern and wrote about the pattern. When I first brought up the CapArray and CapString on mailing lists several years ago - the idea was rejected, ridiculed. A few days later it was accepted as self evident and obvious, and useful. This clearly maps to a quotation from A. Schopenhauer.

See also: TheoreticalRigorCantReplaceEmpiricalRigor


View edit of July 8, 2010 or FindPage with title or text search