I've read around the site and one thing becomes clear to me.... (I cleaned this up a little to be easier to read.)
is not considered mature because we don't have the equivalent of standardized:
- nuts (access method?)
- bolts (methods?)
- boards (classes?)
- washers (interfaces?)
So I am Proposing a new topic line: What is our toolbox?
- We are talking about language independent tools (waste of time to discuss language specifics)
- We are looking for description that can be easily ported to common languages
One thing at least a couple of us agree on is this seems to be a major part of the toolbox.
- Principles of:
- Artifacts of:
I do add the following note: I do understand that artifacts are language dependent, but we do not specify which language the artifacts belong to. -- WyattMatthews
I think there's a certain danger in over-using mechanical engineering as a metaphor for software engineering. SoftwareEngineering isn't mechanical engineering, it isn't creative writing, it isn't math. It's a thing unto itself, SoftwareEngineering is, and I don't think it deserves to be deprecated by unnecessary analogy. That said, aren't our "nuts and bolts" algorithms, data structures, and software patterns?
I would hope so. It is my intent to clear that up not only in my mind, but everyone else's as well.
Is this a duplicate of SoftwareEngineeringBodyOfKnowledge effort ? That one is all but failed for practical purposes, but still has a lot more than this page. I don't think wiki is a good place to start such enterprises. -- CostinCozianu
Costin, it is not intentionally a duplication. I will re-read both pages and make adjustments if needed. -- WM
This page seems to be an attempt to set some lower threshold on what software engineers "engineer". The assumption is that for any given category of engineer there is some scale of matter that separates raw material from finished product. The title suggest an engineer who designs structures using nuts and bolts, but would never design a new nut or bolt. The implication is that every possible type of nut and bolt is available to that category of engineer.
One problem with this metaphor is that engineers do
create new kinds of nuts and bolts when existing nuts and bolts don't fit a new use or new environment. They may not necessarily be in the same category as the engineer who uses the new nuts and bolts, but novel situations demand novel materials.
If we look around at current software technology, it seems that we haven't created every possible type of any building block. We haven't even created the theory that would identify the possible parameters for such blocks or establish their scale.
- In response, I believe in part one of the things I am trying to distinguish is the engineering role from other roles. True, an engineer in any profession uses the set of available tools and building blocks; but it is their counterpart, the researcher, whose job is to discover and refine these. And, before anything else, an engineer can be a researcher as well. -- WM
The first issue that will need to be addressed is: What are our nuts and bolts as Engineers? (contrast with, what are the mediums we use as Artists?) Do they reduce to the Pattern system we are working with, or are they just implementations of the DTSTTCPW?
Secondly, how should we define these meta descriptions of these tools? Again, are they just Patterns that we all already employ?
The problem is that almost every EverythingIsa
item can potentially be made into a standard building block. There are just too many ways to achieve the same results, leading us back to HolyWar
s again. It is a problem of narrowing down the choices, not lack of.
Then in an effort to avoid the HolyWar
Syndrome, let's conceptually discuss how to narrow down the choices. Is there a way? Is it applying the DTSTTCPW and OAOO to everything even a pattern?
I took a stab at responding to this, and then realized I don't quite understand what is being asked. Clarification, please?
It has been pointed out that there are too many candidates for basic building blocks, starting with things such as EverythingIsa
items. The question is:
- What should qualify a technique or pattern as the basic piece of code equivalent (back to the inadequate engineering metaphor) to a nut, bolt, or whatever?
- Can we list qualities that would help identify what a real SoftwareDesigner or SoftwareEngineer would need as a starting point for any application domain?
Feel free to add more potential questions.
Sorry, you'll have to forgive my density here. I still find the above too amorphous to work with. I find it difficult - and maybe this is my own lack of imagination or understanding, I admit - to interpret anything above this paragraph except in terms of algorithms, data structures, and patterns. While it may be possible to further subdivide algorithms and data structures into more primitive and/or abstract notions of reference, transformation, state and so forth, I am not sure this is useful at the level of software engineering except in terms of understanding a given algorithm, data structure, and/or pattern. There are also constructs that are nearly universal, such as identifiers, but I am not sure their near ubiquity is sufficient - only absolute ubiquity will do. Therefore, algorithms, data structures, and patterns are the nuts and bolts of our profession. They are universal and fundamental, as they are applicable to and required by every language and software system, whether implemented explicitly by the programmer or available implicitly in the language or software system. Language features or constructs, programming paradigms, development approaches, and various pre-implemented algorithms and data structures - the attributes of the aforementioned EverythingIsa items - are largely not the nuts and bolts of software engineering. At best, they may be - if the analogy can be stretched this far - ways in which we arrange the nuts and bolts, ways to understand arrangements of nuts and bolts, or ways to assemble nuts and bolts, but because they are (by definition) not universal except from the point of view of a given EverythingIsa item proponent, they cannot be considered basic building blocks. They are structures built from basic building blocks, or tools used to arrange basic building blocks. Only algorithms, data structures, and patterns are inarguably universal. Without these, there is no software to engineer. However, that is not generally true of items on the EverythingIsa list.
Perhaps that's all we need, but I worry that it is over-simplification. I am looking for what is really at the heart of things after all. I guess what I am asking is: "What do we consider core to our discipline?" and "What do we expect to be already present when we sit down to design or engineer a software project?"
Just algorithms, data structures, and patterns?
- Computer systems flip bits. Everything else is about flipping bits more effectively; either a better way of flipping bits or a better way of relating the flipping bits to the system's context. Perhaps the 1s and 0s that are fundamental to the software engineer's universe are analogs of the molecules that are fundamental to the mechanical engineer's, whilst the interpretations of the bits (data, code...) are analogs of the forces that act on molecules (gravity, friction...). The mechanical engineer's nuts and bolts are composite (molecular) mechanical-engineering structures with no independent utility but a range of mechanical engineering applications. A software engineer's composite (bit) software engineering structures with no independent utility but a range of software engineering applications are assemblers, interpreters and compilers: software engineering artifacts in their own right, but useful only for engineering other software.
Again, I know I am being a little dense or repetitive, but: Is that all there is? We now have listed the following:
- Principles of:
- Artifacts of:
Is there nothing else? No standard implementations of the principles in relation to the various artifacts?
No. Are the Principles and the Artifacts not enough? A developer who understands the Principles and can select an appropriate Principle for the task at hand via a rational, reasoned process - i.e., can appropriately apply experience, knowledge, and expertise - and can effectively use the Artifacts to produce acceptably reliable software on time and within budget, is a good Software Engineer.
What would "standard implementations" of the Principles in relation to the various Artifacts be? A library of "reference standard" source code? Due to the diversity of problem spaces, options, and individual requirements, I doubt it would be possible to come up with even a single agreed reference implementation of, for example, a disk-based B+TREE in ANSI Standard C, let alone come up with a reference collection of all Principles in relation to an agreed upon (good luck!) set of Artifacts. That would be a bit like coming up with a "standard implementation" of an electric motor. The closest we have might (emphasis on "might") be TheArtOfComputerProgramming.
I'll give you that currently there is no way of coming up with a single agreed upon reference at this time due to the number of potential problem spaces.
What I think we will need to continue this discussion is a SoftwareEngineer
(let's call him 'SE') who has a specialty in a particular ProblemSpace
(let's call it 'X').
Suppose SE asks others who specialize in X, "Why have we not been able to yet come up with an agreed upon reference that factors out as many tailorable requirements as possible?"
My gut reaction is that there is no single real-world problem space small enough to support a realistic complete enumeration of tailorable requirements, while still being a significant enough problem space to be worth bothering.
- Tell that to Oracle, SAP et al. Common business problems like accounting and inventory management, customer relationship management (CRM) and management information have off-the-shelf tailorable solutions earning major corporations megabucks. Major business sectors like banking, insurance and civil engineering have enterprise solutions available. Microsoft and IBM back component-based architectures precisely because companies want to buy in proven solutions to the diverse problems they face, and get them all working together. That's what makes software engineering a little different: successful generic solutions tend to become marketable software. Low-level generic solutions merge into languages and development environments; specific solutions and partial abstractions thereof become packaged product. Software engineering persists in the relatively imperfect and under-explored middle ground, where suitable packaged solutions aren't available or where improved quality (usability, performance...) are required.
- Yes, but the aforementioned tailorable solutions are neither standards nor reference implementations. A CRM package from Oracle/Peoplesoft is not the definitive CRM package by which all other CRM packages are validated and checked, it's merely a product. A "proven solution" and a reference standard are very different things. In fact software engineers are kept busy because such packages require customization and adaptation to individual problems. Perhaps I should have been clearer above, or at least emphasized the "complete enumeration of tailorable requirements," by which I meant to imply that a standard would only ever require selection of attributes, rather than development of new functionality. -- DaveVoorhis
- "Customization and adaptation to individual problems" sounds a lot like "selection of attributes, rather than development of new functionality" to me! I would agree that they are "neither standards nor reference implementations" ...yet... but nor are they "merely a product", any more than Windows, Java and UML are. There is nothing (in theory) to stop a SoftwareEngineer developing their own operating system, programming language or specification standard, just as there is nothing to stop a mechanical engineer from specifying non-standard nuts or bolts. It would raise a few eyebrows, though. We are now at the level of maturity where no self-respecting CFO would pay for a bespoke accounting system. Does anyone bespeak CRMs these days? And is there a reference standard for internal combustion engines?
- There is no reference standard for internal combustion engines, and the millions of Chevy 350 engines still pounding the roads doesn't make them into some de facto reference standard for internal combustion engines, either. They're merely a popular product. Windows is merely a popular collection of products. Java (the language) is a specification for a collection of products. UML is a diagramming methodology. Customization and adaptation to individual problems may, in some cases, consist of selection of attributes. However, the new Peoplesoft implementation where I work apparently required a year's worth of customization by a roomful of programmers. Selection of attributes surely happened in the midst of it, but so did a fair amount of software development to integrate it into existing information systems and meet specific requirements that apparently weren't met by default. -- DaveVoorhis
- In this section in particular, and elsewhere, I'm beginning to smell something like an agenda on the part of the individual who started this page. Or, if not that, at least a point to be made above and beyond what I assumed to be the academic goal here of identifying the fundamental elements of software engineering. Would that be correct, or have I misinterpreted your intent? -- DaveVoorhis
- No need to be suspicious, Dave. I am not the person who started this page, and my contributions here are consistently in single bulleted paragraphs. From the start, I have objected to the thesis that SoftwareEngineering is not mature because it lacks equivalents of nuts and bolts. I have suggested that there are numerous software artifacts like compilers, operating systems and IDEs that are equivalent (at least) to nuts and bolts, but apparently that wasn't enough. So I further suggested that there are many less fundamental artifacts that purport to solve some generic classes of business problems. Your objection to this was that these are not "standard" but "merely a product". I countered this by claiming that products can achieve a status of quasi-standard and attempted to dismiss the original objection as a red herring, since mechanical engineering doesn't rely on standards for all significant engineering products. Your experience suggests we've still some way to go in SoftwareEngineering, and I don't disagree. But I don't think that makes ours an immature discipline. And if I may allow myself one final quibble, UML is not a diagramming methodology but a (self-styled) "methodology-independent" modeling specification (and not one that I personally would endorse).
- My apologies, then, for a bit of TinFoilHat paranoia - which I attribute entirely to an excess of blood in my caffeine stream - and thanks for the clarification. -- DaveVoorhis
It was not my intent to forward any agenda. I started it hoping to clear up my own confusions about what the readers of this wiki, and perhaps myself, see as necessary tools for the job of a SoftwareEngineer
I also ask, should SE have a standard reference for the problem space of 'X'?
It strikes me that attempting to define such a standard reference is a little like trying to define a reference piston for a universal internal combustion engine. (How could there be a universal internal combustion engine??? Ah hah! See?) We certainly recognise pistons are pretty much ubiquitous in internal combustion engines (rotary engines aside), and we are aware of the general conditions and constraints that determine their design, but the specific real-world variability in those conditions and constraints means we can never create a meaningful general reference piston. We could, however, create a specific reference piston for a specific model of engine. Similarly, we can create (for example) a reference implementation in Java of a given Java Community Process's Java Specification Request, but that applies only to a particular and very small subset of the overall software engineering space. Defining a useful reference implementation of, say, a general purpose CrudScreen or a database table or a lexer or an accounts payable system or an A* algorithm or some other common software device is unlikely, but if someone produces a specification for (say) a particular A* algorithm implementation in a given language, or even one to be accessed via some language-neutral protocol, then it is possible to produce a reference implementation of that specified A* algorithm. But that's not what you mean, is it?
Looking at it from another point of view, let's say you are an SE with a speciality in X. Can you create a reference implementation for even a subset of 'X' that other SEs in X will agree with, in sufficient quantity, to generally regard your reference implementation as standard?
I acknowledge that what I am looking for may be the pattern of the piston rather than the ideal implementation of a piston. Perhaps X's standard reference is a list of standard patterns needed to resolve the OnsiteCustomer
presented to SE.
I think I've got it now. Using the piston example, we can certainly agree on a piston pattern: It will be an approximately cylindrical blah blah blah wrist pin blah blah oscillatory motion blah blah sealing blah forces blah blah... Likewise, for (say) developing accounts payable software there should be a collection of standard patterns that are specific to accounts payable, but are (for the most part) standardized across all accounts payable development. Is that what you're suggesting? In other words, each problem domain has, or should have, a set of domain-specific patterns that constitute standard approaches?
This does appear to be part of what I needed.
All right, I think we've determined that for X, SE needs a standard set of Domain-Specific Patterns. But this is only part of our requirements. Does X demand its own Algorithms and Data Structures, or are these implicit attributes of correctly created standard patterns?
Some X have X-specific algorithms, or at least algorithms that appear more frequently in X than elsewhere (for example, the A* pathfinding algorithm shows up in games more than elsewhere), but the creation of new algorithms and data structures is going to be, in general, relatively rare compared to creating domain-specific patterns. Algorithms and data structures - BTREEs, queues, lists, binary trees, etc., are more far more general and universal than domain-specific patterns, and so are far more likely to simply be used than new data structures invented.
Add to our mix a Software
Researcher, SR. It is his job is to create or identify the Patterns in X and submit them to the standardization process.
SE reviews SR's standardized patterns and goes to apply them. Here's the next question: Does SR's research generate the Algorithms and/or the Data Structures outlined above in the Principles and Artifacts list?
There's no reason why an SR wouldn't create new algorithms and data structures if appropriate. In many problem domains, I suspect this will be infrequent. I can't think of any reason why Accounts Payable development would demand any new, previously unknown algorithms, for example. Cutting-edge game design, on the other hand, may well produce new algorithms and data structures.
I think I can summarize our conversation to this point as follows:
Is this really intended to be a summary of the discussion so far, or is it a draft of a formal development process roughly derived from the discussion? It seems to be the latter. Furthermore "XP Processes" seem to have worked their way into the picture, and while I am a proponent of XP, I'm not sure it's sufficiently foundational to be considered a Principle. The same goes for IDE, and why have Algorithms and Data Structures left Principles and joined Development Environment? Are we still in the territory of NutsAndBolts software engineering foundations, or have we moved somewhere else?
- We have 2 actors, so far, SE and SR both working in X.
- SR is tasked with identifying, creating, or refining X-specific patterns and submitting them to the standards committees.
- As a side question: Does SR need to use any metrics on his patterns?
- While I see value in an inclination to using metrics wherever possible - at least in preference to instincts, gut reactions, blind suppositions and the like - I am not sure how to apply metrics to patterns. I can easily apply performance metrics to my choice of (say) an array vs. a linked list, but I'm not sure how it would apply to my choice of (say) a VisitorPattern over a HierarchicalVisitorPattern or vice versa. -- DaveVoorhis
- SE uses his tools (see list) to develop the project.
- SE's toolkit includes
- XP Processes (Hopefully Universal)
- Patterns (X-Specific or Universal)
- Development Environment (X-Specific or Universal):
- Data Structures
- SE then uses the Artifacts to generate and test the project.
Dave, you are right, I think I drifted from the intended summary to a process. I rearranged things because it seemed where the discussion was heading. Perhaps a better summary would have been:
- We have 2 actors, so far, SE and SR both working in X.
- SR is tasked with identifying, creating, or refining X-specific patterns and submitting them to the standards
- SE uses his tools (see list) to develop the project.
- SE's toolkit includes
- Principles (X-Specific or Universal)
- Data Structures
- SE then uses the Artifacts to generate and test the project.
And Ask, do the following belong in the list:
- XP Processes (Hopefully Universal)
Next question: (As a preemptive strike, Yes I know that LAX is Los Angeles' airport designation.)
Presume that SE is using a specific language (call it L).
Presume that SE is using a set of artifacts (Call it A).
"Best" and "should" are only meaningful in the context of metrics, otherwise they are purely subjective and therefore questionable.
- Can we say the relationship L-A-X is the best?
- Do we need a new metric to quantify the L-A-X relationship?
- Is there a specific L-A relationship that SE should use?
- Is there an L-A combination that is the best on the market for SE to use?
(Man, I think I need to Refactor).
Yes, I guess, SoftwareEngineering
is not considered mature until we see the equivalent of standardized nuts listed at the top of this page ;-)
- Which implies SoftwareEngineering is considered mature, since there are so many standardized components and tools deployed between the SoftwareEngineer and the constructed software. Please add to the following list:
[there are so many standardized components and tools] hmmm, many surely, but standardized? I'd consider RFCs to be standards and languages are also standardized, IEEE does its job as expected. Let me annotate the table:
- Algorithms - I know of no standard to formulate an algorithm.
- Assemblers - no standard
- Code tables - OK
- Compilers - no standard whatsoever (but see languages below)
- DBMSs and query languages - SQL is standardized, but servers and configuration and ... ?
- Enterprise Solutions - only as far as processes are standardized and that's not CS
- File Systems - no standard, but only interoperability
- IDEs - no standard, most IDEs not even fulfill basic requirements for workstation applications.
- Interpreters - see compilers
- Macros - well for C maybe
- Markup Languages - XML: OK
- Methodologies - no standard; best practice at best
- Middleware (WebSphere...) - only de-facto by Sun etc.
- Operating Systems - hah! but at least there are standards for security (rating levels A-F)
- Patterns - see methodologies
- Programming Languages - OK
- ServiceOrientedArchitecture - see middle ware
- Templates - for C++
- Utilities - hah! You call the *nix tools standard or what?
- WebServices - OK
But maybe you mean de-facto standards as opposed to de-jure. If a large company is behind it (MS, IBM, Sun), whatever it is, it will quickly be "standard".
Most of what you list that were not on previous lists are specifically Representational or Execution artifacts.
Perhaps we can summarize your list as follows:
- Principles - Patterns, Algorithms, Data Structures (left off your list)
- Language Specific Principles - Templates (as Data Structures), DB Related Data Structures
- Methodologies - ServiceOrientedArchitecture
- Generative Artifacts - Assemblers, Compiler, Interpreters
- IDE Artifacts - Programming Languages, File Systems, Code tables (ASCII, Unicode...), Macros, Utilities, Markup Languages
- User Artifacts - Operating Systems, Browsers, WebServices
- Other - Enterprise Solutions, Middleware (WebSphere...)
What I am looking for is to place the categories for Language Specific Principles, Methodologies, IDE Artifacts, User Artifacts, and Other off of SE's shoulders unless they apply to X.
If you do not like my summary, please feel free to comment.
- I don't understand your summary, so I certainly don't like it. And I don't support your "agenda" of relieving the SoftwareEngineer of some categories of responsibility. There are many different types of SoftwareEngineer, just as there are many different types of software. A JavaVirtualMachine is software; in constructing one, the SoftwareEngineer is constrained by the instruction set of the chip it is written for (and this instruction set is also software). An SQL query is software; in constructing one, the SoftwareEngineer is constrained by the instruction set of the DBMS it is written for (and the DBMS is also software). Analogously, the machines that make nuts and bolts are held together with nuts and bolts.
Again, If SE's specialty is some ProblemDomain
'X', he should not be responsible for having to develop Non-X equipment. Does the bolt-maker also press out the bolt-maker's parts or only the bolt? If X is a VM, then yes, SE would be responsible for a JVM, but if X is accounting, then why should SE be responsible for the design and construction of the VM, Interpreter, and/or Compiler? That's too much knowledge/work for any one person.
- I entirely agree, with regards to the individual; but the discipline integrates all values of X, so where does that leave us? A particular SoftwareEngineer may use the same programming language for a range of applications, not caring what compiler will be used (until they want to use some special feature of the language that may not be supported by all compilers). To them, the source language is a standardized component. To the compiler writer, the source language (or superset of dialects) is a RequirementSpecification?. What is the source language to the SoftwareEngineering discipline?
I would think that would be the Pattern, Algorithm, Data Structure - not including their language based implementations. At the individual level, we need to decouple all the problem domains from the main body of the definition of who SE is. SE needs a common system of development notation (the three mentioned above) and then possibly in their specialization, X, more specific data. Do I really need to include in the discipline's definition all X, Some X, or No X? I think the discipline should include very little direct data on X and concentrate on what is common to all who practice. X is not common to everyone. Take a look at medicine. Does every doctor know the general principles and all the specifics?
- I would think that's a complete non-sequitur. If you want to conclude that SoftwareEngineering reduces to Pattern, Algorithm and Data Structure, why don't you produce an argument? I'm not even sure we share the same concept of Data Structure, so you might clarify that to begin with. My concept of SoftwareEngineering is quite different: mastering the design brief, understanding and/or formalizing the context, analysing the functional and non-functional gaps, evaluating prospective solution ideas for their impact on the gaps, progressing the most promising solution ideas, iterating ad nauseam or until successful. That much, I believe, is common to all software engineers (and, indeed, all systems engineers: so if you want to define SoftwareEngineering, it's SystemsEngineering where software is expected to contribute significantly to the system's goals).
- I don't think the author above meant that SoftwareEngineering reduces to Pattern, Algorithm and Data Structure, but that these are the fundamental materials - the NutsAndBolts, as it were - of software engineering. In one way or another, we are always using patterns, algorithms or data structures, regardless of what processes we might use to manipulate, choose, or evaluate those patterns, algorithms, or data structures. What you've described - mastering the design brief, formalizing the context, analysing gaps, evaluating and so forth - sounds like part of a systems engineering process, as opposed to the materials manipulated by the systems or software engineering process, which is what I thought this page was about. However, to me the processes you mentioned seem more reminiscent of what I consider systems integration than software engineering. I've always seen software engineering as being programming with an eye to refining the process of programming itself, rather than (shall we say) merely programming and being done with it. In other words, software engineering is a reflective and scientifically-based craft whose ongoing goal is to produce both better programs and better programming. Systems integration, on the other hand, is focused on the application of software engineering products - those primarily being pre-produced software. That said, I'm beginning to get that twitchy feeling I get whenever a hair is being split in my general vicinity, and I don't like it. -- DaveVoorhis
- A data structure, by the way, is an arrangement of data designed to facilitate the application of a given algorithm or set of algorithms. An algorithm is a procedure or set of steps for solving a problem. A pattern is an abstract description of the interacting components - i.e., problem-solving units - of a system. That funny noise you hear is the sound of a can of worms being opened. -- DaveVoorhis
- Okay, Dave, I can see where you, at least, are coming from. Mine certainly is a process view, since I hold SoftwareEngineering to be that which a SoftwareEngineer does (the output from the process being software, of course, or its specification). If you want to introduce a dichotomy between those who "merely" program and those who learn from the process, I can't really object to that. (I don't feel any need to differentiate systems integration from SoftwareEngineering, but I would assent to the proposition that, if there is no original software, it can't be SoftwareEngineering. Apart from that, it just seems a matter of degree.) Accepting your distinction between JustaProgrammer and SoftwareEngineer for the moment, what are the inputs to the SoftwareEngineering process as distinct from the "mere" Programming process? It seems to me that "data structure", as defined above, would generally be an output but not an input; algorithms would be either input or output; and patterns would be input and/or output, providing a feedback mechanism for your "better programming".
- As for the "materials", in addition to the actual problem or needs and the resources (personnel, cash...) available, we have the tacit knowledge contained in the development context (policies, standards, technical environment and infrastructure, programming languages, frameworks...) and the SoftwareEngineer's expertise (algorithms, patterns, heuristics).
It is not my intention to reduce SoftwareEngineering
to Pattern, Algorithm and Data Structure; but to explore these as tools of a SoftwareEngineer
. I pushed the methodologies/practices off because I don't want to get bogged down into the politics of them.
I ask that someone else summarize this if I don't do so by the sixth. I am trying to objectively find a way of doing so that does not get into a politicized discussion. It may turn out to be that there is no easy way. I know that I am not trying to find out the body of knowledge itself, but beyond that, I am not certain.
See also: IsComputerScience