Quality and Security

There is no single software quality assurance (SQA) approach mandated for nuclear facilities in the USA. DOE has its standards and directives; while the NRC has its regulations and regulatory guides. These in turn reference many other national consensus standards. For example: from ASME, IEEE, and NIST. All of these allow tailoring and none mandate a specific implementation of quality requirements.

If there is one foundational document from which all others can be said to flow it would be ASME NQA-1, Quality Assurance Requirements for Nuclear Facility Applications. IMHO, if any software quality assurance program is in full compliance with the requirements of NQA-1, it can be reasonably argued to be in compliance with both DOE and NRC SQA requirements. (Any issues will be related to shortcomings in the DOE and NRC regulations and directives or their related national consensus standards. Not with the consensus approach to software quality assurance.) Personally, I would like to see all SQA independent verification and validation (IVV) and uncertainty quantification (UQ) programs be traceable to NQA-1 requirements. I think anyone involved with software at a nuclear facility should be generally familiar with the SQA approach taken by NQA-1.

But NQA-1 has its shortcomings. These shortcomings are not specific to just nuclear facilities and so are of general interest.

For one thing, NQA-1 is not freely available. (It's current cost is $220.) It is unacceptable that finding out what "the law" is should cost money, or that documenting NQA-1 requirements and justifications should be a potential copyright concern.

A larger and technical issue is cyber security. Cyber security plans and management are mostly developed, approved and administered independent from a nuclear facility's general SQA staff and management. (For example, the NIST 800 series of cyber security standards offer many checklists and templates cyber security personnel can choose from.) NQA-1 does not offer any specific guidance on the proper relationship between software quality and cyber security requirements, processes or personnel. This is not so much a criticism of NQA-1 as the current state-of-the-art.

Another issue is guidance on the quantitative relationship between cyber security and nuclear safety. How is cyber security to be integrated with a nuclear facility's probabilistic risk assessment (PRA)? For example, how are cyber security threats to control systems (SCADA, DCS, PLC) to be quantified? Again, this issue of risk and uncertainty quantification is mostly a function of the current state-of-the-art.

SQA - Important Owner Responsibilities

This is just a short reminder that custom software developers, even the best, cannot fully assure the quality of the code they write. Software quality assurance (SQA) is the software owner's responsibility too. That’s because quality is not a property of the software. Therefore, quality can’t be built into the software by the programmer. Quality is an assessment, a prediction, of how well the software will perform when used. It's a belief about how well the custom software will meet its requirements. Each stakeholder will have different knowledge about the software and thus have a different belief. (And it's a reason that software with multiple owners tend to be higher quality software.)

This is a very important concept with a lot of ramifications. I want to mention three things one particular stakeholder, the custom software owner (e.g., the client of a custom software consultant), can do to efficiently and rationally acquire the knowledge that will provide him/her with the necessary level of confidence.

  • Before the start of the project, and periodically thereafter, review the corrective action processes within the development organization. How are bugs in the software discovered and eliminated? How are defects in the software development processes identified and corrected? (This will give the owner confidence that the development organization's processes must improve over time. It's the idea behind such things as CMMI.)
  • Review the development organization's software life-cycle methodologies. What types of development processes do the developers follow? How do the developers justify the tools and technologies used to produce custom software? (This will give the owner confidence that processes known to produce quality software are being used.)
  • Trace the progress of the code being developed. How is progress measured? What evidence of progress is being produced and documented? (This will give the owner objective evidence the desired development processes are actually being followed.)

Note that besides independent verification of SQA processes being used by the development organization, the software owner's input is critical for SQA tailoring. The appropriate scope, rigor, and intensity of SQA processes depend on the nature and intended usage of the software. This validation activity is the owner's domain and responsibility. (This is an explanation for why, whenever it is possible for the owners and developers to be the same people, "eating your own dog food" is such a common technique for producing quality software.)

Some Empirical Findings

There is a fairly recent American Scientist article that presents some of the latest empirical findings in software engineering. (Such stuff is important since the analysis of programming methodologies tends to be dominated by theoretical arguments.) The findings are not presented in list form. For convenience, here is a partial list. Please read the article for more detail and the authors' perspectives on the findings.
  • More lines of code mean more errors in the code, all other things being equal. (Such is the state-of-the-art in quantifying software quality!)
  • Pair programming is approximately as productive as solo programming. (That is, the useful output of a pair is roughly comparable to the output of two individual programmers.)
  • It is not possible to reliably identify great programmers based on their personality type. That is, there still is no personality test that can be used to skillfully identify outstanding programmers from mediocre ones. (By focusing on personality types, the authors of the article seem to imply, at least to me, that it is generally already known and accepted that in-depth knowledge of languages, design patterns, or standard algorithms; or having formal certifications or degrees from prestigious universities, won’t distinguish the outstanding from the merely mediocre either.)
  • Trying to imitate traditional engineering practices doesn’t have a positive return-on-investment. For example, it seems that UML tools, while useful for contemporaneous communication of ideas for the software design, are not worthwhile for generating or even documenting the actual software design. (As noted in my last post, this has been my observation being both a mechanical engineer and a software engineer. Traditional engineering and software engineering are two very different hats.)
  • Code imitates life: “The organization that designs a system will produce a design whose structure is a copy of the organization’s communications structure.” For example, if a programming project involves four teams from four different groups in the organization, the program created will have four major parts. And the probability of bugs increases with greater organizational (not physical!) separation between team members.
  • Open-source software does not produce software any less vulnerable to security flaws than closed-source software. (To me, this seems to imply that open-source software is not inherently any less bug free either.)

Empirical Software Engineering

I found this American Scientist article rather interesting, though flawed. It's titled: Empirical Software Engineering - As researchers investigate how software gets made, a new empire for empirical research opens up.

Software development is quite logical and tends to attract programmers of a mathematical mindset. Innovative programs can be valuable so programming also attracts creative talent. But the code produced by programmers, unlike the works produced by mathematicians or fiction writers, can be embodied in computers. Given life so to speak. (Some AI programmers believe this quite literally.) Therefore, since life can be studied scientifically -- so can code. And such study can be labeled a "soft" science, for example, like biology. Embodiment means that reviewing code can be qualitatively more than just a peer review or literary critique. I see nothing wrong with looking at code this way.

But this can be taken too far. A seemingly similar point can be made about software development processes. The methodology for generating code (that performed by programmers, not compilers or byte code generators) is certainly embodied. Thus, software development can be labeled software engineering and comparisons can be drawn between the methods programmers use developing code and the design methodologies used by engineers in the various "hard" engineering disciplines. For example, it is very common to draw an analogy between object-oriented design and the design of mechanical structures, systems, and components.

Anecdotally, being both a professional mechanical engineer and a professional programmer, this is not an analogy I would ever draw. IMHO, there are advantages to having a hard engineering background in software engineering, but it's more fundamental than drawing analogies between methodologies.

Further, computing, the activity of using computer hardware and software, can be labeled computer science and an analogy drawn between the knowledge used by the hard engineering disciplines (i.e., scientific knowledge) and the knowledge acquired by computer scientists. It is not at all clear to me that this is a good analogy to make.

Thus, my complaint with the American Scientist article can be seen in the first sentence: "Software engineering has long considered itself one of the hard sciences." The authors basically agree and argue that a new empire for empirical software engineering is opening up. However, I don't take the analogy between software engineering and hard engineering very far. I certainly would never equate them.

Besides, this conflates science and engineering. Science is the acquisition of scientific knowledge. Engineering is the application of scientific knowledge. Engineering is not science. So not only is software engineering not hard engineering, it is not a hard or soft science.

But flawed terminology notwithstanding, the article had some interesting things that I will get to in my next post.

Res ipsa loquitur

This came up in another context recently and got me to thinking. One of the interesting things about object oriented design (OOD) is that it generally subscribes quite literally to the principle of res ipsa loquitur which in Latin means "the thing speaks for itself." Tying functions to data structures means that the resulting objects are responsible for how their data can and should be used. This goes well beyond using OOD to create simple designs by reducing the number and complexity of relationships between program components. OOD software architects generally create objects with the intend of making them easy to (re)use for specific programming tasks rather than designing objects to have as little interaction with the rest of the software as possible.

This is very much different than the approach taken in science where the axiom is that the data does NOT speak for itself. Where theory does all the talking and the data is used only to test theory. In science, the same data may fit several different theories. For example, at low speeds and ordinary precision, velocity measurements confirm both Newton's Laws of Motion and Einstein's Theory of Relativity very well. It's only when you have high speed requirements that the theories start saying different things (make different predictions).

I've noticed that Google's Go language (golang) takes this different approach. It is not an object oriented programming language. In Go, you can define a data structure (type) and then afterward associate any number of methods or interfaces to that specific data structure. These methods and interfaces are not intrinsically part of the data's type themselves. A programmer never creates Java-like objects. A different programmer could take the same data structure and define a completely different interface for it.

I call Go's style of programming "elephant" programming. (With apologies to the parable of The Blind Men and the Elephant.) I tend to favor elephant programming in situations where the user requirements can be organized into distinct groups of requirements. Like the blind men feeling different parts of an elephant and interpreting an elephant to be like a wall, a snake, a fan, or a tree trunk. Like when the same software suite will be used in different ways by different groups of people within a company or other large organization. The data structure is often a relational database or even large flat files for many scientific applications. The sets of functions and interfaces embody the different user requirements and comprise what is called the middle-tier in the client-server programming paradigm. Very practical and efficient. Yet nary an object speaking for itself.