Software Quality

Studies by various people suggest that code inspections are the cheapest way to fix bugs. The argument goes like this:

1. You get 3 pairs of eyes looking at the code before it is tested.
2. Any bug that is found is fixed before it is built, and tested. i.e. Code going to test is of better quality.
3. Bugs are cheapest to fix at code inspection time instead of at build time, or integration time, or after deployment.
4. Yada, yada, yada.

So they say, do the following:

1. Print the code.
2. Gather 3 people, and give them the code ahead of time to study.
3. Inspect the code and log the bugs.
4. Fix the bugs and do a follow-up inspection.
5. Inspect code at a rate of about 5 lines per minute maximum.

This is really cool, but it doesn't happen. Why?

1. When schedules get crunched, code inspections preparation is the first to go. People show up without studying the code.
2. They then pick at stupid formatting errors and other low-value items.
3. Inspections are written to logs that are never read again.
4. Code gets inspected at 20 lines per minute in order to avoid "wasting time". People go through the motions but don't add value.
5. Nobody gets rewarded for finding bugs in somebody else's code, but they do get rewarded for meeting and beating deadlines. Designer incentives make code inspections a poor investment of designer time (from the designer's perspective).
6. The opportunity cost of code inspections is very high for designers in that they could better use the time to meet their own deliverables.

So...the $64000 question. How can we get desingers to benefit personally from inspecting code?

1. Assign 1 code inspector for each author and make the code inspector the first point of contact for any bugs on a module he inspects. That gives the inspector the incentive to inspect properly while greatly reducing the code inspection load for the average designer.
2. Do code inspections on the "diff" of the file that is submitted into the version control system (e.g. Subversion, CVS, Clearcase, Perforce, SourceSafe...), and put the log in the submission comments or in the feature or bug tracking system for that submission. Electronic inspections on diff's reduce the need for booking meetings and other time wasters. Differential inspections speed up the process.
3. Write a unit test and/or a system regression test case for every bug fixed. That ensures that if the bug reappears, the test case will catch it.
4. For each function inspected, ensure that the inputs are always checked for validity. For example, if the input of a C function is a parameter that could take the values "red", "greeen", or "blue", and the type of the parameter is a "char*" then ensure that the input is always checked to always be one of the three values, and that the function returns an error if it is not.
5. Ensure that all functions return valid outputs on all exit paths. If the possible return codes are OK=1, SOSO=2, and NOWAY=3, then ensure that no other value is ever returned by the function.
6. Ensure that the data that drives the behaviour of the function is normalized to at least 3rd normal form. This is not only database data, but also data in flat files, global variables, class variables, file static variables, local variables, etc. You can always de-normalize if performance suffers, but that should not be your first step. Normalizing is a way of ensuring that some thing or concept is only stored in one place. This step really forces designers and inspectors to understand the code.
7. Functions manipulate data and send data to outputs such as the calling function or output devices. Ensure that a function does everything or nothing. (This is often called an atomic transaction). Recovering from partially completed functions (in terms of data manipulation and outputs) tends to be hard to write, and bugs related to these are hard to reproduce.
8. Check algorithms for O(n-squared) or worse behaviour that could be easily converted to O(n) or O(log(n)) or something less painful. This can give a quick and cheap performance boost to a function.
9. Use as many of these techniques as you feel comfortable using, and don't worry if you don't get them all. Usually the first 4 steps provide 80% of the benefit unless the programmers are not very good.

There is a field of software development on code complexity metrics. It measures things like lines of code, parameters, branching, etc. Each of these quantities have an effect on "code complexity". Why do you care? Complex code is hard to test, hard to debug, hard to fix, and hard to know if it is complete. i.e. The more complex the code, the more (over)time you'll waste trying to get it right. It is not an enjoyable experience.

There is an Eclipse plug-in called Eclipse Metrics which plugs into your Eclipse IDE and gathers metrics for you. This helps you to target and fix your riskiest modules, and it will waste a lot less of your (over)time than a manual code inspection.

If you have other IDEs, or command lines, and if you want an industrial strength product with support, try the Klocwork Insight product. It automatically inspects code as you go along, and flags the most common software defects and security vulnerabilities even before you check it in.

There is also a great article at Embedded.com
on taming software complexity.

Today many software companies from wealthy countries are outsourcing software development to low-cost development centers such as India and Vietnam. Properly executed, this offers a great way to improve product quality. Why?

1) Indian and Vietnamese people cost less.
2) With time-zone differences, they work while you sleep. This means that you could give work requests to them before you go home, and have some of them completed in the morning.
3) Outsource partners are not emotionally attached to the product or your people. This can be used to your advantage.

So how do your improve software quality by outsourcing?

1) Decide that it is "black box testing" that you want to outsource.
2) Pick a project that needs 2-5 testers. Contact an experienced and credible firm such as Taraspan or TNA Solutions to find the appropriate talent.
3) Provide your testers with functional specs, system use cases and the software without the code, as well as a relevant testing environment.
4) Integrate your bug tracking systems.
5) Build and transfer a software loadbuild to the outsource daily or as often as possible.
6) Start the virtuous cycle of development, testing, bug-fixing, development...It will go quickly.
7) Enjoy the benefits.

Why does it work?

1) Black box testing doesn't require the transfer of too much technology expertise, and minimal amounts of product expertise.
2) Round-the clock dev-test-fix-dev cycles accelerate development.
3) Emotional divestiture leads to less fear in testers with confronting software defects. e.g. An outsourced tester is more likely to be straightforward with finding bugs with a software designer with whom he has no emotional ties or political dependencies.
4) Testing is a low-risk activity to outsource, and it keeps your technology expertise close to home.

So there you have it. Outsourcing your black-box testing will help you to improve your software quality.

Today many software companies from wealthy countries are outsourcing software development to low-cost development centers such as India and Vietnam. It is quite fashionable among business leaders to outsource work to low-cost centres because it costs less money. There is one case where outsourcing will increase costs and reduce quality. That case is new product development.

Over the years I have seen it in big companies and small companies. Here is the typical scenario:
1) Somebody, usually a business executive, decides that they'll save money or quickly acquire development talent by outsourcing the development of new software.
2) A provider (local or overseas) is selected and everybody moves forward.
3) Product specifications and functional specifications are written by product development primes and transferred to the outsourcers. There is also a strong coupling between locally developed software and the outsourced software (such as shared code, data, and data schemas, all of which are evolving).
4) Outsourcers and product development primes assure each other of the greatness of the idea, and both make up schedules that show how beneficial the relationship is.
5) The outsourcers start writing detailed functional specifications and design documents. Reviews take a long time because of time, distance, and cultural differences.
6) The outsourcers start developing the system. The functional and product specifications become progressively more irrelevant.
7) Outsourcers start making assumptions that are either just plain wrong, or don't reflect the (usually tacit) intent of the product development primes OR they constantly ask for clarifications and guidance.
8) Schedules start slipping, but every swears on their great grandmother's grave that everything is on track.
9) To mitigate the project, the product primes put more time into (micro)managing the outsourcers, which costs them and the outsourcers more money. Communication continues to be difficult because of time, distance, and cultural differences.
10) Outsourcers deliver an incomplete or low-quality product on time, or a high-quality incorrect product late. The quality is perceived as appalling.
11) Product development primes are upset because the outsourcers screwed up. Outsourcers are upset because product development primes didn't tell them exactly what they wanted.
12) Product development primes cancel the project OR spend tons of money to fix the mess and do it right. Either way it ends up costing more than doing it locally.
13) Outsourcers are blamed for the fiasco, software development primes are frustrated because they know they could have done it better locally, and the business executive moves on to outsourcing the next new software product.

Why does outsourcing new software development lead to low-quality products or the wrong products?

1) It is hard to document all of exactly what you want, especially when the writer doesn't have the complete business or technology domain expertise. (This is almost always the case with new products). When you build a product locally, you can always go to the developer directly and make changes quickly and cheaply without a lot of process or time. Easy communication between product manager and developers leads to the right product more often.
2) Managing remote projects costs twice as much as local projects if done properly. The communication barriers make remote development much more expensive, and often lead to the wrong product.
3) For new products, technology domain expertise often does not exist with the product development primes. They are building it as they go along. If they don't have it, it is hard for them to know if they have it and if they have properly transferred it to theiroutsourcers. Lack of domain expertise means that different quality-time-scope trade-offs are made.
4) The business interests of outsourcers are very different from those of product development primes. Outsourcers must "complete" the project or milestones to get paid, and they are often paid as they go along. Software development primes get paid only when they sell their product. The problem is that a "complete" project in the eyes of theoutsourcer is often not a paid product in the eyes of the product development lead. Often "complete" outsourced projects have unacceptable defect rates and clunky, slow, or of inconsistent functionality.
5) Outsourcers often stall if a requirement is not perfect, thus requiring continuous direction from the product development prime. If they don't stall for guidance, they make incorrect decisions (usually not their own fault).

How can you make outsourcing new software development succeed at building high-quality correct products? It is very hard, but here are a few techniques:

1) Align the interests of the product development prime and the outsourcer. For example, lower the up-front payments until the right product is delviered with acceptable and mutually understood quality metrics.
2) Keep the product managers and technology architects with the outsourcers. Product managers are the folks that translate business needs into technology requirements. If your outsourcer can speak for your business needs, then you don't have to micromange them. Of course, if you have product managers there, you are only a short step from having a full-blown development lab in the low-cost centre. (That is not a bad idea either especially for new product development in a low-cost centre).
3) Bring in local expensive "mercenary" software contractors on-site if you are fighting a talent shortage. Transferring technology and product domain expertise is much easier with the guy in the next cube than the guy on the other side of the world. It actually ends up being cheaper to build high-quality correct software because the benefit of easy transfer of technology domain expertise exceeds the extra cost of the local software contractor.
4) Outsource development of products that the outsourcer has already built for somebody else. i.e. The outsourcers already have the domain expertise, and are more likely to build high-quality correct products.
5) Outsource the development of strongly decoupled components. Usually that means that the only interaction between the components is through buffered messaging protocols and where the outsourcer has no control over the protocol.

Does this all mean that all outsourcing for new product development is bad? Of course not! Outsourcing the development of new software exacerbates the problems that already exist regarding the development and communication of domain expertise, understanding requirements, aligning interests, and clarifying accountability. I'm sure great examples exist, but I have not yet seen them or characterized them.

You have the good, bad and ugly of outsourcing new software development and its effect on quality. Tread carefully...

Test automation is the act of writing a computer program to run tests on your computer programs. How can this help software quality?

1. Tests defined in an automatic program are set up and executed exactly the same way every time. This lets you ensure consistent functionality every time the tests are run.
2. Tests defined in an automatic program are run by a computer, which is much cheaper than paying a manual tester. Because the tests are cheaper, more tests can be run on every test run, thus exercising more of the code under test, thus leading to a higher probability of finding defects.
3. New tests are easy to add to an existing test suite, thus enablng even more test coverage on subsequent test runs.
4. Running automatic tests regularly ensures that new functionality or even small code changes don't break the system. Defects introduced by "small" changes to the system are very hard to track down and fix.

So why don't more projects use test automation?

1. The project is too small. For small projects, it is just often cheaper to run the tests manually, or at least it seems that way.
2. It is often hard to develop test automation expertise in small teams.
3. It is often hard to understand what to automate and what not to automate. This can lead to the automation of inappropriate tests.
4. Automated tests take too long to run if not designed carefully. That often requires more computing resources than some project leaders want to acquire.
5. Maintaining and evolving the automation scripts take too long, which can cut into manual testing time. Although the project benefits, it is often at the expense of tester (over)time.
6. Test automation can give productivity gains of 100% or more in some testing teams.
7. Automation? We don't need no stinkin' automation. Our developers are top-notch and we just have to test the success paths every so often.

How can you implement test automation?

1. Decide that you want to do it. You may have to leave the why's and how's until later.
2. Decide that you will not automate usability testing, UI visual testing such as table formatting, spelling, label naming and the like. Those really need a person to tell if such a test passes.
3. Define a unit test suite comprised of a small number (e.g. 2) of unit tests. Have the build script run that suite automatically after each software integration/build, and gather the results. Once you have done 2 unit tests, you may find yourself wanting 200 or 2000.
4. Define an automatic test suite that automatically deploys the software and then runs a small number (e.g. 2) of system tests. This in itself will reveal a lot of defects because software deployment requires a lot of data, software, ordering, timing, access priviledges, etc to get it right. Not being able to deploy software automatically is itself a defect because undeployable software can never be tested properly.Once you have done 2 system tests, you may find yourself wanting 200 or 2000.
5. Try to make the deployment tests an automatic part of the build.
6. Define test suites for success paths.
7. Define test suites for invalid inputs.
8. Define test suites for failure paths.
9. Define test suites for load, stress and performance testing.
10. Define test suites for upgrades from previous versions of the product. This is a great suite for reducing customer complaints when they upgrade.
11. Don't worry about failure. Believe it or not, the first 4 steps will bring about half of the benefits, because it will and will change the mindset of the team towards wanting more automatic testing.

So...go ahead...just do it! Test automation will benefit your project and save you (over)time.

We all know that automated testing is good. Why not take it to the next level, which is "continuous" automated testing. What this means is that there is an automatic process that periodically checks out your software and builds it to ensure that there are no build errors.

The reasons for continuous building & testing is that you don't need an expensive build test team to this kind of work. (You will still need these folks, but not as many of them). i.e. You save money. Also, automatic and continuous tests ensure that that your changing code base is regularly put through its paces to ensure that a small change in one place doesn't break the test suite without somebody knowing it. This lets you fix bugs much sooner after they appear, and that leads to higher software quality.

Cruise Control does just that. Cruise Control:

• Lets you configure how often you run the build


• Automatically generates reports and summaries.


• Notifies interested people by email or RSS feed.

Cruise control works for Java, languages in the .Net family, Ruby, and there are plenty of 3rd party tools that work with Cruise Control.

Mary Poppendieck has done it again. This video goes on about some of the history behind quality management, culminating with the works of Ohno and Deming. The history is pretty interesting. The Americans had a great approach to it when they ramped up factories for World War 2. She then shows how some of these ideas work for software development. It is a definite eye-opener especially if you have worked in a lot of modern software development places. She covers building the product right as well as building the right product.

You can see the video at
infoq.com.

Heroku is a little startup in San Francisco that lets you build Ruby On Rails applications online. They only require you to install the Firefox web browser on your computer and then off you go. What do you get?

1) You get a Heroku subdomain for your application online.
2) You can limit access (read or read-write) to only named developers and testers, and later flip this access
3) You get Subversion for controlling your source code.
4) You get a full-blown Ruby On Rails development environment including rake, migrations, etc.
4) You get a command-line shell to run key scripts.
5) You get syntax-highlighting editors through your browser to edit your code.
6) You can install most Rails plugins and Ruby gems.
7) You project develops against the PostgreSQL database which you don't even see.
8) Heroku is built on the Amazon EC2 platform.
9) Deployment of the test application is done at the flip of a button.
10) Deployment to a production is done at the flip of another button.
11) Your application scales up because of access to Amazon EC2. If you need more data, there are interfaces to take advantage of Amazon S3.
12) You can take code and data snapshots any time and save them to your computer if you need to.

Why does this help you build high-quality applications?

a) One development environment, which is the hosted Heroku account. That eliminates workstation configuration related software defects. This also allows local and remote development and test teams access to the same system on the same terms.
b) Source code control via subversion. 'Nuff said.
c) One-step deployment. This reduces and eliminates software defects related to inconsistent deployment processes.
d) Ruby On Rails reduces defects related to having various designers writing application application boiler plate code by hand.
e) Heroku and Amazon EC2 reduce defects related to scaling out processing, bandwidth, and storage because Amazon EC2 allows you to build a system that can scale either of these out gracefully.

The biggest surprise of Heroku overall is how fast you can go from zero to deployment, without losing quality in the intermediate steps.

Jeff Sutherland presents this great video at InfoQ on how Google used Scrum and Agile practices to improve software quality and productivity at Google. It is certainly worth an hour if you want to learn something.

He also goes into how various techniques like pull-driven development work better than push-driven, and how feature-oriented burn-down charts work better than task-oriented burn-down charts. I especially liked how he extended the definition of the word "done" (for software development), not just to include requirements, design, and coding (which is typical), but also unit testing, system testing, deployment, and going live.

Luke Francl has put together a great presentation on why testing is overrated. He explains quite plausibly why testing in immature testing organizations is not effective. He also points out other ways to achieve software quality.

The video can be found at InfoQ.