July 2002

Software Testing, the Missing Key Process Area

NJSPIN's own Jim Heil gave the presentation at the July meeting. Titled Software Testing: The Missing KPA , it started off with the members taking a 10-question survey of testing practices in use within their organizations. (One case in point: "Is testing performed as a systematic and organized activity, per an established, documented, and carefully considered methodology?") Each answer was given a weighted score. After the answers were totaled up and applied to a simple scale, the room let out a collective chuckle.
Don't let your users jump into untested waters

Although testing requires 40 to 50% of the software development effort on most projects, there is no Software-CMM KPA that focuses specifically on software testing. Even the Level 3 KPA for Software Product Engineering (SPE) is very general, and does not discuss testing. There was an effort several years ago to introduce a separate KPA for testing, but it was eventually rejected.

In response to this gap, a 20-point "Test Process Improvement Matrix" was created for the purpose of evaluating and improving an organization's software testing processes. A total of 20 specific areas are identified for testing, and each area provides specific levels of process maturity, with related criteria.

Test Maturity Matrix Key Areas

1-Test Strategy11-Commitment/Motivation
2-Life-cycle Model12-Test Functions & Training
3-Moment of Involvement13-Scope of Methodology
4-Estimating and Planning14-Communication
5-Test Specific Techniques15-Reporting
6-Statis Test Techniques16-Defect Management
7-Metrics17-Testware Management
8-Test Tools18-Test Process Management
9-Test Environment19-Evaluation
10-Office Environment20-Low Level Testing

The actual Test Maturity Matrix is a kind of report card developed around these key areas. During a "quick scan" assessment, which is performed by an independent reviewer, each key area is given a grade. Typically taking three days to complete, a quick scan allows an organization to quickly find out the overall health of its software testing processes.

In addition to quick-scans, the Test Maturity Matrix also supports more in-depth assessments. Jim provided some insights into conducting and using these assessments. For example, doing quick-scans for pilot groups will help ease the introduction of larger assessments within organizations. The trade-off to this approach is that the benefits of undergoing an assessment would be delayed.

A common theme throughout Jim's presentation is that testing should start early in the project lifecycle. Even testing a project's requirements while it is still in the requirements stage can be a useful and cost-effective way to prevent defects later on. (Web master's note: NJSPIN's March 2002 Program goes into the effectiveness of inspections and reviews throughout the development lifecycle.)

To view the complete presentation, click on the hyperlink below.

July 2002 Program: Software Testing: The Missing KPA

Bio - Jim Heil

Jim Heil is the Chair of the North Jersey SPIN. He is the director of Testing Services and Configuration Management for Software Management Solutions, a software consulting and training firm located in Clark, NJ. Jim has taught numerous courses and seminars on software testing, software quality, software standards, software requirements, risk management, etc. He has also organized and led national conferences on Software Testing, Automated SW Test Tools, and Process Improvement.

Prior to joining SMS, Jim led the Software Quality and Process Improvement programs at Lucent-Mt. Olive. Earlier, he worked at Telos Corp. on Software Process Improvement and served as a Sr. Consultant to the U.S. Army Software Engineering Center at Ft. Monmouth. Earlier, he worked at MITRE Corp., and IBM, in Software Quality, testing, software development and SPI.

Jim is a Certified Software Quality Engineer (CSQE), and teaches the CSQE class for the American Society of Quality (ASQ). Jim also taught the CMM and SEPG classes at Telos and Lucent.

Jim chaired the industry Task Force on MIL-STD-2167 and 2167A, and was a key member of the taskforce on standards MIL-STD-498 and IEEE-1498, as well as the SQA standard MIL-STD-2168. He also was a member of the U.S. Technical Advisory Group to ISO for Software and Systems standards.

Jim has Masters Degrees in Operations Research and E.E., plus an MBA degree, and has completed the course work for the Doctorate in E.E.

Business Items

There will be no meeting during August. The topic for September 18th's meeting will be a panel discussion, featuring computer science faculty members from NJIT, Stevens, and Rutgers. The September meeting will be held at Telcordia Technologies.

The NJSPIN.ORG Web Site Survey was distributed at the meeting. The feedback generated from this survey will help your web master to determine what features are the most useful to SPIN membership, as well as what features the membership would most like to see in the future. Thanks to everyone for contributing their comments and suggestions!