CHEAPER, BETTER, FASTER: A MEASUREMENT PROGRAM THAT WORKS

 

By Janet Russac

It is not enough to measure. The collection of size, effort, time and defect data is just that, a collection of data. In order for an IT organization to improve, it must use these data to determine best practices, model process improvements, establish benchmarks, analyze trends, improve estimation, educate customers and enlighten the IT organization from managers to developers.

 

This article provides a basic introduction to metrics and an organizational metrics program. First, the concepts, benefits and uses of metrics will be discussed. This will be followed by an overview of the metrics process including definitions and activities. Emphasis will be given to measures that have the greatest impact on IT, that monitor performance trends and that identify best practices within an organization.

 

Metrics: Concept, Benefits and Uses

A measure is a number that assigns relative value such as function points or work effort. Combinations of two or more measures produce a metric such as delivery rate (hours/function points) or defect density (number of defects/function points). Metrics are standards by which a process or product can be objectively evaluated.

 

In order to maximize the success of a metrics program, an organization should follow the following guidelines:

 

Metrics can be used to identify best practices, model process improvements, establish a baseline from which to determine trends, assist in estimating and planning projects, manage budgets more effectively, identify the quantity and quality of the delivered product, compare the effectiveness and efficiency of current processes and tools, provide a basis for industry comparisons and enable better communications between customers and developers.

 

Metrics Selection Criteria

The metrics selected for your organization’s metrics program should be industry standard metrics, meet the goals of your measurement program, be clearly definable and easily understood, have the ability to be collected consistently at a project level and be usable at a variety of summary levels, be realistic and measurable and align with customer satisfiers and the needs of the development organization.

 

Metrics Categories

When commencing a measurement program within your organization, start with a few metrics that can be easily, accurately and consistently collected. The metrics categories discussed below are recommended for a start-up program.

 

Productivity

Productivity is the measure of the efficiency of a process to consume inputs and produce outputs. In the IT environment, this can be measured using the following formula:

Productivity Rate = Effort in Hours / Size in Function Points

 

Rework is another measurement of productivity that can be calculated as follows:

Rework Percentage = Rework Hours / Total Project Hours

 

It is important for an organization to provide an exact definition of rework so that those hours are captured consistently across the organization. Rework is the process of redoing, correcting or discarding a deliverable from a previously completed life cycle. Rework is not due to a change in the customer requirements.

 

Quality

Quality is a measure of both the software process during the development life cycle and the software product delivered. The software process can be measured using the following formula:

Defect Removal Efficiency = Total number of defects found prior to delivery / Total number of defects discovered (before and after delivery)

 

The software product can be measured by the following:

Production Defect Rate = Total Production Defects / Function Points Installed

(Note that production defects should be collected for a finite period such as 6 or 12 months after product installation or longer if quarterly and/or annual processing is affected).

 

Numerous quality metrics exist, but the two aforementioned are recommended for a start-up program as they are more easily captured. One other quality metric, however, may illustrate a problem that has a significant impact not only on product quality but also on cost and time to market. “Scope change”, also known as “scope creep”, measures the amount of functional specification revisions that occurs during the development process. Scope change reflects added, changed and deleted functionality and is tracked throughout the development life cycle. Scope change should be captured for each change request after delivery of the initial requirements and is calculated as follows:

Scope Change = Added + Deleted + Changed Function Points / Original Function Point Count

 

Cost

Cost is one of the most important factors in the decision on whether or not to pursue a project. Costs overruns can cause a project to shut down before it is completed or at the very least result in a very unsatisfied customer. Cost efficiency can be measured using the following formula:

Cost Efficiency = Actual Cost / Function Points Installed

 

Time

Time is used to develop schedules and is a factor in determining the resources needed to complete a project by the due date. Time can also affect the quality of the product produced. Scope creep can negatively affect the time estimated for the project. An organization can track time and use it to assess the impact of various factors on the actual delivery date of a project or to assist in estimating future projects.

 

Project duration (in months) can be measured as follows:

Project End Date – Project Start Date

 

There are many other possible metrics such as customer satisfaction and estimating efficiency that your organization will want to eventually implement. They can be important and useful to your organization, but it’s best to start with a few metrics and build from them as your metrics program matures.

 

Key Measures

Function points (size), hours (effort), defects, calendar months (duration) and dollars (cost) are the key elements of a start-up measurement program. We can combine them in a variety of ways to produce the above metrics. It is imperative that an organization clearly defines and effectively captures these measures consistently. If these measures are faulty, so will be the resulting metrics.

 

Function Points

Function point analysis is a standardized method of measuring the functionality of the software from a user’s point of view. It considers the logical rather than the physical design and thus can be used independently of the technology used for implementation. There are many benefits for using function points to size an application or project including the ability to use function points across languages, platforms and other physical characteristics, the ability to use them at different phases of a project’s lifecycle, the ability to easily incorporate them with other measures to produce metrics such as delivery rate and the ability to use them in industry comparisons.

 

More information on function points can be found in the “Function Points as Part of a Measurement Program” Chapter in this book and the Website of the International Function Point Users Group (IFPUG), www.IFPUG.org.

 

Hours

It is critical that each project team accurately track time, including overtime. The metrics produced will be as good only as the accuracy of the time tracking. Time should be carefully tracked by hours spent on original work and time spent on rework. Additionally, project team members should NOT include any time spent on maintenance tasks in their project hours.

 

Defects

A defect is a difference in the expected result from the achieved result. One way to measure quality is by capturing the number of defects. By tracking the defects through the project life cycle by both the phase when the defect was introduced and the phase when the defect was found, the impact of late detection on cost, time and productivity can be determined. The goal of the organization should be to identify and correct defects early in the life cycle.

 

Calendar Months

Project duration is determined by the number of elapsed months developing a project. Only active months should be included. An organization must determine the definition of the project “start” and “end” date. Typically the project start date is the date when the requirements phase is commenced, and the project end date is the date when the project is installed in production.

 

Dollars

The dollars spent on a project include project development staff salaries and overhead, any hardware or software purchased exclusively for the project and any outside expenses such as consulting and training.

 

Project Attributes

Unlike the above measures, project attributes are “soft data” that are collected on a project to help define the project characteristics so that like projects can be compared. At the minimum, you need to know if the project was a new development or an enhancement, if it was an “in-house” or vendor product, and the identity of the development and delivery platforms and source code languages used. As the metrics program matures, other attributes can further define the development process. You may want to track attributes regarding the tools used, development methodology, project team experience, consultant usage, review and inspection techniques, testing methods and so on. Attributes help identify the best processes and those that are detrimental to a project’s success. Your organization needs to determine those attributes applicable to your environment that are most important.

 

The Measurement Team

It is imperative that any metrics program has the full support of the organization’s executives and managers in order to succeed. Initially resistance might be encountered from the project teams, but this can be overcome with management backing. As project teams begin to see the benefits of the measurement program and what it means to them, the resistance will diminish. Project teams may actually begin to seek out measurement.

 

Another key to a successful measurement program is to carefully select and then train the personnel who will be doing the function point counts and collecting the metrics. Depending on the size of the organization and the amount of work to be done, these may be one person or different people. The critical factor, though, is to make sure that the person(s) selected has the right qualifications and aptitude for the job. Some organizations choose to have outside consultants who are Certified Function Point Specialists come in and accomplish their counts. If an organization chooses to have internal counters, they should select a person who has a background in systems development, possesses analytical and people skills and is willing and able to be a “cheerleader” for the metrics program. This last factor is extremely important in that the function point counter may have a lot of contact with the project team and thus can wield a positive influence. His or her attitude and belief in what they are doing can go a long way in ensuring the success of the metrics program. It is imperative that any function point counter receives the proper education and training for their job. Having the function point counter become certified by IFPUG will lend legitimacy to his or her qualifications. (See the IFPUG Website for more information on education, training and certification). It is also a good idea to have the function point counter mentored for a period of time by a Certified Function Point Specialist so that he or she becomes proficient at the job. One way to get to accomplish both the task of getting correct function point counts and to get new function point counters trained is to initially bring in an outside consultant to do the function point counts and simultaneously mentor the internal counter.

 

The metrics person will also need to possess analytical and people skills as they will be responsible for collecting other measurements, calculating and analyzing the metrics and meeting with project teams to discuss the results and arrive at conclusions. They must be able to help the project team determine success factors and potential process improvements for their project.

 

The project manager and the project team have the final role in the measurement process. To ensure success of a metrics program, all team members must be educated in the fundamentals of both the metrics program and the function point counting process. Taking the “mystery” out of the program goes a long way in winning over any holdouts. Additionally, team members should be reassured that the process, NOT the individual, is being measured and evaluated. Of course the “proof is in the pudding” is the best way to convince team members in the value of the program. Once they see the results of process improvements, they will be more willing participants in future measurements. In the end, the project team needs to see how the whole process benefits them.

 

Data Collection

Now that your organization has identified the measures to collect and who is going to collect them, a process must be set up for the actual collection of data. Remember, in the beginning keep it simple and minimize the time and effort involved.

 

To start, develop a simple data collection form that the project manager can use throughout the life cycle of the project. The form should include places for recording project name and type, project start and end dates, effort hours, costs and any project attributes that your organization has decided to collect. There should be a grid to collect defect data by phase introduced and phase discovered and a grid to record function points by phase. If your organization has decided to collect function points at the end of the requirements stage and when the project is installed in production, as well as every time the “scope” changes, there should be a place for the function point counter to record each count. You can also capture the start and end dates, effort and cost for each phase and scope change in order to evaluate the impact of each event on the project as a whole.

 

Data Analysis Process 

The project has been installed in production, and the metrics data has been collected. Now begins the most critical piece of the entire metrics process: data analysis. What good is a bunch of data if it is not analyzed for those good and bad aspects about the project?

There are several steps in this analysis process.

 


First, the metrics person must calculate all of the metrics (delivery rate, scope change, defect removal efficiency, etc.). This information should be entered in a chart for easy reference (see Chart 1 for an example). If you have a database of project metrics established, you might also include in the chart how the project did on each metric compared to similar projects within the organization. Or you may choose to compare the project with industry data (available through various sources; see the IFPUG Website for vendors).

Chart 1: Example of a Project Metrics Report

Next, you want to provide this information to the project manager and team and give them time to review and analyze the metrics. Then set up an appointment with the team to go over the results and most importantly, identify the “best practices” and the potential “process improvements”. At first the team will have trouble with this step, but patience and persistence will pay off. You can help by pointing out some weak areas and asking for the factors that may have contributed to the result. For example, if the productivity rate is much lower than similar projects, look at the possible reasons. Maybe the team was using a new tool and did not have adequate training before beginning the project, or maybe their time collection methodology was flawed. They may not have a way to accurately and consistently collect time, thus exposing a very much needed process improvement. Conversely, look at what went well with the project and identify any reasons for the success. You want to make the team aware that they should continue all best practices.

 

After meeting with the project team, the metrics person should enter the results in a report. The report should include a brief description of the project, the project results and the identified best practices and potential process improvements. This report should not only go to the project team involved but to all project teams and managers so that everyone in the organization can benefit from the lessons learned. Shared knowledge can help the organization as a whole meet their goals and become better at what they do.

 

 Organizational Metrics Reporting

The project metrics discussed in this article are all easily rolled up to produce metrics at the organizational level. Probably best done on a quarterly basis, results can be compared for the different metrics across time. As exhibited in Graph 1, creation of graphs for each metric for the preceding time periods will allow management and executives to see at a glance any organizational trends and alert them to areas that require further improvements.

 

 


Graph 1: Example of Partial Organizational Metrics Report

Tools

There are several tools available on the market today to assist you with your metrics program. There are IFPUG certified function point repository tools that perform the calculations involved in the counts as well as store the counts. They save considerable time when doing the function point count as well as provide an audit trail and a base from which to do future enhancement counts on an application. See the IFPUG Website for further information about these tools.

 

In addition, there are also tools available for estimating and assessing project attributes using industry data. Again, the IFPUG Website is an excellent starting point for information on vendors and their products.

 

At the beginning of the metrics program, an organization might want to create and use their own database to deposit the data collected. Initially, Excel spreadsheets work well for metric calculations and trend analysis. As the metrics program matures and the amount of data grows, however, this will become cumbersome. At that point an organization should look for more powerful tools to assist them.

 

Conclusion

A metrics program can be highly successful if implemented correctly. Starting small, having short-term goals, focusing on key measures and obtaining buy-in from everyone from the executives to the project team members are all important in ensuring a successful program. Results will not be evident overnight, but by following the guidelines in this article, your organization should start seeing favorable trends and realizing a pay-off on their investment in the metrics program within a short period of time.

 

REFERENCES

Garmus, David, and David Herron. Function Point Analysis: Measurement Practices for Successful Software Projects. Addison-Wesley, 2000.

The International Function Point Users Group. Guidelines to Software Measurement, Release 2. Princeton Junction, NJ: IFPUG Standards, 2004.

The International Function Point Users Group. Function Point Counting Practices Manual, Release 4.2. Princeton Junction, NJ: IFPUG Standards, 2004.

The International Function Point Users Group Website. www.IFPUG.org.