Sample Data for IS Metrics

Once an organization decides to implement a measurement program, the problem is usually deciding what not to measure, rather than deciding what to measure. Many organizations collect data for so many different metrics that participants in the measurement program become cynical and the effectiveness of the program is greatly reduced. In other organizations, the problem is to determine what kinds of data are available to be collected. Exhibit 2 lists categories of metrics for each perspective of the balanced scorecard framework, data for potential metrics in each category, and sample metrics. The exhibit is not comprehensive but rather an abbreviated list of possibilities for each perspective of the framework.

Exhibit 2. Metric Categories, Data, and Samples for the Four Perspectives of IS Performance Measurement

Category

Sample Data

Sample Metrics

Project Perspective

Financial, type and scope

Total estimated and actual time and estimated and actual cost per predefined project activity, type (e.g., development, maintenance), estimated and actual project function points

Cost per function point (FP)

Personnel

Experience level, experience type and education of personnel, years using a specific development environment, number of contractor personnel, number of employees

Productivity ratings: time/FP for different levels of experience and education

Methodology

Type(s) used, level of automation, testing techniques, number of models

The metrics for methodology are summarized for the entire software process rather than for a particular project

Interface

Number of meetings, meeting type and length, number of requirements and design

Percent of time in meetings and by meeting type

Category

Sample Data

Sample Metrics

changes, pages of documentation, hours of customer training

Product Perspective

Financial, type and scope

The same data and metrics used for a project are also applied to a single product; one project could result in many products, or it might take many projects to produce a single product; product measurements exist over the life of the product, whereas project metrics are closed out when the project is completed

Quality

Number of defects and errors, number of test cases, number of change requests, number of changes, amount of usage, complexity rating, number of reused modules, number of support calls

Number of defects/FP

Results

Business objectives translated into quantitative goals

Inventory percent level

Efficiency

Amount of memory, disk, processor cycles, response time, operator time

Average/peak response time

Process Perspective

Organization

Number of general meetings, type of meetings, communication methods, hours by activity; amount of office and desk space

Maturity level assessment

Personnel

Data in addition to that gathered for a given project includes vacation days taken, vacation days worked, number of working days, number of employees, number of contractors, number of training days

Metrics in addition to those listed for a given project

Methodology

Gathered by project

Overall productivity by methodology

Performance Perspective

Satisfaction

User satisfaction survey, number

Average time to

Category

Sample Data

Sample Metrics

of system users, number of workstations, number of reports generated, number of reports used, number of screens, number of screens viewed, number of help requests

respond to help requests

Integrity

Number of errors discovered after delivery, number of errors discovered within a selected period of time, error severity

Average number of errors (classified by severity) discovered after delivery

As the exhibit indicates, there are more types of data than there is time available to collect them. A measurement program would not be cost-effective if the data necessary to produce all interesting metrics was collected. The best method is to focus on basic metrics such as size, defects, effort, and user satisfaction before moving on to other metrics.

Under the G/Q/M approach, the choice of metrics follows the definition of goals and formulation of specific, answerable questions. To achieve a balanced scorecard of measurement, IS managers must ensure that the metrics selected span each of the four perspectives. If one perspective is not measured, a distorted picture of IS may be the result of the measurement program. Because the balanced scorecard framework is used as a guideline to pinpoint areas for control and subsequent improvement, the implications of such a distortion are enormous.

Project Management Made Easy

Project Management Made Easy

What you need to know about… Project Management Made Easy! Project management consists of more than just a large building project and can encompass small projects as well. No matter what the size of your project, you need to have some sort of project management. How you manage your project has everything to do with its outcome.

Get My Free Ebook


Post a comment