Challenge
This project demonstrated how important a complete benchmarking system can be to even the best performing companies, including software developers. Too often, companies base their assessments of performance on how close they come to meeting the product goals set forth in the original plan with no full understanding of the ways in which they could improve efficiency and reduce or eliminate costly corrections. Extending internal benchmarking to comparative analysis through involvement of peer companies serves to ensure an even more rigorous adherence to excellence in program management.
TMNG Global, with a professional team that has spent years in management roles with the types of companies comprising its client base, knows how to go beyond the sterile confines of number crunching to judge the nuances that make a real difference in project management performance. It’s this business sense underlying application of benchmarking tools that translates good practices into real-world success.
A large supplier of software-based services to the telecommunications industry required expert assistance to develop a comprehensive approach to evaluating and strengthening its software lifecycle management process.
This is a company with an unchallenged market leadership position which has enjoyed ongoing success with each new product release. Yet management believed the company could do even better, and further enhance its status with customers, if it had a more thorough way of assessing its weaknesses and strengths through all aspects of product development, release and price management. Their goal was to establish technology and service level standards equal to or greater than those of their competitors.
To achieve this, the company asked TMNG Global to apply its proven system for benchmarking the performance of software program management through the entire software life cycle. TMNG was to use this benchmarking system to evaluate the client’s program management processes and, in the second phase of the project, the team would expand the evaluation to include comparison of the client company’s program with those of other software developers.
TMNG Global Solution
To meet the client’s objectives, the TMNG Global team drew on its deep experience in benchmarking and comparative analysis across a vast range of project management categories. Through this experience, the company has compiled an extensive baseline of best practice guidelines and metrics to be used in project assessments, known as Quality Business Controls (TMNG QBC™), which it employs in conjunction with clients’ in-house programs to create the best possible benchmarking capability specific to the business at hand. The team also employed the Quality Systems Audit (TMNG QSA™) to facilitate the comparative analysis of performance in several areas.
Program management at the client company includes process management, customer deployment and communication, industry testing, product rollout and oversight of outcomes within these areas, including setting performance targets and measuring actual performance against these targets or metrics. In calibrating the benchmarking system to this application, the team had to identify and assign value to best practices against which the practices of the client could be measured.
Setting benchmarks also required creating a system of metrics suitable for measuring how these functions are applied across the following phases of the company’s product development cycle:
- The scoping of product requirements, including users’ roles in identifying needed enhancements and their communications of those requirements to the company;
- The initial design phase where IT staff begins to translate the requirements into a new product design;
- Design approval from company management and users prior to actual coding of the design into a product;
- Internal testing with respect to each module, how modules integrate and how the entire system operates under stress;
- External testing by participating users;
- Final testing sign-offs and migration of the software to product release.
Beyond identifying and setting metrics for each area of program management activity, the team had to identify and measure how activity is coordinated across all groups and establish the criteria that define project success.
For Phase 2 of the project, the TMNG Global team compared the performance of the client’s program management processes with those of several other software producers.
After assessing the client company’s performance, the team identified performance gaps and prioritized best practices into actionable plans that would have the greatest impact on the client’s business. The challenge here was to weigh relevant values of various practices and gaps so that the performance assessment was optimized to the real-world dynamics of the client’s business concurrent with adding real value in the customers eyes.
In this case, to compile the data required for analysis, the TMNG Global team undertook a detailed review of program and release management system flows and operations documentation provided by the client and conducted a series of interviews with the client’s program management and release management teams.
In developing the benchmarking framework, we drew on the client’s program management requirements and metrics based on parameters set by the International Standards Organization. The team utilized widely accepted best-practice guidelines, as formulated in the Project Management Institute’s Project Management Book of Knowledge (PMBOK), as a starting point for comparative analysis and leveraged its understanding of the client’s business along with the methodology and metrics of QBC and QSA to create a best-practices framework that precisely mapped to the requirements that are most relevant to successful program management.
Through its long experience helping clients ascertain and apply best practices in a variety of operational environments TMNG Global understands certain requirements must be met regardless of the specific application. These requirements, which informed all aspects of the evaluation process in the current case, include ensuring that:
- The project team understands and follows an appropriate lifecycle.
- Customers, users, and all important stakeholders are involved.
- Good communication occurs during the process of deriving requirements.
- Difficulties that stem from the inherent complexity of the product or the computing environment are addressed early on.
- A list of requirements is maintained in a database where they are more easily managed.
- A requirements configuration management process is implemented.
- Requirements are not added or changed without (1) conducting a risk analysis to determine impact on project plans, and (2) re-estimating the cost and schedule of the project.
- Some operational capabilities are delivered during the development process with the intention to add the remaining or new requirements.
An added dimension of complexity in this case entailed the role of an industry organization as a point of oversight and evaluation of the client’s product releases. The benchmarking process had to take into account priorities and performance metrics established at the industry organization level. And the effectiveness of the client’s communications and execution of requirements at this level had to be included in the benchmarking.
Setting Metrics
To develop a benchmarking system that covers all areas pertinent to effective program management performance requires a system of metrics that accounts for all aspects of the program management processes, the dynamics in the progress of a specific project and the actual effectiveness of the product itself. Typical metrics categories in each of these areas will address the actual versus planned schedule, actual versus planned budget and delivered versus planned functionalities.
The task of establishing the metrics to be used in all these measurements requires deep experience with various approaches to assigning value to raw data in specific applications categories. The team chose appropriate options from the base of metrics it was accustomed to collecting and evaluating and augmented those with metrics specific to the client’s business.
Getting the metrics right is crucial to efficient evaluation. Frequently companies compile immense amounts of data respecting performance across many functions but then wrestle with trying to evaluate the data because they have not established a uniform system for assigning value to the raw information. For example, in the case of software releases, one must measure the impact on overall effectiveness caused by software defects, as reflected in levels of severity by phase and by release. The negative impact of those defects must be tabulated in terms of the volume of lines of code associated with the defects and the severity of the defects relative to overall product performance.
Being able to measure the effectiveness of testing programs is another area where taking account of the nuances is essential to useful benchmarking. What percentage of tests was executed in a given phase of product development? What was the level of user participation in the testing process? An efficient measurement system must capture the whole picture at a level of granularity that provides insight into where improvements can be made.
Risk management, too, is a tricky area of oversight that requires sound measurement. In this case, TMNG helped the client develop specific risk management tools as a foundation to ensuring a systematic approach to risk management.
The Evaluation Process
With the metrics defined, TMNG developed a survey covering nine areas of project management to be used in the internal and external benchmark evaluations. The areas included project scope, quality, resource management, schedule, and risk.
Once approved, the newly-calibrated benchmarking system was used to evaluate and recommend refinements in the client’s program management processes. The client’s performance in each area of program management was evaluated against industry practices as prescribed in the PMBOK on a scale of one to five and weighted according to that area’s importance relative to other areas reviewed. The evaluation and recommendations were understood to be subject to further refinements upon completion of the Phase 2 survey of other software companies. Final recommendations were prioritized based on the estimated benefits to the effectiveness of the client’s operations.
External Survey
To perform the external survey requirements of Phase 2, TMNG had to identify and secure cooperation from a well-balanced selection of telecommunications, information technology providers and high-volume transaction institutions with track records that displayed a commitment to process and quality principles and measurements, and that utilized these tools to manage the health of their businesses. In exchange for sharing information about their environment, SLAs and actual performance, survey participants were provided with a high-level report of benchmarks among the participating companies without exposing the names of the participants.
TMNG was able to complete the external benchmarking process with 10 companies, each of which had project management/release management requirements that were applied against high transaction volumes in consumer and business markets in highly competitive, leading-edge environments with regulated and customer-visibility characteristics similar to those of the client company. The frequencies and sizes of these companies’ software product releases were analogous to the client’s.
The team prepared the participant survey based on the findings and recommendations from Phase 1 internal benchmarking. The survey was crafted to provide not only best-in-class practices from all participating companies, but also to provide sufficient participant background information to determine how closely the participants’ operations mirrored those of the client’s project and release management operations.
TMNG used the results from these surveys to further refine the metrics and underlying best-practice and key-enabler parameters of the benchmark system. The company then assessed the client’s performance against the performances of the survey participants. This allowed the team to identify and prioritize the gaps in the client’s performance for purposes of developing actionable plans to achieve the highest business impact on the client’s program management processes.
Benefits to the Client
These results and recommendations were exposed to the industry organization as well as to the client for approval. The willingness of the client to undertake such a thoroughgoing analysis of its program management performance, knowing the results would be relayed to customers was an unusually bold move, especially for a firm which was under no pressure to engage in such intensive self scrutiny. But the experience demonstrated the wisdom of such an effort, since it served not only to identify weak spots in the internal management processes but also to enhance the firm’s reputation for honesty and excellence among its customers.
As it turned out, the client’s performance was generally outstanding across the nine areas covered in the benchmarking survey, but TMNG found there was room for improvement in two important areas – risk management and project integration management. Risk management includes the processes concerned with conducting risk management planning, identification, analysis, responses and monitoring and control over a project.
Project integration management includes the processes and activities needed to identify, define, unify and coordinate the various processes and project management activities within the project management process groups.
With regard to risk management, the client’s ability to structure projects so as to minimize risks was heavily based on the institutional memory of key staff people and their ability to apply lessons learned from previous projects. TMNG recommended creation of a risk-management regime that relies on use of standardized quantitative and qualitative risk management tools and techniques to identify and generate a record of potential areas of concern within a new project based on previous experience and general industry benchmarks.
As for project integration management, the client lacked an enterprise environmental system that could readily integrate work authorizations, risk tolerances, resource allocation, etc. into a single repository. TMNG recommended a solution that would provide senior management an immediately accessible overview of the time-sensitive performance aspects of any given project as well as a better understanding of how company resources are being utilized across different projects at any given time.