This article was originally published on CFO.com.

The results can be meaningless or misleading — or both.

Two years ago, the chief information officer of a Fortune 20 company whom I have known for many years asked me to take a look at benchmarking his IT organization against its peers.

On the surface, this was a reasonable request. It was the start of a new long-term strategic planning and budgeting cycle. The board and the executive leadership team wanted to know how well every part of the corporate structure was doing comparatively as well as absolutely. The CIO wanted to know how well he was doing against some of his key performance indicators and if there were any areas in which he was significantly behind the competition on policy, practices and performance.

However, in my view, there were two major potential issues with the request. First, just who were his peers? His company was in an industry where there are only a dozen or so global players, only a handful of whom were really of comparable size (and just as importantly, organizational “shape”). Roughly half of them ran their IT largely in-house; the other half were largely outsourced, so valid analysis would be difficult and complex — and meaningful comparisons could be hard to make. Even if we were to drop “industry” as a selection criterion, there were only a few companies as large as his, and they were extremely diverse, making useful comparison even harder.

Actually I knew exactly who the CIO wanted to include in the benchmark study (because he told me), but with the resulting small sample size came the second problem involving confidentiality of data and results. In benchmarking, if the resulting sample is too small, it’s easy to use publicly available information to “decode” supposedly anonymous benchmarking information and get straight to a ranked list of competitors’ performance. That’s why getting other companies to participate in benchmarking efforts is generally difficult, and many companies rely on a few large and diverse benchmarking databases in which they can hide individual results.

The issues of finding peer companies and keeping data confidential, plus the cost of collecting and analyzing the information required, make benchmarking the largest organizations extremely challenging. And all too often the results can be meaningless or misleading — or both.

A few years ago I worked on a major reorganization of an IT group that had had very good comparative benchmarking results for many years. According to the company’s series of annual benchmarks, it was in the lead in its industry and among the best performers in any industry. Yet when we were done with a yearlong reorganization — which involved standardization, the introduction of new data-driven operational policies and practices, as well as a new organization structure — the company had succeeded in reducing headcount from around 700 to just over 250 without loss of delivery volume or quality.

That result put into question the company’s years of benchmark comparisons and “excellent” performance ratings: Either everyone in the benchmark had the same set of chronic problems, or there was something wrong with the measurement and analysis process.

Back to the the CIO who wanted to benchmark his IT organization. A Fortune 20 IT organization is generally pretty large: most IT budgets at this level would be close to the revenue of a Fortune 1,000 company and a few might rival a Fortune 500. In general, these IT organizations are also pretty diverse, thanks to the necessity of serving all the different areas of a Fortune 20 business. In many cases such large IT organizations are organized like a collection of smaller IT groups, usually along functional or (increasingly) core business process lines, with or without corporate shared services. It’s not unusual to have a dozen or more divisional CIOs reporting to the corporate IT leader.

With this is mind, I suggested to my CIO acquaintance a slightly different, three-part approach to benchmarking.

First, we would do a quick, mostly qualitative, benchmark on about 20 global organizations selected solely for comparable size and shape, using publicly available information and extracting any quantitative information available from industry and financial analysts. We would use this to build a “positioning map” to give the CIO a sense of who among the group was most like his organization and who was least like it. The “map” would have several dimensions, but we would not guarantee that it would serve up any specific or useful quantitative comparisons. Instead, it would let the CIO decide who, if anyone, he wanted to know more about.

In particular, it would let him and his team and the business executives who were his peers think strategically about the role of the IT organization and how to organize IT resources. Were there “out of the box” practices from companies that were radically different from his own business from which he could learn, but where adoption would require a strategic shift? This could have had a significant impact on his strategic planning process, but not on immediate day-to-day operations.

Second, we would look at about 50 product and service organizations (in the technology products and services arena) with revenues about the size of his IT budget. Here we would go for a more quantitative analysis and some specific rankings. We would, in essence, be benchmarking his IT organization against the best companies that operated at the scale of his organization and did comparable kinds of work. These results would let him know more about how the broader market valued specific aspects of performance in technology services and give him some additional basis for internal return-on-investment decisions and perhaps suggest some new performance measures to drive strategic changes.

Third, we would benchmark each internal part of his IT organization against the others(his company’s IT organization was organized along business process lines, with a shared-services model for infrastructure management), taking care to compare like with like. These results would tell him if he had any major differences in his internal performance, which were his best and less-than-best performers, and how much variability his IT customers were experiencing. We would also be able to give him an idea about how much better his organization’s overall performance would be if he focused on improving the bottom quartile of his teams — using the practices and experience of his top quartile.

The CIO liked all three parts of the proposal, especially the last, because the company had never done a formal “internal” comparative benchmark before. He saw immediately that the company could link the benchmark to the company’s balanced scorecard process and to his managers’ individual KPI set, both of which the company had been struggling to implement effectively.

If properly designed, the benchmark could be used periodically to monitor the effects of process improvement programs, such as ITIL (Information Technology Infrastructure Library) and Lean Six Sigma (he was working on initiatives for both). And because the various groups being benchmarked were all within a single organizational context (although distributed geographically and distinguished to some extent by different local management styles) many of the external variables that make comparisons among separate corporations difficult or expensive would not be factors, or could be more easily controlled for.

The two external benchmarks took about two months, running in parallel, and did in fact yield some interesting insights that were worked into a new strategic plan. The internal effort required an additional six weeks, because the basic information systems to support the analysis had to be established and some data rationalization was needed. Subsequent internal benchmarks will be much faster (about a week for the analysis and report generation) and will be run once a quarter. Fortunately, most of the basic data we needed was already in place, courtesy of the ITIL initiatives that the company had been working on for some time.

I’ve gotten good results from this internal comparison process over the past 20 years and I’m always surprised that more large organizations don’t use it routinely, though some companies do (GE comes immediately to mind). It can have implementation pitfalls if some of the IT divisions being measured feel they are being unfairly compared, or if the bottom quartile is immediately penalized rather than encouraged to improve. So internal benchmarks, also, have to be carefully designed, well implemented, and effectively communicated.

Do all that, however, and you may get a much better return on investment than from an external comparison, which can obscure more than it illuminates.

About the Author
John Parkinson

John Parkinson is an Affiliate Partner at Waterstone. John brings extensive experience to the topics of technology strategy, architecture and execution having served in both senior operating and advisory roles.