So you'd think that by now, Harrah's has pretty much nailed which IT metrics it needs to track, right? Think again. "I'm literally holding in my hand a revised list of metrics and scorecards for this year, and I continue to nosh through this with my team, year after year," says Stanley, senior vice president and CIO at Harrah's.
Changes in the business and competitive landscapes force Harrah's IT department to regularly reassesses which key performance indicators to track. "We're constantly asking ourselves whether we're measuring the right things," says Stanley.
He isn't alone. IT leaders at companies such as DHL Express, Cendant Hotel Group and Kaiser Permanente say they continually wrestle with what they're measuring to gauge the effectiveness of their IT organizations. But by focusing on what business leaders are looking for, they're getting a lot more sophisticated at what to measure -- and what not to.
Beyond rote
Best-in-class IT organizations have gone beyond rote metrics such as system uptime and help desk problem resolution. Some IT shops on Wall Street, for example, are studying things like how many stock trades or customers they're able to support for each IT dollar they spend, says Howard Rubin, an analyst at Gartner Inc.
But these kinds of metrics are much more difficult to use than the traditional IT time and transaction measures. One of the big challenges that many IT leaders face in developing effective metrics, says Rubin, is determining which activities are under the auspices of the IT department. For instance, some user departments at financial services companies have their own LAN administrators who are outside the purview of the IT organization, says Rubin.
"There's IT everywhere, not just in the IT department but in bank branches and on the plant floor," he says. The business activity supported by IT "drifts back and forth like the eye on the head of a flounder," he adds.
Metrics get even more complicated as IT organizations move to service-oriented architectures, explains Stanley. "In a classical IT world, you measure a particular application," he says. "But when these systems are all highly integrated, what do you actually track? The platform? The services?"
Among its current measurements, Harrah's IT department is gauging how much investments in IT and business projects are helping the hospitality and gaming company to meet its business objectives.
"So let's say we're targeting 15 percent earnings-per-share growth and tracking that to the right IT projects, customer satisfaction with those projects, etc.," says Stanley. "Over the past three or four years, I can actually show the return on invested capital to the business results and sales growth."
Some companies are also tracking how effectively technology is supporting external customers. At DHL Express, the IT organization monitors customers' ability to use the company's Web site to track and trace package shipments, says Simon Bentley, interim CIO of U.S. and Canada DHL Express business IT in Scottsdale, Ariz.
And that's just the beginning. "In terms of IT operations, we measure just about anything you ever heard of," says Bentley.
Bucking tradition
Still, many IT organizations continue to struggle with mapping their performance against business outcomes. "IT has traditionally measured itself in very technical terms that don't mean much to people outside of IT," says Craig Symons, an analyst at Forrester Research Inc. These include statistics about availability and costs per megabyte of computing.
Business leaders agree. "There's been a huge disconnect between what business people care about -- i.e., capabilities delivered -- and what IT people care about," such as the percentage of projects that are being delivered on time and within budget, says Christopher Lochhead, chief marketing officer at Mercury Interactive Corp., a Mountain View, Calif.-based provider of business software.
But even conventional technical measurements often have a direct bearing on business outcomes. "If our systems aren't up, we aren't generating revenue," says Pete Gibson, vice president of IT infrastructure and operations at Phoenix-based Cendant Hotel Group, which operates 6,400 hotels.
It's up to IT leaders to translate those metrics into tangible business terms. "Business and IT people alike have a hard time making percentages feel real," says Stanley. "You don't feel a percentage of systems uptime. You feel the business or customer impact of an outage."
To address that, Harrah's is creating new metrics that will quantify the negative effect that a system outage has on revenue or other aspects of the company's business, says Stanley.
As companies have become more sophisticated about which metrics to evaluate, they've also determined which ones to drop. For example, until a few years ago, DHL Express' IT organization used to track the number of e-mail messages that employees were receiving to determine if there was a link between developing effective ideas and communication. "It turned out to be a pointless exercise," says Bentley. "We're not afraid to abandon metrics."
That flexible attitude has also led the IT group at DHL Express to experiment with more sophisticated measurements. Instead of just assessing the number of IT projects that are delivered on time, within budget and within scope, IT also measures what the business cares most about: the percentage of projects that have met their business objectives, says Bentley.
IT executives have also been able to use metrics to determine which IT projects to shelve. For instance, Gibson and other IT and business executives at Cendant recently put the kibosh on a three-month-old e-commerce project after metrics indicated that it wasn't going to deliver a substantial enough return on investment.
The business view
For all the emphasis these IT leaders are placing on IT metrics, there's a lot of variation in how often senior business leaders want to review them. At Harrah's, Stanley and his team use their metrics to put together a quarterly operating report with project updates and IT spending summaries that senior executives can dive into.
At Hannaford Bros. Co., interest among senior executives in IT metrics depends on the project being measured, says CIO Bill Homa. For example, top managers at the Scarborough, Maine-based grocer are closely tracking the rollout of a task management system to its 140 stores that's intended to provide managers with a clear message about which operations to focus on each day, says Homa.
At Kaiser Permanente, metrics focus on how the IT infrastructure relates to business costs. The health care organization's IT group, called KP-IT, has created a set of "cost pools" within its data center services. If one of the health care units is running a call center application, for example, KP-IT can measure the percentage of a server's load that's being consumed by the application and the costs that are placed on that business unit as a result, says Michael Blake, vice president and chief financial officer at KP-IT in Oakland, Calif. "We have much more educated conversations with business leaders than we had before," says Blake.
"There's much better delineation [of those costs] than there was two years ago," Blake adds. "So instead of saying, 'I can't understand it,' you can get to the IT price, and the business can figure out how to control the price."
Aidan Collins, a partner at PricewaterhouseCoopers who works with KP-IT, calls the group's approach "an IT view of the world with financial transparency so you can talk with business executives about true life-cycle costs and true accountability within IT services."
Like Harrah's Stanley, Bentley says the metrics that the DHL Express IT organization tracks continue to evolve and mature over time. "When you start off, you're measuring the easy things," says Bentley. "Then you start linking to IT's impact on the business."
Homa puts a more personal spin on it: "The best metric is that I'm still here after 10 years."
Sidebar: Metrics to ponder
Want to move beyond traditional IT measures that don't mean much to your business partners? Here are some sample metrics from Gartner analyst Howard Rubin.
Current state
-- IT spending as a percentage of revenue n IT spending as a percentage of operating expense
-- IT head count as a percentage of total head count
Agility
-- Ratio of percentage change in revenue to percentage change in IT spending, year over year
-- Ratio of percentage change in operating expense to percentage change in IT spending, year over year
-- Ratio of percentage change in business head count to percentage change in IT head count, year over year
Portfolio
Percentage of IT spending to:
-- Run the business
-- Grow the business
-- Transform the business (innovation)
Service delivery
-- Percentage of end-to-end commitments to users met
-- Business cost of failure to meet end-to-end commitments
Customer
Customer satisfaction with:
-- Operations
-- Development and support processes
-- Personnel
Employees
-- Employee satisfaction
-- Employee/key employee turnover
-- Compensation costs
-- Percentage of employees on-/offshore
-- Percentage of contractors on-/offshore
Infrastructure
-- Unit costs of all systems
-- Market basket cost (unit cost for each platform multiplied by platform volume)
-- Reliability/availability end to end and by system
Risk
-- Impact of virus
-- Impact of security breach
-- Impact of compliance failure