As we pursue a vision of excellence in public relations enabled by excellence in research, measurement, and evaluation, discussion often turns to the measurement outputs and outcomes. Several members of the Commission on Measurement and Evaluation engaged in an email discussion last May about measuring outputs and outcomes. I compiled the initial email exchange, and circulated the document for further comment.
The discussion expanded in so many interesting directions that I will not even going to attempt to summarize; you will be better off reading the entire thread. I think it comes to the following question: Which of the following should public relations measure: (a) outputs (b) outtakes (c) outcomes (d) business results (e) all of the above?
Read on and let us know what you think.
Forrest Anderson (Independent Communications Research and Strategy Consultant): I would argue it is preferable to measure outcomes rather than outputs, because business objectives are outcomes. Organizations do not employ internal or external communications to achieve outputs unless they lead to some kind of behavior – a positive vote, increased sales, reduced employee turnover, higher share price. What organizational leaders are interested and what communications disciplines should try to achieve are behavioral objectives or outcomes. So, when you can, that’s what you should measure. Outputs are only a proxy for the real thing – outcomes.
David Michaelson (David Michaelson and Company LLC): I think Forrest has the right answer here. Outputs are merely a mechanism for achieving outcomes. Public relations is not conducted to achieve placements. It is conducted to affect change. Outputs are only a tactic to get to the end result. Outputs are part of the process to reach an objective. They are not the objective itself.
Frank Walton (RF Binder): Yes – to all the comments from my colleagues. I’d provide a slightly different frame. I’m not sure that measuring outcomes is “better” than measuring outputs. But they are two different things – and reside on two different places in the organizational value chain. Measuring outputs is an Operations Metric; by measuring outputs you can demonstrate increased efficiency, show resource allocation, etc. Measuring outcomes, as my colleagues note, can document contribution toward meeting organizational objectives.
PR people are what they count.
PR people (individually and as a profession) need to ask what they’re doing when they go to work. (As Aristotle tells us, without categories, all is lost – so here goes:) Is PR (and all that PR people do) there to reduce friction in the overall system so that the “real value drivers” succeed? Or is PR one of the “real value drivers”? One is not base, and the other noble, but they’re not the same thing.
Fraser Likely (Likely Communication Strategies Ltd): I would argue that Outcomes are not more important. From a Chief Communication Officer (CCO) perspective, they are as important as two other measures.
1. Outputs and outtakes are important. They are important to the PR/Communication Department. They are measures of productivity, be it in-house or consultant. They also lead to measures of efficiency and cost-effectiveness (for example: comparing communication channel to communication channel; or communication activity/product to communication activity/product). Getting to an outcome requires knowledge of where to expend resources. These measures are important for the CCO to manage properly. But, no one else in the organization cares about these measures.
2. Outcomes are important, as Forrest and David have enunciated well. They are important for the PR/Communication Department to be able to show its contribution to the objectives of another department (HR for employee change; Marketing for customer change; Board for corporate reputation changes; etc.; etc.). These are contributions to business line objectives, not necessarily to corporate objectives. Achieving a measurable outcome from a planned communication program in support of another department’s strategy and objectives is important for the CCO to be able to demonstrate value in strategy execution. Only the immediate client department cares about these measures.
3. Outgrowths (or outflows as Ansgar Zerfass would call this third order of PR/Communication Department measurement) are important. They are important for the PR/Communication Department to be able to show its contribution to the strategic management of the organization itself (Board and Executive Committee for intended strategy formulation, strategic plan and emergent strategy formation; risk planning; large scale change management programs; issues/crisis management; etc.). These are contributions to the full strategic management process, including on-going decisions around corporate objective setting. They include for the CCO such leadership actions as direct advice, training, messaging, etc. They include for the PR/Communication Department the provision of service lines and service offerings, the organization of resources (make or buy) and the capabilities and competencies of staff (including consultants). Achieving these benefits (perceived and/or real) against cost (as a cost centre) are important for the CCO to be able to demonstrate value in corporate strategic management. The CEO, Board and Executive Committee all care about these measures.
Therefore, from the perspective of a CCO, these three levels of measurement are ALL important for the successful management of a PR/Communication Department.
It is not either or, it is all.
Mark Weiner (PRIME Research North America): In my experience, it’s almost impossible to separate outputs from outcomes if we try to make an ROI assessment: both are required. But in my opinion, Frank’s is the best response I’ve ever read to this age-old question. “Efficiency” is a key ROI metric and if outputs lead to outcomes, streamlining outputs leads to a better ROI on outcomes. So maybe outcomes are not more important than outputs, per se, but rather outcomes are more important than they are now in the minds of most PR people.
Katie Paine (KD Paine & Partners): Short answer is, that if you walk into an Executive Leadership Team and talk about Outputs, and the Social Media/Marketing/ or guy in the next office comes in and shows business outcomes, he/she will get a raise, and you’ll be lucky to keep your job.
Mark Weiner: But are most social media guys doing any better than anyone else? The ad guy, the marketing guy and most other guys cannot or do not make a case for outcomes on everything they do, and even if they do, it’s usually no better than the case made by PR.
Katie Paine: Actually, they are, but that’s primarily because they come from a different perspective. In Social Media the outcome is a registration, or a follower or something that shows engagement because primarily the desired outcome is an increase in to the marketable universe, and presumably if you take the time to register for something, or to follow someone on Twitter, you’re saying “yes, I’m interested enough to be marketed to” – The social media equivalent of Outputs – i.e. the number of Tweets sent of Facebook status updates are irrelevant unless there is a response.
David Geddes (InfoTrend): This raises the question of whether the social media people are getting an easy pass on what constitutes an outcome. Shouldn’t an outcome be something that drives business results? If I were a CMO, I would not give much credence to a count of registrations, Twitter followers, or anything similar until you can link this to desired business results. Read Avinash Kaushik on this subject.
Sean Williams (CommunicationAMMO): I’m not sure there is a question about that, David – I interpret Katie’s observation is that showing engagement is an outcome … a communication outcome that may or may not lead to business results, but that doesn’t necessarily matter.
Some of the “pop research” (done often by firms with a vested interest in positive results) argues that followers spend more than non-followers, or that companies higher on the social media engagement index have better business results than those lower. As many of you know, I’m quite skeptical about how generalizable those “findings” are…
Not so long ago, the demand from companies (such as Goodyear) was to show financial impact for social media – true ROI – but now it seems OK to go with more intermediate results: engaged followers, perceived value, participation, etc. Does that constitute some sort of “pass?” Maybe, but no easier a pass than mainstream media placements get – companies still think they’re important regardless of measurable impact on the business (I know of a few that I cannot name who’ve bailed on measuring media because the C-Suite doesn’t require it. It accepts on the face that media placements are good for the firm.)
David Geddes: My bias is to hold social measurement up to the standards – sometimes implicit and sometimes explicit – in the writing and blogging of Avinash Kaushik. He makes it very clear that there is a lot of junk measurement out there. Shouldn’t we set a high standard when it comes to measuring what “engagement” is? We should strive to get people to be, to borrow Avinash’s term, measurement ninjas. I forget his derogatory term for those who count things merely because they are able to count them.
Sean Williams: Point taken. The trick is that complying with a really high standard can be expensive (i.e., marketing mix modeling) – also, in social media it seems to be more of a marketing-led measurement: looking for captive audiences to sell to rather than develop strong relationships with…
Katie Paine: I disagree. Is PR responsible for selling cars for GM or getting people to consider knocking on the showroom door? For the social media folks I’m working with, some can track results directly to online sales, but many, who don’t sell on line or require a more consultative sell, consider getting a registration or a follow or even a like shows both awareness and potential consideration. They then turn that “lead” over to the online marketing people who complete the sale.
Brad Rawlins (Brigham Young University): I believe that there are three potential outcomes from public relations efforts: cognitive (awareness, understanding, remembering), affective (attitudes, opinions, feelings) and conative or behavioral. You can’t jump straight to behavioral without measuring the cognitive and affective indicators that would predict some behavior. So, I believe measuring outcomes includes these indicators as well as behavioral.
I’ll give you a couple of examples. A while ago I helped a large regional bank conduct a communications audit. I talked them into including a cultural audit. The results indicated that their channels were pretty effective in getting their message out (outputs), and that a large majority were aware of their key messages (the customer comes first), which is where we would have stopped with the communication audit. But the cultural audit gave us evidence that the employees didn’t believe any of the key messages because the bank had demonstrated its values through rewards that reinforced a strong sales culture, and no-one really believed you would get ahead in the bank by catering to what customers really needed rather than selling them as many bank services as possible.
So, including measurement of affective variables with the cognitive variables informed us about the behavioral variables. They were aware and understood the key messages, but didn’t believe them and weren’t going to behave accordingly.
Another example is a recent survey that I helped conduct with pipeline companies. They are required to survey people who live next to pipelines to make sure they are aware of the risks and how to report pipeline failure. We replicated the section that measured how they get informed and the level of their awareness and understanding, but we added in questions of trust to see whether their efforts to inform had a positive impact on their relationship with these communities (it did). It was an easy way to show how output and cognitive measures could contribute to an important nonfinancial outcome such as trust.
Forrest Anderson: Brad, really good comment!
Mike Daniels: I’d second Forrest’s approval of Brad’s comment – it’s a really very helpful categorization for framing client conversations (and expectations) around optimizing measurement programs to meet specific business/organization goals. With your permission, Brad, I would like to reference this structure in the intro to a workshop on using the AMEC Valid Metrics outputs to outcomes grids that Ruth and I are leading in Lisbon.
I also think Frank’s defense of the importance of outputs in measuring operational efficiency shouldn’t be underestimated, at least from a tactical perspective. It’s often the first (and very visible) win from an outputs measurement program, and can have significant business outcome impact in reducing costs and enhancing productivity.
In the end, though, business requirements self evidently differ from organization to organization, and over time. Just as advertising can be focused at generating non-sales outcomes, depending on where the business stands and the challenges it faces at a given time, so PR and communications will have different priorities for outcomes and output measurement from time to time. Measurement programs need to be flexible enough to accommodate these real world dynamics…
Bert Regeer (Royal Dutch Shell): In line with Fraser, this is what happens with us in practice. Our annual communications program has a series of campaigns, driving strategic messaging (fuelling the conversations around our company positively) and mitigating issues (reducing the critical and negative conversations around our company). If we are successful, we will see that behaviors of key influencers, customers will change (outcomes)…and that is our part of creating business value.
But before we reach that stage we want to see whether our message are actually being picked up (so media analysis looking at outputs and NGOs reducing energy levels around us or the industry as a whole) and whether we see that opinions among what we call special publics are changing – and that is the measure whether our efforts have an impact on outtakes…
In other words, all levels are useful – the ultimate impact is in the outcome, but we’ll need the other stages as well to check whether our efforts are picked up and resonate…
Sean Williams: Just to build a bit on Bert’s and Fraser’s excellent comments: The CEO recognizes that there are many inputs to both revenue and expense, and doesn’t expect PR (for the most part) to isolate its contributions (though he/she does expect us to contribute…). It’s therefore reasonable to create objectives for outputs and outtakes that are within our direct control. Thus, our performance against those objectives becomes very important and worthy of measurement/evaluation. If we have the resources to expend to determine our precise contribution (the business results/outcomes of that activity), good on us!
Julie O’Neil (Texas Christian University): Outcomes are preferable to outputs. But, as many have articulated, outcomes (i.e., increase in sales, donations, votes, attitude changes, etc.) can be long-term in nature. Outputs tend to be short-term snapshots of public relations efficiency and effectiveness in reaching the desired outcomes. For example, I recently worked with a university that wants to increase its academic reputation among key groups. To accomplish this long-term objective, the public relations department first altered its messaging strategy and placement. I helped the university conduct an audit to discern the presence and tonality of these key academic messages. That’s step one: the outputs. This audit helped the department alter its messaging strategy for the second half of the year. The next step will be to link these outputs via correlation with longer-term outgrowths or outcomes. Did people process those academic messages? Do they believe them? More importantly, will the university receive applications from a higher-caliber student and change opinions regarding its academic stature among key constituencies (outcomes) in the upcoming year?
Forrest Anderson: Rereading the exchange reminds me of why I enjoy being a member of the Commission. Where else can you find discussions like this?
I started with the point of view that outcomes were the most important to measure, but Frank’s and Fraser’s comments quickly turned me to their position that other metrics are important as well, because they help us evaluate the success of the process intended to lead to the outcome.
Then Brad came in with his results from adding culture questions to a communications audit and finding misalignment between what the organization was communicating and what it was doing. This brought a whole new dimension to the discussion because not only was it useful in terms of managing the organization, but it was an unexpected discovery. To me this suggests that not only are the measures important but also the context in which we measure. This, perhaps, falls into Fraser’s third category of “Outgrowths.”
Fraser Likely: I believe that there is a consensus from the first round, or close to a consensus, that measuring outputs is indeed important.
Here, I would also include measuring the effectiveness of different media channels in distributing the communication product that would become the communication Output. I’d also include Outtake measures here, which are once the Output has been received, was it paid attention to, was the message in the Output understood and did the receiver act on the receipt of the Output (did they seek out other information; did they share the message; did they become engaged with the message).
These are all communication production and distribution effectiveness measures. They are important because they are milestones that indicate the potential effectiveness of a more lengthy communication campaign. They also are important because they can lead to cost-effectiveness, efficiency and ultimately productivity measures.
Certainly, the CCO needs to know how a campaign is going (are the right mix of channels being employed; are the message source and message appropriate; are intermediaries mediating our message in the right way). The CCO also needs to know where and how to spend operational monies and thus be able to manage resources.
I believe we should be talking about Communication Effects here rather than using the term outcomes. Brad perfectly outlined the three categories of communication effects from communication campaigns. Outcomes has come out of the Program Evaluation field. Communication campaigns support revenue generating, expenditure reduction or risk reduction programs. Typically, these programs are run by other functions; Marketing; HR; IR; etc. We support these programs with relationship building and communication campaigns or projects. It is these programs conceived by other functions that produce outcomes: short-term; intermediary; and long-term. It is from these outcomes that a ROI measure can be made. A ROI measure can not be made on communication effects (communication outcomes if you will). This is in line with what David Geddes said in round one: ” ” … shouldn’t an outcome be something that drives business results.”
I believe that measuring communication OUTPUTS (with channels and Outtakes) and communication EFFECTS/OUTCOMES is not enough. Primarily, Output/channel/Outtake measures have importance only within the PR/C department itself. Communication Effects/Outcomes are important to the PR/C department and to the department’s internal client (Marketing; HR; etc.). The most important measure, though, is the quality of the learning that comes from the communication and relationship building operational and campaign work. That learning includes: intelligence; analysis; scenario building; strategy; advice; and counsel. The greatest measure of the value of the PR/C department and of the Chief Communication Officer is the quality of the intelligence; analysis; scenario building; strategy; advice; and counsel proferred as part of the organization’s strategic management processes: strategy formulation; strategy execution; and emergent strategy formation. This is the measure that has the most value to the management team and Board.
In conclusion, I believe all three levels of measures are important. But, the most important is the ability to measure the third and show results. Of course, there aren’t results at the third level unless there are results at the first two levels.
2 thoughts on “Setting Priorities in Measurement”
I concur with Mark; thanks David for generating this discussion and allowing us to see the different perspectives.
Thanks to David Geddes for posting an to all participants for generating a lively and provocative discussion.
As a long-time member of the Institute for Public Relations and as a participant in the posted debate, I am reminded of the importance of the Institute as a source not of “dictated standards” (like the AMA or ABA) but for fostering intellectual and practical debate to help clarify the reasoning and ramifications for professional communicators to make more informed decisions in the areas of research and evaluation.
When even an astute collection of research mavens find it difficult to agree, we are reminded that there is a “science of PR” through which we foster a dynamic process of hypothesizing, testing and validation. And the profession is all the better for it.