Last week I made a case for the need for an action-oriented set of priorities for the public relations research, measurement, and evaluation function. Our goal at the managerial level, is to quantify public relations’ contribution to meeting organizational goals and building organizational value. To achieve that goal, we – research measurement, and evaluations professionals – need to attend to issues of a measurement framework, standards, and data quality.
This week I address three the first related topics: a measurement framework, metrics, and standards.
1. A measurement framework.
First, we must adopt a common measurement framework based communications theory and public relations theory. Note that I do not say a single measurement framework. A starting point is a model based on activities, outputs, engagement, outtakes, outcomes shown in the diagram below (for additional discussion see Michaelson and Stacks 2010). These models have their origins in information theory (Shannon and Weaver 1949, Berlo 1960). Any similar model is suitable provided that it cover the panorama from activities to business outputs. Feel free to adapt this model to your needs if it proves useful.
All public measurement programs – including those that are limited to measuring outputs – should be expressed in this broader framework. Let’s expand the perspectives of our clients, internal or external. Many are using this model, or something close, already. Failure to address all phases of the communications process opens the door to output-centric measurement programs, which are long distance from organizational results.
Why is this essential? First, this framework aligns our work as public relations professionals with organizational goals and building value. Second, this framework builds on foundations in communications theory, information theory, cognitive psychology, and the social sciences more broadly. Third, this model aligns with models used in sales, marketing, and advertising. Finally, failure to address all phases of the communications process opens the door to output-centric measurement programs, which are long distance from organizational results.
2. Metrics.
Associated with the measurement and evaluation framework, we need a set of standard and recommended metrics adapted to stakeholder audiences and public relations goals for to these audiences. But not just any metrics suffice; we need metrics that have what a statistician calls reliability and validity.
Validity is not a term that can be loosely applied to any set of metrics. “Validity refers to whether a measure is actually measuring what you defined it to measure” (Stacks, 2002, 130). Today, we are a low and qualitative stage of validity – content validity – a “measurement validity that is based on other researchers or experts evaluations of the measurement items contained in a measure” (Stacks, 2006). In other words, everyone is taking an educated guess at the appropriate metrics. This applies, incidentally, to the recently-developed AMEC Valid Metrics, reasonable as they seem.
A measure cannot be valid without being reliable. “Reliability is the ability of a measure to measure the same thing comparably over time. A reliable measure is one that is stable – it does not fluctuate without reason” (Stacks, 2002, 131). Our field lacks reliability measures for many types of public relations measurement, particularly media measurement (traditional and social alike).
As part of our research and measurement agenda, we need to advance towards the more rigorous standard of construct validity, “a statistically tested form of measurement validity that seeks to establish the dimensionality of a measure” (Stacks, 2006). Validity needs to be demonstrated; it is not just a label.
3. Standards.
The public relations field needs to develop standards for its research, measurement, and evaluation function. We have an important resource in a paper “Standardization in Public Relations Measurement and Evaluation” and a summary slide deck by Institute Research Fellows David Michaelson and Don Stacks thoroughly treating the topic. Why are standards essential? To cite David Michaelson:
With a standardized system of comparative evaluation, public relations professionals will be able to gauge the absolute performance of specific programs and specific program elements. In addition it will allow the comparative performance of prior and competitive programs. And, finally, allow us to compare that performance within industry and category, as well as the performance of the program relative to other industries or categories.
Quite practically, standards will save us time, effort, and allow us to deliver better research, measurement, and evaluation.
What standards do we need? How should these standards be developed? The International Standards Organization delineates the standards development process as follows:
- Consensus. The views of all interests are taken into account: manufacturers, vendors and users, consumer groups, testing laboratories, governments, engineering professions and research organizations.
- Industry wide. Global solutions to satisfy industries and customers worldwide.
- Voluntary. International standardization is market driven and therefore based on voluntary involvement of all interests in the market place.
The Institute for Public Relations Commission on Measurement and Evaluation has joined with the Council of Public Relations Firms and AMEC in an initiative to develop social media measurement standards, and with the Council of Public Relations Firms on standards and best practices in measuring return on investment. We hope this will be a model for further standards initiatives.
Be prepared for RFPs, especially those emerging from purchasing departments, to require adherence to industry standards, best practices, and truly valid metrics. We have the opportunity now to develop our own standards. Failing this, standards will be imposed from the outside.
Participate in the discussion and let me know what you think by commenting below. You can also reach out to me directly; my contact information is on the InfoTrend company web site. And look for more detail on these topics in coming weeks.
David Geddes, Ph.D., is Chief Consulting Officer at InfoTrend, Inc. (http://www.infotrend.com), a business research firm offering predictive analytics for public relations, marketing, and branding.
References
Berlo, D. K. (1960). The Process of Communication. New York, New York: Holt, Rinehart, & Winston.
Michaelson, David and Stacks, Don. (2010). A Practitioner’s Guide to Public Relations Research, Measurement and Evaluation. New York: Business Expert Press.
Michaelson, David and Stacks, Don. (2011). Standardization in Public Relations Measurement and Evaluation. Presentation at the 9th Institute for Public Relations Summit on Measurement, Philadelphia, September 18, 2011.
Rothwell, J. Dan. (2010). In the Company of Others: an introduction to communication, 3rd Edition. New York: Oxford University Press.
Shannon, C. E., & Weaver, W. (1949). The mathematical theory of communication. Urbana, Illinois: University of Illinois Press.
Stacks, Don. (2002). Primer of Public Relations Research. New York: Guilford Press.
Stacks, Don, ed. (2006). Dictionary of Public Relations Measurement and Research. Institute for Public Relations. Accessed October 10, 2011 at https://instituteforpr.org/wp-content/uploads/PRMR_Dictionary.pdf.
I think we both agree, David, that there can be different frameworks. What’s important is that the terms used in each framework be defined the same way and, most importantly, used and understood the same way by everyone. The immediate consensus has to be on the terms used.
David, you are spot on in pushing out the definitions by a degree. Communications that are amplified in the mass media have quantifiable behavioral impacts (outcomes), which can be measured separately from attitudinal influences (outtakes).
Content-based metrics are a time-honored construct for measuring the effectiveness of PR in generating news coverage, and producing functional and tactical benchmarks for a PR team to demonstrate success.
But content metrics face significant challenges when applied to business models. Two news stories with identical impression and tonality scores, for example, can and often do have significantly different impacts on stakeholders.
Raising the bar a notch by measuring attitudinal influences and behavioral impacts separately is a good first step toward isolating discrete variables that can be aligned with business results.
Fraser — You make good points. Let me answer concisely. First, I wrote “note that I do not say a single measurement framework.” Indeed, we should welcome competing models as long as they stretch from activities to attitudes, behaviors and business results. We should reject smoke-and-mirrors “just so story” models that are so common in the discussion of social media influence.
This model is also a heuristic device to explain the role of measurement and evaluation to organizational clients and to avoid output-centric discussion. The largest number of people in the room will probably always be those heavily involved in or invested in media measurement. That has been the case at both the Barcelona and Lisbon meetings. We need a broader framework. For example, we need to get away from media-based presentations that purport to measure corporate reputation, when, in fact, they are just measuring outputs or the presentation of a corporation in the in the media.
Second, we should derive terminology from theory-based models, not the reverse. In my terminology, I would equate my use of engagement with the Dictionary’s use of “outtakes,” and my use of “outtakes” with the Dictionary’s use of “outcomes.” But social media has changed the game with the rise of the term “engagement.” Clarity is essential, I agree.
David, I agree with the importance of each of these: framework; metrics; and standards. The devil will be in the details, for achieving consensus. Just for example, within the framework category, your description of engagement, outtakes and outcomes above is different than the definitions in the Dictionary of Public Relations Measurement and Research and from their use in Walt Lindenmann’s “Gold Standard” paper: Guidelines and Standards to Measure the Effectiveness of PR Programs and Activities.
I believe that one of the main obstacles to any form of measurement consensus is a common understanding of what happens in a communication process (direct or mediated; traditional or social media) between a communication output and an outcome from that output. Simply look at the papers on the IPR site or any of the writings post Barcelona on evaluating communication campaigns (for example: Valid Metrics), in all cases terms like outtakes, engagement and outcomes (short-term; intermediate; longer term) are used differently.
Before we can come to a common understanding about frameworks, metrics and standards, we need to understand and use the same definitions. Otherwise, a house of cards will be built.