Katie Paine: Black Box to Lucite

As measurement is increasingly seen as a requirement for communications professionals rather than a “nice-to-have,” the number of different methodologies and techniques to measure results has proliferated.

Now that the tipping point seems to have been reached and measurement is appearing on almost everyone’s to do list these days, its time to move beyond “any measurement is good measurement” to the stage where CEOs, clients and agencies aren’t just demanding “measurement” but rather demanding that measurement be accurate, that the methodology be transparent, and declare the days of the “black box” dead.

For far too long communications professionals, who might know a great deal about writing and editing and pitching, but know little about research, have been taking numbers and manipulating them to make themselves look good. The vast majority of “scores” “indices” and other ultimate measurement numbers are based on at best old research data, and at worst nothing more than the gut feeling of the person inventing the number.

The problem with these various scores is that people are increasingly basing decisions on them, and that’s like trying to land an airplane when your altimeter and compass haven’t been calibrated.

To address this issue, I am calling on the PR industry and the media that cover the industry to raise and address the issues of accuracy and transparency in media evaluation.

Here are three things that every responsible public relations and communications professional AND the organizations they belong to should be doing:

• Demand a standard set of questions and answers that every customer should know and what every vendor should provide full transparency on.
• Raise the issue of measurement accuracy and transparency every chance they get.
• Talk to their clients about the difference between good research and shoddy research practices.
• Reject from their membership any organization that continues to promote bad research and the “black box” mentality.

I realize I’m being radical here, but if we are serious about the notion that there is science behind the art of public relations, to quote the Institute for Public Relations, we need to demonstrate exactly what that science is.

Here are the prime areas where methodological mistakes and hocus pocus take place:

Data Collection:

• In order to compare apples to apples in research, you need a comparable universe. In media evaluation, this starts with the search string that you use to collect your media. You need to understand the methodology by which media are included and the entire search string used.
• If you are using a select group of publications, a vendor should provide sampling methodology.

Coding:

• Positive and negative are very subjective terms. A good vendor should provide a clear definition of how an article is rated for tonality and what they mean by tonality.
• Visibility, prominence, focal point impact score, whatever you want to call it – the degree to which an article is visible to the reader is used by most vendors as a criteria for success. How that visibility is defined is critical. Vendors should be able to provide a clear methodology for how they select “highly visible” articles.
• Coding methodologies should be transparent and appropriate to the medium and the audience and the channel.

Statistical robustness:

• Multipliers have long been used by shoddy research firms and agencies to “juice” results and make numbers bigger. Unfortunately they have no basis in science. There is no evidence whatsoever that PR is three or five times more credible, believable, or impactful than any other form of marketing.
• Too often we’ve seen very pretty charts and graphs based on two or three articles, moving the data from the realm of science to one of anecdote. Conclusions must be based on a sufficient volume of data.
• Circulation figures should be audited and no multipliers should be used and consistent sources should be used.
• If weighting is used, the rationale for the weighting should be provided.

These are just a few areas of concern. I’m sure there are others and I invite the industry into this discussion.

Katharine D. Paine
President & CEO, KDPaine & Partners
Member, Commission on PR Measurement & Evaluation