Proposed Interim Standards for Metrics in Traditional Media Analysis

20 thoughts on “Proposed Interim Standards for Metrics in Traditional Media Analysis”

1. The document states, “OTS must be specific to a particular channel.” I completely agree with that.

   The document further in the same paragraph states, “For Facebook it is the number of fans to a page.” In short engagement will increase OTS beyond the number of fans by drawing from the Friends of Fans “network.
   Further, paid promotion on Facebook will greatly increase the OTS and provide (beyond the paid OTS) a natural organic viral spread of the message which is 100% EARNED; thus the OTS shouldn’t be limited to just the number of Fans.

   Here are my thoughts in more detail.
   Limiting OTS (on facebook) to fans of a page is fundamentally flawed, albeit easy to compute. There are several reasons why this is flawed, but primarily there are three major considerations which impact OTS on Facebook.

   The first issue is the percent/dollar value (if any) of paid promotion. The second reason is the impact of EdgeRank. Thirdly, there is the complex relationship on Facebook of engagement, EdgeRank and OTS

   This third issue is highly complex. A fan page with 5700 fans may have a “friends of fan” network of more than 1.6 million people. Depending on if, how or when promotion is paid for and depending on who and how fans engage with the page depends on how many people EdgeRank will allow to see the post. If we are looking for a true baseline of OTS and need a simple easy to compute number the Friends of Fans is far more accurate, although harder in fact nearly impossible to achieve.

   The point is just because a page has a 1 Million Fans doesn’t mean there is 1 Million OTS; it could be far higher (because of high engagement and the Friends of Fans “network” or lower (because of low engagement and Edgerank).

   To be fair no matter how good the engagement EdgeRank is always going to limit OTS.
   Understanding that PR measurement is trying to measure Earned media, we have to understand that Facebook is an amalgamation of Earned, Owned, Paid media, and as in the real world each of these influences each other.

   Facebook breaks out impressions, reach, and many other metrics on a post by post basis. It is my belief that an easy to compute but complex ratio of these metrics including the number of people in the Friends of Fans “network” would yield a far more accurate measurement of OTS on Facebook fan pages. We should define it and then request Facebook adds the metric to their insights

   Assuming that a page has at least the average PTAT ratio of 2%, implying a basic level of engagement; the true
   OTS for a post is in far excess the total number of Fans, even though EdgeRank may simultaneously prevent many of the fans from seeing the post.

   Warmly yours,
   /harry hawk

   PS
   I have clients with PTAT that range from 10-50%

   PPS
   Paid promotion on Facebook will greatly increase the OTS and provide (after filtering out the effect of paid OTS) a natural organic viral spread of the message which is EARNED; that should to be measured.

2. Thanks for your comment. I can appreciate the measurement challenges of a small organization.
   The standards outlined in this paper are about HOW to measure traditional media coverage, but WHAT you measure should be customized to your organization’s objectives. If your resources are limited, then you should select measures that are achievable and in line with the objectives. Without the resources for in-depth media analysis, consider developing a proxy. For example, create a short list of priority media titles and track how many mention or feature your organization in their coverage. Use this as a benchmark to measure against in the future. Or, if the objective is a call-to-action, such as drive registrations on a web site, ask the web site owner to set up analytics to track that specific action over time. These are examples of smaller steps that you can take to track impact which would be better than relying on instincts and use far less resources than AVEs.
   Calculating AVEs is an incredibly time and resource intensive activity – measuring column inches, researching ad rates for multiple media outlets and making the appropriate calculations based on placement (e.g. front page has a higher rate). And, there is no standard for measuring AVEs, so you need to determine how to account for quality factors, like the inclusion of a photo or chart, for your organization’s negotiated ad rates and for the portion of the coverage that is relevant. After all that effort, you don’t have a measure that reflects the value of PR or provides a benchmark for future measurement.
   For other ideas on how to replace AVEs check out the Valid Metrics Framework first introduced by the Association for the Measurement and Evaluation of Communications (AMEC) in 2011 and available through their website.

3. Thank you for producing this document. My comments are below:

   Standard #1:
   Newspapers
   I believe that the most accurate and widely used sources of print readership in the US are Scarborough for newspapers and GFK MRI for magazines. I think they should be included in your list.
   The document contains no mention of newspaper section readership levels. According to GFK MRI, around 35%-40% of newspaper readers actually read the editorial section and around 37% of them read the business pages. Applying this factor to the overall newspaper readership section results in more accurate than overall impression levels and results in a better “opportunity to see” measure.

   Online Media
   comScore is a highly respected source of unique visitor information for websites and should be included in your list. They also publish an average daily number of unique visitors. According to my calculations (based on comScore data), this number is 16 times lower than the monthly unique number for news sites. comScore’s daily UV number is calculated based on observed behavior and is more accurate than a pure estimate.

   Also, you did not include any guidelines for measurement of mobile media consumption or online video consumption.

   Thanks.

4. All of this is great, but I think there are a few things to remember.

   For the majority of organizations, a PR staff is small (possibly a 1-person shop) and likely underfunded. The luxury of detailed analysis is often not an option due to many reasons not excluding staff time and budget. Most of us are off to the next project without choice. At best, we do a quick visual analysis and use our instincts. Furthermore, we’re often trying to justify our jobs to beancounters who do not understand the value of PR, and may never. While rejected by this paper, AVE does play a role in proving your worth in situations where detailed and expensive analysis cannot be afforded.

   This report is wonderful for large scale operations with significant budgets. But I suspect it won’t be helpful to small operations. It would be of great value to find a simplified process that takes into consideration staff, time, and budget limitations.

5. Dmitry, yours is a very good question. The standards for traditional media analysis do not include blogs, only the online version of print media publications (e.g. New York Times vs NYT.com). However, blogs become a grey area because many publications also have one or more blogs. In practice in media analysis, we often include the posts (not the comments) on these publication blogs and other relevant, targeted blogs. I am currently working on a project to test the reliability of the standards and we are including blogs in the analysis for this reason. In practice we find that blogs are often where stronger opinions are expressed and many are influential based on the readership numbers, volume of comments, links, etc. So, it is a bit of a grey area that we are coming to terms with for our revised publication.

   Many thanks for your insights.

6. Being more specific, I’ve got a question: why blog posts are included into list of items for traditional media analysis but comments, Facebook and Twitter are not included? May be blogs should be excluded from traditional media and regarded as social media only together with social networks etc?

7. JP and others, I assure you that my team and many of my research colleagues are going far beyond mere content analysis when it comes to measuring PR results. One example is the METRIC Model for measuring stakeholder engagement, an award winning piece of work which can be found here http://bit.ly/ROWPEi. [Full disclosure – I am the author.]

   However, there is still a need for standards around the basic measures of traditional and social media analysis. Clients are asking (or insisting) that their agencies align on how they calculate impressions, etc. In the absence of industry standards they are creating their own to ensure they have transparent, replicable metrics.

   The paper acknowledges that this is only the very beginning of what needs to be addressed in standards and asks for suggestions on other essential metrics to be included. Yours would be most welcome.

8. Great comments everyone. Jim, Michael, and David M, you are all right that we need to raise the level of discussion. How many years have wee been trying to focus on showing impact on target audiences (notice, I am not using the ROI word), and showing how public relations contributes to organizational value? Look at the Michaelson/Stacks paper in Public Relations Journal for an excellent example of the standards we need farther along the value chain. However, we will still need standards for calculating impressions and other mundane elements of traditional media and social media analysis.

   We need to root out pseudo-science (for example, look at most of what goes for measurement of “influence,” whatever that is) and put rigor in our profession. We will not solve world hunger this year, but we will make progress.

9. David, are you are referring to your Zip Chips study? One of my personal favorites.

   You are absolutely right to call me out for citing unpublished research linking news to brand and business impacts, including impacts well in excess of advertising. The work is out there, but because the research is proprietary, it has not endured the academic rigor of published research.

   Fair enough.

10. JP,

    I am curious about the research you cite on news being more compelling than advertising in shaping consumer opinions and driving purchase decisions. The work done by myself and Don Stacks reaches a very different conclusion. You may be interested in reading it. It is published in Public Relations Journal volume 3, number 3. In addition, an earlier version of that research is available on this website. Please provide the source of the research you cite.

11. Michael, you are absolutely spot on. The PR industry shys away from taking any responsibility for brand and business results, and the researchers enable this by focusing on content analysis.

    The irony is that there is research out there showing that news is infinitely more compelling than advertising in shaping consumer opinions and driving purchase decisions.

    But instead we focus on content analysis, for various reasons, and allow marketers to integrate news effects into marketing analyses, and credit ads for the lift generated by news.

12. Many thanks to all of you who provided comments on the standards document. They are very thoughtful and each one will be carefully considered as we move through this standards process. And, quite a process it is. This document is only the very first step – identifying areas where standards are needed and drafting a discussion document for broad industry feedback.

    We intentionally started with standards that need operational definitions. Revision work will be ongoing and eventually result in an interim standards document for further review, testing and revision before final, more definitive standards can be published. This is in line with the International Standards Organization (ISO) process for standards development. To have standards that the industry can truly embrace, we need to follow this course.

    Standards development is an iterative process. We have started here with some of the basic, but most pressing items that our clients are demanding for consistency within their own organization’s measurement. We will expand this list as we move through the process.

    Yes, these initial measures relate to communications outputs, as some of you have pointed out, but outputs are part of management of the public relations function, and part of improving program performance. We fully intend to include outcomes within the standards as this document evolves. For more on standards on how to conduct research for measuring public relations outcomes, you may want to see https://instituteforpr.org/topics/standardization-in-public-relations-measurement-evaluation/.

    Unlike other marketing communications disciplines, standardized measures for public relations activities have never been recognized and practitioners have consistently failed to achieve consensus on what the basic evaluative measures. So while this effort may seem insufficient, it is more than anything we have to date.

    Standards development is also a collaborative process. What would be most helpful is feedback on the items included here and recommendations for other output or outcome measures for inclusion as we expand this document.

    I am looking forward to your continued input and an ongoing dialog on the standards.

13. Response to Harry Jenkins – Thanks for the challenge.
    Hmm, well, I did say what I meant. But rather than just saying it was a poor example I tried to give some direction to what I thought should be done, which is, of course, my opinion.. Can you build on or correct my thoughts … to advance the profession?

14. In response to Michael Kelly, personally I feel you are being too polite. This work is a poor example of what a standard should be and does not advance the profession. Please say what you really mean.

15. This paper is an example of why standards have not been adopted. Overall this paper does not present standards. At best, they are general guidelines. What it does present are measures that hold no meaning for the practice of public relations. The purpose of measurement is to improve program performance. These recommended measure fail to provide that benefit.

16. 1. Too polite, too professional. Make version 2.0 more direct and more relevant.
    This seems to me to be a well-written, clear description of how to do the wrong thing more accurately. The author and contributors seem to know this but are too polite and professional to be more direct. They put in a note here and there putting some of the topics, eg ‘impressions’ into perspective, but fail to say that these are all measures of activity and not effects in the minds and actions of the audience. Although they get close to saying so, they don’t and this is a disservice. Professionalism is not Polite. Professionalism is working for the bigger picture, the results, and being totally realistic about the situation and what has to be done and helping others, respectfully, to see that and act on it. This may mean making a direct, almost impolite statement or two to get attention and open up minds for possible change.
    2. I’m not sure where this fits with other work being done to make PR more effective, for example, the PRSA new ‘PR Defined’ statement or Page Society’s ‘Building Belief’ monograph. Even the Barcelona Principles went further than just stamping out AVE’s.
    I wonder if the effort of the good and capable thinkers and organizations working on these standards would be better put to bear on the big picture of what organizations need from PR/Communication professionals instead of helping do the wrong thing better. Perhaps by starting with what the standards should be to implement the new PR definition and build belief?

17. Thanks Jennifer for your translation.

18. Google translation of above comment:
    Good morning: Stressing the great contribution made herein to the measurement of traditional media, I want to comment on the use of the term NEUTRAL. I disagree. The reason is simple. First of all, because a neutral is something that does not move the balance either positively or negatively. Our company has done several tests with customers in Colombia, which show that a story that only describes events associated with a brand, they do generate a favorable impact on the reputation of a company. Course is required for continuity and permanence in time for these impacts can be generated. Our position is that simply include a check in any context, is already generating a meaning, which is far from being considered a neutral tone.

19. Buenos días: Destacando el gran aporte que hace este documento a la medición de medios tradicionales, quiero hacer un comentario sobre el uso del término NEUTRO. Estoy en desacuerdo. La razón es muy sencilla. Primero que todo, porque algo neutro es algo que no mueve la balanza ni positiva ni negativamente. Nuestra compañía ha hecho varias pruebas con sus clientes en Colombia, las cuales muestran que una noticia que solo describe hechos asociados a una marca, sí generan un impacto favorable para la reputación de una empresa. Se requiere por supuesto que haya continuidad y permanencia en el tiempo para que estos impactos puedan ser generados. Nuestra posición es que el solo hecho de incluir una marca en cualquier contexto, ya está generando un significado, lo cual dista mucho de ser considerado como un tono neutro.

20. Thank you for this contribution to the important issue of identifying standards in PR measurement such as media analysis. I note that the document is titled ‘Interim Standards’. Can I suggest that in further versions that more international focus and relevance is provided. If IPR is the Institute for Public Relations of America, the document is fine, but many of us look to IPR as an international leadership body. Other countries have their own audit bureaux for circulations, their own ratings agencies, etc. A narrow US focus only contributes to the view that PR thinking is US-centric. Second, the final section of the paper seems to have either been heavily edited or not completed. It mentions intercoder reliability assessment, almost in passing, but makes no mention of using more than one coder and doing double blind coding of at least a sub-sample. Without this, IRA is impossible. Also, there is no mention of establishing a coding list or coding book. These are fundamental elements of reliable content analysis to ensure quality. So I hope the project goes further in identifying standards. Cheers.

Comments are closed.