David Geddes, Ph.D., Julie O’Neil, Ph.D, and Marianne Eisenmann, MBA, received the Jackson-Sharpe Award for this 2014 International Public Relations Research Conference submission.
Abstract:At the 16th Annual IPRRC in March 2013 the researchers presented the first phase of the study Assessing the Reliability Metrics Proposed as Standards for Traditional Media Analysis. Researchers received considerable, thoughtful feedback both during and after the Conference that has informed Phase Two of the research – an enhanced, stronger piece of research building on the previous study. The initial study tested the metrics using inexperienced coders, trained using a coding guidebook and instructions developed by the researchers and based on the Proposed Interim Standards for Metrics in Traditional Media Analysis guidelines. The Phase One study results yielded low to moderate intercoder reliability based on Krippendorff’s alphas. The most significant feedback on this work was the recommendation to repeat the study using trained coders to see if/how reliability improves with experience.
The research paper addresses efforts to repeat the methodology from Phase One with two pre-tests followed by independent coding of the identical set of 106 articles about Wal-Mart, using three experienced coders. Results indicate that coding for the metrics of standards of traditional media analysis is reliable. Ten of the thirteen media items had moderate to high alphas, indicating that the three coders were in agreement the majority of time. The paper documents additional best practices and includes updates and improvements to the coding guidebook gleaned from working with the experienced coders.
The standards document evolved to meet the practitioners’ need for replicable, comparable and transparent outcomes in media content analysis. The implementation of Phase Two of this study allows for clearer, more definitive conclusions and provides readers with robust guidance and best practices, and a ―ready-made‖ set of tools in the form of a tested and effective coding guide and coding grid, to implement media content analysis with the necessary transparency in methodology and confidence of replication.
