This roundtable discussion is provided by the IPR Measurement Commission
Members of the IPR Measurement Commission gathered virtually to discuss the use of artificial intelligence (AI) and technology in communications measurement. Measurement Commission and ELEVATE member Brittany Paxman, Managing Partner at Point 600, moderated the discussion.
Participants discussed what they have been hearing about AI and measurement, including the efficacy of current AI tools for analyzing qualitative or quantitative data sets, AI assistance on smaller measurement projects, and how AI tools aid with laborious tasks such as creating translations or Python code.
Key themes from the discussion include:
1.) Clients, stakeholders, students, organizations, etc. alike have had similar questions about the rise of AI. Some of these questions included:
— What is the validity of measurements created by AI?
— How reliable is AI?
— How does AI create sentiment statements/evaluations?
— Can AI automate definitions for concepts such as “trust?”
— Will AI make the job of doing good measurement easier?
— Where does AI belong in the measurement “toolbox”?
2.) AI will move backward before it moves forward – progression will be incremental rather than revolutionary.
3.) The paid version of ChatGPT’s data analysis option can answer specific questions from large data sets but cannot provide in-depth insights the way humans can.
— The tool could not evaluate thematic or content analysis successfully.
4.) AI tools are not ready for complex tasks or qualitative decision-making today; however, professionals should not wait to start experimenting/familiarizing themselves with these tools or they will fall behind.
5.) AI can be used as a tagging structure for basic measurement and analytics.
6.) The best sentiment AI analysis can now exceed human performance, but researchers need to be very domain-specific to achieve research-grade levels of analysis.
— It’s crucial that researchers understand the performance of AI models before starting to report on the collected data.
7.) Vendors should disclose the performance of their AI models to potential and current clients.
— Organizations and brands that use AI should encourage their vendors to provide thorough performance data. The information provided could help AI users make more informed decisions moving forward.
8.) Communications professionals who are familiar with AI should use their knowledge base to help less-experienced users become more informed and cautious when using AI for measurement or data analysis/reporting.
