Recently, members of the IPR Measurement Commission participated in an online discussion about internal communication measurement. Led by IPR Measurement Commission Member Sean Williams, the IPR Measurement Commissioners discussed how to measure internal communication impacts and behaviors. Here is what they had to say:
Sean Williams (Bowling Green State University): I had the honor to work with Julie O’Neil, Stacey Smith, and Michelle Ewing on a project for the IPR Measurement Commission about internal communication standards. We ran into an issue that a lot of internal communication scholars have, which is a lack of interest in businesses to let us inside and see what it is that they really do when it comes to internal communication.
The challenges in internal communication measurement are the same as those for any other kind of measurement, as all of us well know. It turns out that there’s a lot of difference of opinion about what constitutes an effective draw for employees. We know from the organizational communication literature that socialization in general and onboarding are critical and HR likes to claim credit for all of that. We know that a lot of organizations simply aren’t very strategic when it comes to their internal communication and so it tends to be a sort of a reflection of marketing or external PR being brought inside the house. And so, this presents a challenge as well.
The other thing is that there’s not widespread agreement on metrics that stipulate what the value of internal communication is. We’ve got all these different areas that are claiming credit for when employees are well informed, when they take action in support of objectives, etc., and this makes it a real challenge for us to get where we need to get.
We landed on outtakes, outcomes, and business impact as the three main areas where we would measure. Our research found: when we talk about an outtake, we’re referring to whether the audience paid attention to it, received it, understood it, retained it.
This became crucial when I was at a previous job because we had a complicated message strategy called “the seven reasons to believe in (company name redacted).” Why are there seven? Who can remember two? But the point is that the powers that be really wanted our employees to remember what these seven were. And three of them had nothing to do with internal communication. And so, we can reasonably say that if you retain any part of those seven reasons, then we must be doing a pretty good job of communicating it and putting it into context.
When we established these standards, we also said, “Okay. So, after we’ve done that, what is the outcome?” So, changes in attitudes, in willingness to advocate, a perception of authenticity, a sense of being empowered, a willingness to collaborate.
Then we got to organizational impact, and we were able to narrow this down pretty quickly. The concept of quality is important here. We used to say in manufacturing that output per worker hour, safety, and quality were the three main metrics that we wanted to realize in our workforce. And so, I think quality belongs here but notwithstanding that, productivity… whether people are trying new things… is there permission to fail? Is there permission to take risks and chances? Are we doing things that result in continuous improvement?
The problem is trying to isolate each contribution that goes into each of these things is rather difficult and can be time-consuming, costly, etc.
And with that, our first question is, where are we now? The typology says we should be measuring organizational impact levels. But are we doing that?
Alan Chumley (Real Chemistry): We look at all the usual channel and content engagement metrics, the workplace Facebook, Slacks, Intranet metrics so on and so forth. We’re looking at trends in Glassdoor scores and reviews. We’re 100% healthcare so Cafepharma is a source of rich, often caustic commentary of employees. So, we’re grabbing reviews from Glassdoor and Cafepharma and driving it through our text analytics, content analysis, topic modeling, theme clustering, all that understand the language employees are using and drivers and detractors.
We’ve experimented with a platform that uses natural language processing on multiple review sites. They’re enriching the data from Glassdoor and Indeed many others and they purport to be able to surface 72 (which is astonishing,) indicators of culture. I’m not convinced there are 72 indicators of culture.
We’re building Twitter panels, listening panels of employees who have either directly or obliquely flagged that they are an employee of company XYZ. So, we integrate all that into one place, the Twitter handles and the content running text analytics on that at an aggregate level. It’s done in a GDPR-compliant way and not used to identify nor target individuals.
We are seeing larger clients build their own dial testing into their internal platforms for major leadership announcements and such, so they can see the real-time sentiment or emotional credibility, authenticity. A number of metrics allow you to pinpoint a spike or a drop coincident with the delivery of a messaging moment in the content.
Sean Williams (Bowling Green State University): I think there is an opportunity because of the growth of internal social media, platforms like Yammer and Slack. You have to think about compliance and legality, but these insights are very helpful.
What about others? We’re striving to connect all these things to the organizational impact level but is anybody aware of doing that?
Dr. Julie O’Neil (Texas Christian University): I think when we looked at those standards, it’s very much a functionalist approach. We examine how that helps the organization. The organization does this, sets the goals, the objectives, implements, and measures. And that is still crucial, of course. It’s always important to have goals and tie them with the organizational strategy. I think in the last six to nine months, the emphasis is increasingly shifting to the employees as our marketplace changes and employees are wondering what to do. We talked about this great reflection period that many people are going through right now. Do I want to go back and commute an hour each way and work X number of hours? The pandemic has shown us that there are different things to think about right now. And I would think companies today have to be grappling with employees’ mental health, their happiness, and what are they getting out of this. Particularly women. I would think some of that has to be in the measurement toolkit and thinking about those important issues facing us right now.
Elizabeth Rector (Cisco): I have internal communication insights now as part of my team and in the past, we had a real bifurcation with our HR and our employee communications. I have been working to bring those two functions together. We manage big company meetings that we do once a month and then we have follow-up surveys. So, we’re able to analyze the discussion for the sentiment of what’s being discussed, a Q&A, and then we have perception questions that are aligned to our business goals. There are consistent perception questions that we’re asking and if there were specific goals in the company meeting, we ask about those.
We also have our intranet, almost like an external website. We do a lot of analytics on engagement, the type of content being shared, how the content is aligning to goals, is shifting the needle, etc.
I think we’ve moved the needle a little bit by at least bringing the insights in from HR, taking those into account, and trying to make the link. It’s not yet a perfect, integrated plan, but at least it’s starting to come together and we’re getting those conversations going between the two teams.
Sean Williams (Bowling Green State University): What I found during my practice as a consultant was that there was a lot of what I would term “professional jealousy” between the PR and HR regarding who owned what. And it became more of an exercise about how big your budget was gonna be on the HR side and how many people were gonna report to you and your team versus the other and which one was “more important” on the basis of that budgeting. I’m delighted to hear that there is some movement to try and put these things together because in the end, we’re all trying to contribute to the same set of organizational objectives and we need each other in that respect.
Rob Jekielek (The Harris Poll): Employees are one of the most critical stakeholders. So, I think it’s really important to figure out. Different companies have different structures that can support different levels of information. I think I would reinforce that just having a survey component to whatever you’re doing is a critical element. It’s certainly not your answer to all things, but having some sort of an annual read, if not an ongoing read of how people are thinking about a variety of things, engaging with the company, their level of involvement and activity, etc. And in particular, their ability to align to the strategy assuming that you’ve articulated one is a critical component.
Surveys are not dead if you will. The work that we do cross-stakeholder looking at employees is probably one of the most critical elements for understanding both the opportunity and risk ahead. Being able to look at very similar perceptions internally versus externally.
I think having strong relationships with HR is really, really important so that you have good intersections. There are some clients that we work with where there’s a very one-to-one relationship and so all this stuff just gets immediately translated into a lot of HR-related work. So, in terms of how it gets built into internal management reviews, it’s still very much separated.
I think when people look at a lot of external surveys, they think, “Oh, it’s too expensive. I can’t do it.” Internally, it’s more about building the right buy-in. If you articulate it right, like topics around culture or strategy, those are number one or two out of five priorities for any CEO right now. So, I think there are really big opportunities. I think there’s a lot of tactical things under the hood. There are some decent perspectives that you can glean that provide a unique data source because they add a human component. You can get to more sophisticated ideas than what you just see generally in Glassdoor.
And when you’re looking at employee bases that are very large, you can map different networks of people based on identifiable features. So, building networks of 10,000, 20,000, 30,000, 40,000, 50,000 employees is reasonably straightforward to do these days.
Sean Williams (Bowling Green State University): Thanks, Rob. So, the next question is about what the organization expects in terms of impact. In practice and now being in the academy, there seems to be a widespread disagreement about what the C-Suite expects from internal communicators. Way back, one CEO told me that I didn’t have to convince him that internal communication was important, I just had to convince him that what we were doing to accomplish employee communications was effective. To what extent is this still a factor? Rob touched on a few of the modern ways that we can investigate the impact, but what sort of impact does your organization or your C-Suite expect when it comes to internal communication?
Allyson Hugley (LinkedIn): It’s interesting when you think about the value or the impact because part of that is often tied to how the communication function serves as a feedback mechanism for the broader organization. At the organizations that I’ve been with in the past, it was viewed as communications are put out and what we receive back is less a measure of the impact of the communications in and of themselves, but more as a way of assessing the overall health of the organization. So, what are the mechanisms through which we’re looking at the different types of content that people are engaging with? The cultural content or is it purely anchored to productivity as one measure? Now that we have a distributed workforce, I think understanding where the breakdown is in some of the systems is now required in order for organizations to function in an exclusively remote or hybrid environment productively. So measuring not just the communications in and of themselves, but also looking at the signals we can detect in terms of end-user performance analytics that would say there’s breakage somewhere in the chain of communications that would compromise the overall knowledge health or cultural health of the organization on a certain point.
In my more recent experience overall, we’ve been asking “how is this a calibration of the health of the organization in terms of technology, infrastructure, and culture?” and from there, putting together an aggregate view of those health signals and metrics.
Sean Williams (Bowling Green State University): I think this has given us a lot of opportunities lately – there is a lot more information available to us than there used to be. Now I think it is easier to do insightful listening.
Allyson Hugley (LinkedIn): One more thing – there is a broadening of what is considered to be internal communications. I think the landscape through which communications have an opportunity to exercise influence over the organization is intersecting with HR and more pieces of content or channels are being seen as critical paths. I think the challenge is how do we expand, from a measurement standpoint, the lens through which we assess the signals we want to focus on? Do we really want to go down the path of almost everything that touches an employee is in fact a communications mechanism? In that case, we have a license to be much more involved in the aggregation and analysis of employee data because it is different when viewed through the lens of communications and employer or brand building versus just talent acquisition. I think the runway for insights and impact on strategy is broader when viewed through the lens of communication.
Sean Williams (Bowling Green State University): Great comment. I think our friends in marketing have long held that every single touchpoint, whether it’s inside or outside the company is intimately related to the brand. And for a while, they seized that from internal communication and said, “No, no. It’s all about building brand health and it’s all about making sure that everybody’s a brand ambassador and that they’re prepared.” And without internal communications, they couldn’t do it so they had to turn everybody into a brand ambassador. And I think the opportunity is to reassert that we’re the only ones in the organization who are uniquely equipped based on our understanding of the marketplace itself, the internal marketplace, our understanding of the organization, and our understanding of the employees.
Allyson Hugley (LinkedIn): I would agree with that. I think one of the unique aspects of the communications function is risk mitigation. I think over the past year when we got into issues of equity, etc., communication plays a unique role in a lot of that reputational risk assessment that is not just the positive side that’s part of brand building but that also is the front line for brand protection. And I think all of these signals that are communicating and building the employer brand have heightened levels of scrutiny right now where communication should be at the fore of both measurings and then influencing from the standpoint of brand protection and reputational risk mitigation.
Johna Burke (AMEC): I love what Allyson just said. I was talking to someone earlier and they were talking about managing some of their reputations, they actually brought someone in to monitor the dark web which I thought was just for buying uranium but apparently, it’s not. But looking at that as it’s related to your organization, some of your employees, any activity there if that creates a trust variant there. And I think with so many people looking to combat misinformation in many different ways, that’s become an early indicator if there is some trust variant, especially with financial information or health data. And I think if communication isn’t on the cusp of that and understanding what data is there that’s available, how that can correlate to anything across the PESO Model, I think we’re really vulnerable to letting IT sweep that up and take that over when that’s really valuable data that can help lead the communication conversation.
Sean Williams (Bowling Green State University): Let’s switch gears and talk about improvement. So, if we understand what our current limitations are and we understand what our aspirations are and we have a clearer understanding of what the expectations are with respect to impact, then what does the improvement plan look like? What resources do we need in order to accomplish our goal to move more toward business impact measurement and to make sure that the internal communication landscape and infrastructure cover the things that we’ve just discussed?
Allyson Hugley (LinkedIn): I think it comes back to the partnership. The challenge is that the farther we drill down, the less receptivity there’s gonna be in organizations to having outside parties own that data. They’re going to want to see this building of in-house resources and capabilities with technology to sort of pinch it all together in a way that feels secure.
And so, I think that partnerships with marketing so they get the benefit of the insights and the aggregate data and the unique perspective that communication provides and these closer partnerships with the CHRO and HR organizations are important. Technology should also be much more of a primary partner and communications should develop that technical proficiency and fluency to be able to work effectively with those partners alongside compliance. I think that’s needed to get to the next phase.
Dr. Julie O’Neil (Texas Christian University): I have two thoughts on improvements. One would be data proficiency. We’ve talked so much in the last 20 years about measurement evaluation and now we’re gonna throw data in there which is yet another skillset. And I think increasingly practitioners probably need more training in becoming more data proficient in data analytics in general. So, I’d say that’s one improvement for the profession as a whole. And I know we’re looking at that as educators right now.
And then second, I think internal communication practitioners have to be advocates for diversity, equity, and inclusion (DEI) in their organizations right now. Again, these are huge issues facing all organizations and so I would think that needs to be part of the conversation as well in terms of measurement evaluation.
Sean Williams (Bowling Green State University): Yep. We’re attempting to diversify our own ranks as PR people but also there are a lot of organizations that are struggling with meaningful ways to address DEI issues.
Dr. Jim Macnamara (University of Technology Sydney): I agree with Rob that traditional methods like surveys are still a necessary and vital tool. I also agree that we’ve gotta get better at data. But where I would depart from some is that we are a little bit obsessed with quantitative data. And I think we have to recognize that people speak and tell their stories in words and that requires qualitative analysis. And where I found the biggest impact in employee and internal communication is when we start gathering data using richer, deeper methodologies. It might be participatory action research, it might be an appreciative inquiry. I think we rely too much on traditional methods in research. I think we’re not using deep methods and a lot of those methods come back to one of my areas of research and that’s listening.
And too often senior management thinks of internal communication as telling people stuff. The biggest movement on the needle is when you’re listening to your employees. Really deep listening. Not just how they feel but I’m working with an organization in Europe that is involving all its employees in product development, giving awards for the best ideas and the best designs, publishing what the employees are saying, and really giving recognition and it’s absolutely transformed the internal culture.
I think that making sure we understand data as qualitative data as well as quantitative data is important. We need to make sure we understand deep listening and use deeper methods at times.
Sean Williams (Bowling Green State University): Yes. A follow-up comment with respect to the content analysis, I think one of the limitations of content analysis is the desire to take inherently qualitative data and make it quantitative.
Dr. Jim Macnamara (University of Technology Sydney): And it’s highly reductionist. You’re reducing deep levels of feeling and emotion down to a few numbers. So, we’re always being very reductionist in our quantitative work.
Sean Williams (Bowling Green State University): I agree with you, Jim. Having the qualitative is much more robust and even if it’s not directly projectable into wider populations, at the very least we get snapshots. One last comment on that respect is that one of the things that I’ve done in a couple of different assignments is a daily poll. And, you know, it’s not scientific, but it gave us a snapshot of what people thought about certain topics or issues. Four days a week they were serious business topics and then there was the Friday fun topic to lighten things up. The participation was excellent and we used that to guide us as to what sort of content we would put onto our intranet. And we reported the results to, as we call it, the fourth floor.
For me, that was a classic way of taking a snapshot, listening to what people were saying, and then trying some things out and seeing what the response was to it.
Michael Ziviani (Precise Value): There’s something else going on here in the background which is probably worth noting…in my experience, you’ve either got a fourth floor that you mentioned, Sean, that is open and aware to receiving back any and all information or I actually want to avoid detail, have just one or two metrics and peg a lot to that. So, there’s a mindset of outlook which is really important to understand first as a contextual aspect to measurement and evaluation.
Johna Burke (AMEC): And there are some interesting things that happen in internal communication, even from PR reporting on their coverage into their internal teams. We worked with a group where they took their news coverage and looked at a couple of variants with it as far as what they put first. Was it company news? Was it employees quoted? And what they found through the link tracking of where everybody was visiting was that their employees were first and foremost interested in what their peers were saying and how they were appearing in the news. Their peers mean all of their coworkers, from the CEO to the product manager who may have done the interviewing. So, once you uncover what’s driving interest with your audience, that’s where I think it allows you to effectively drill down into some of those other needs that they have. By missing out on that, people start to ask the wrong questions or leading questions that don’t have to be asked when they can be led by the data that they already have. They miss that piece that I think is critical.
Rob Jekielek (The Harris Poll): I would double down on the qualitative insight from Jim. I think it’s mission-critical. We found qualitative focus groups to be particularly important and impactful with the mid-level manager. Groups where you can get people’s perceptions and see how things are going and that’s usually one of the spaces where everything gets very actionable very quickly because of their level of responsibility, both in terms of communication as well as executing the strategy.
Also, figuring out a sampling plan for employees is not just, a “nice to have.” You have to be able to stage and look at that in a very strategic fashion. And in most cases, a good rule of thumb is 10% of the population just as a starting point. Obviously, you need to go bigger, deeper, wider if you wanna get cross-cuts into things like tenure, age, level, and a variety of other factors.
Sean Williams (Bowling Green State University): Awesome. So the next question is, what would be most helpful to your organization in terms of research to support your measurement ambitions?
I’ll go first. I have two thoughts. One is that the relationship between the internal communication function and other departments would be interesting to explore to give people some sort of baseline to understand what the relationships are like. We keep looking for that magic metric and I don’t know that it is worthwhile for us to continue to pursue that. I think I would rather see us do more qualitative work that gives a broader and more interesting depth of understanding as to how different sorts of communication with the enterprise work. And then lastly, I think the SOS model, sending out stuff, is the G-rated version that indeed has been the modus operandi for internal communication for some time in a lot of organizations. And I think changing that in part requires that internal communicators stop thinking of themselves as internal journalists or internal PR people and instead think of themselves as businesspeople for whom communication is their superpower. And this concept of actually making the communicator the expert in communication in the same way that other experts within the enterprise are experts is also critically important. And to achieve that, I think we have to see more of a crossover between the organizational communication literature and the internal communication literature.
If you study organizational communication, you’re getting a master’s class in the history of management. And overall management helps you understand not just how it is that the people in the C-Suite are telling people, but also how it is that people who are being told relate to and understand their position in the enterprise.
So those are two things on the research side that I think would be very valuable.
Michael Ziviani (Precise Value): I think your comment on management is the crux of it, Sean. There’s a lot of window dressing and it’s easy to get distracted with that – whether a single metric or proprietary method, whatever. But to me, it starts from that management place and works outwards.
Rob Jekielek (The Harris Poll): Especially when we’re looking at internal measuring and theories and things like that, they really need to be tuned to the organization. Otherwise, they fall flat. I would just also reinforce the importance of asking practical questions and not using concepts that are too broad or too esoteric. Asking behavioral questions is also an important indicator. So, things that people are doing or not doing is a really good mechanism with employees for understanding what’s valuable or not.
