The Science Behind the Inclusion Survey

Guidance on our Inclusion survey and the science behind it

Jared Ellis avatar
Written by Jared Ellis
Updated yesterday

What can I learn from this page?

Guidance on our Inclusion survey and the science behind it

Who is this guide for?

Account Admins, Survey Admins, Survey Creators

Diversity, Equity, and Inclusion are each meaty topics on their own - building a survey to measure these constructs and help organizations move forward was quite the task!


Purpose: Why You Should (and Shouldn't) Run the Inclusion Survey

Running a single engagement survey won’t solve all of your organization’s issues. It will, however, start conversations and get you on the path forward. Similarly, an Inclusion Survey brings data to the table; highlighting if, where, and how different groups of employees experience your company culture differently.

This template combines the measurement of evidence-based and research-driven constructs, inclusive demographics, global benchmarking, text analytics, and an action framework driven by collective intelligence. By collecting, understanding, and acting on feedback related to diversity, equity, and inclusion, you can expect to learn how employees with intersectional identities experience your organization's culture.

We do not recommend using the survey for other intended purposes - for example, we do not recommend using this survey to solicit feedback on sexual harassment. If you intend to use the Inclusion Survey for a non-obvious purpose (or find yourself using lots of unique survey questions that are different from what is in the template), please check in with support@cultureamp.com to see if we can assist you.


Questions: How We Identified the Survey's DEI Constructs

For a full list of the Inclusion Survey template questions, please see the Template Library inside your account. The questions were based on our original template developed in partnership with Paradigm but have been revised based on scientific analysis and incorporation of current research.

We identified 7 constructs where we know negative stereotypes impugn minority groups and can prevent all employees from achieving their full potential; but also where optimism in these areas can lead to new pathways for innovation and business success. These constructs are Inclusion, Equity, Growth, Decision Making, Diversity, Voice, and Contribution to a Broader Purpose. The questions in each of these constructs originate from applied experience, modern research, and scientific analysis.

Insight.png

Applied Experience

Some questions come directly (or very closely) from our engagement template. They might not even seem like they are a “diversity, equity, and inclusion” question. For example:

"At ACME there is open and honest two-way communication"

  • In 2020, Culture Amp surveyed over 1.6 million people, and found that men score around 6 points higher than women employees on this question.

  • We know from applied experience that minority groups may be overlooked in communications and face barriers in receiving and providing honest feedback.

  • We also see a 10 point gap between the youngest and oldest employees when it comes to communications.

Taken together, this shows that the experience of communication differs for employees based on their demographics.

Modern Research

Some questions were adapted from modern research topics, such as Lynn Shore’s work on diversity and inclusion, or Amy Edmonson’s research on psychological safety. This question is an example of measuring psychological safety as an outcome of inclusion:

“I feel safe to take risks at ACME”

  • Minority group members are often the subject of negative stereotypes. This is the basis for stereotype threat, where minority group members see risk in conforming to these negative stereotypes.

  • While there is risk in conforming to negative stereotypes, there is also risk associated with challenging them. Role congruity theory states that when employees fail to fit stereotypes, they will be evaluated more negatively than employees who fit into stereotypes. As a result, minority group members may face more negative consequences when engaging in the same behaviors as majority group members.

Taken together, these concepts align with research around psychological safety. Minority group members are often reluctant to express their ideas, questions, and concerns, or be their authentic selves for fear of reprisal or negative repercussions.

Scientific Analysis

In 2021, we conducted statistical analyses on our Inclusion Template questions to assess usage and validity. While most questions performed well against scientific standards, we did identify several questions that did not meet our criteria. As a result, we considered updating the question text, moving the question into a different factor, or eliminating the question entirely. One question that was updated is:

“I know where to find information to do my job well”

  • This question was one of the least likely to be used by customers from our Inclusion Template, and analysis revealed that its previous factor, Opportunities & Resources, suffered from inconsistent measurement, meaning the questions didn’t ‘hang’ together.

  • We adjusted the wording to align stronger with the literature on organizational justice and informational power.

To see what else what updated, check out the section summarizing the 2021 changes.


Demographics: How We Created the Survey's Demographics and the Messages They Send

Keep in mind that, similar to all employee surveys, the questions you ask in a survey are bi-directional; while you are collecting feedback from employees, you are also communicating to employees with the questions and demographics you include (and those you leave out). We put a lot of time into our demographics because they send a strong signal to your employees about your diversity ideology. With this thinking in mind, many practical and pragmatic considerations factor into how we iterate on our template demographics.

More Inclusive Options

If you decide to list “Man” and “Woman” as the only options for gender identity, you are communicating to your employees that you believe that gender is binary. The Inclusion Survey template provides 3 gender identities to choose from, 4 sexual orientations, and many race/ethnicity options. When choosing the demographics for your own Diversity, Equity, and Inclusion Survey, we suggest administrators consult with Employee Resource Groups, your legal team, or any other resources you have available to you if you want to make any changes to our template's demographics.

Inclusion2.png

Confidentiality & Reporting

Ultimately, our unique identities can be complex and checking a box doesn’t always feel inclusive to everyone. We could have provided even more granular options in some areas, however, we had to consider the impact for reporting. We provide demographics at the level where we believe most organizations will have enough responses to identify organizational patterns, protect confidentiality, and provide meaningful feedback for analyzing results and taking action.

Consider your own employee base and where you may need to condense options or expand them. For example, our template lists “Black/African-American” as a single choice; if you have a large population and want to understand the nuance behind how this population is feeling, consider splitting out Afro-Caribbean or other identities from the African Diaspora.

More Than Visible Attributes

The Inclusion Survey template promotes the thinking that diversity goes beyond visible traits. By including structural aspects, organizations can foster a culture that includes a broad and encompassing definition of diversity. We included Veteran Status, Family Status, Disability Status, and Socio-economic Status (measured by the degree of education obtained by parents/legal guardians).

You may decide to add more factors, such as religion/faith or political affiliation, as we have seen this in some Diversity, Equity, and Inclusion surveys that have been run on the Culture Amp platform. While these factors are not listed in our template, you may decide to include them in your version after consulting with your team.


Format: Why the Survey Is Set Up as an Unattributed Survey

Given the sensitive nature of the topic, we believe that running the Inclusion Survey in the unattributed format may give more assurance to survey takers that no identifiable information is tracked to survey responses or accessible to administrators. We recommend this approach particularly for organizations who have just begun to build their employee feedback strategy or diversity, equity, and inclusion strategy.

However, you are also able to run the survey attributed, which is particularly useful if you want to view personal demographics in conjunction with demographics included in an employee's data file or to use the Inclusion Survey to test the impact of interventions and nudges related to DEI. For example, an organization may want to survey as a baseline, expose half of the workforce to allyship training, and survey again (seeing if the allyship training had positive impacts on the 50% of employees that were granted access to the training). This sort of efficacy analysis is not possible under the unattributed/completely anonymous format because responses are not tied to individuals in any capacity. However, you could use a self-report demographic and ask employees if they took part in the allyship training if you choose to run an unattributed survey


2021 Changes to the Survey

In 2021, prompted by the results of scientific validation of our Inclusion template, several changes were made.

New Outcome Index

For the last 4 years the outcome index has been Engagement to help DEI folks make the case that this work truly is related to business outcomes that are already tracked (like Engagement). While Engagement is a core part of the employee experience, it is not necessarily a direct outcome of a diverse, equitable, and inclusive work culture. In the last year many more organizations are wanting to get a pulse on DEI in its own right and see the true drivers for Inclusion. Inclusion is now the outcome index, which encompasses questions that directly measure the outcomes of inclusive behaviors.

The Inclusion factor contains three questions from the previous Belonging factor, with the addition of two new questions. Belonging is an important outcome of inclusion, but a more robust measure of inclusion requires that we more fully incorporate the concept of authenticity, in that employees are able to express their unique selves and are valued for it.

“I feel valued for the unique contribution I can make to ACME”: This question was added to the factor as it measures the perceived organizational worth of an employee’s work that is based on their unique traits, skills, and experiences, a core facet of authenticity.

“I feel safe to take risks at ACME”: This question measures an employee’s perception of psychological safety within their organization, whether or not they feel safe to learn, contribute, or challenge the status quo. The word “risk” may foster negative connotations for some organizations, however, it accurately describes the experience of employees from under-represented or marginalized groups in that their actions and words are often under greater scrutiny and compared to different standards than their majority counterparts. For example, if a woman engages in the same assertive leadership behavior as a man, she may be viewed more negatively than the man, and it may not yield the same results. For her, the behavior carries more risk. Differences in risk perception pinpoint work to be done creating a truly inclusive company culture.

Items Removed

The question “Even when something bad happens (e.g., when I get critical feedback from my manager, I have a negative social interaction with a peer, etc.) I don't question whether or not I belong at ACME” was removed from the template. The reasons for its removal are 1) its length indicated poor wording in measuring its content area, 2) the passive nature of the question, whereas something “bad” must have happened in order to answer the question.

The question “ACME enables me to balance work and personal life” was removed from the template, because it statistically was unrelated to the other questions, and there is a Work Life Blend factor on the Engagement survey template, which makes this question duplicative.

Items Added

A new question was added to the Diversity factor, “I am comfortable sharing my personal background and experiences at ACME.” This question speaks to the willingness of employees to share important and unique aspects of their identity & experience, such as disability, SES, education, immigrant experience, political affiliation, etc. Essentially, this is the lived experience of an employee in a company that values diversity and has established psychological safety.

Factors Changed

The Equity factor contains three questions from the previous Fairness factor, along with one question from the previous Opportunities & Resources factor. The wording for “I know where to find information to do my job well” was changed to “I am provided the information I need to do my job well.” The original wording put the burden of obtaining information on the employee and may have pointed to documented information, such as job descriptions or SOPs. However, one aspect of information sharing that is critical to an inclusive culture is the provision of information from those who know, to those who need to know. The changing of the wording places the responsibility of information sharing on the source.

The Growth factor contains two questions from the previous Opportunities & Resources factor, as well as one question from the previous Fairness factor. “People from all backgrounds have equal opportunities to succeed at ACME” was added to this factor as it is a logical outcome of a growth mindset, that all employees can grow and succeed.

Demographics Changed

From many organizations of different sizes using the Inclusion survey, we found that we needed to strike a new balance between our demographic options being inclusive, while also providing large enough groups for organizations to report on. For that reason, two demographics have been condensed: Gender and Sexual Orientation.


FAQs

How will using the updated template affect trending over time?

Most of the questions remain unchanged with the updated template, so question trending is largely unaffected. For the one question whose wording was updated around information sharing, it is up to your discretion as to whether or not it should trend to the previous version. However, we recommend against it since any changes in the score may be influenced solely due to the wording. For the factors, since many factors were renamed and their questions changed, if you change to the updated survey you will not be able to compare factors over time. If this is very important to you, you can create two factors, one with the historical factor and one for trending moving forward.

How do I decide which outcome index is right for me?

We recommend using the Inclusion factor as the outcome index, which statistically and conceptually fits together better with the driver questions in the template. This will enable you to identify which questions drive inclusion at your company, which makes it easier to select focus areas and take action and improve inclusion, in the same manner as the Engagement Survey. However, if you are trying to get buy-in for your DEI program and believe connecting it to a well-established construct you have options. First, you could add the Inclusion factor to your Engagement Survey to be able to realize the relationship between these two important concepts in the Culture Amp platform. Second, you could add the Engagement factor to your Inclusion survey and flip between outcomes to see what is driving Engagement versus what is driving Inclusion with driver analysis.

What if I want to use the old version?

As with all our survey templates, you are able to edit questions to best suit your company. This includes retaining questions from the old version of the Inclusion Survey template, which you can do by adding items back in or duplicating a previous Inclusion survey. However, keep in mind the changes made to the updated version are based on our applied experience, research, and scientific analyses to create an improved way of measuring inclusion within your organization.

What alternatives can be used to assess psychological safety as part of the Inclusion Outcome?

Based on the latest academic research (Shore et al., 2011; Jansen et al., 2014) and our own comprehensive validation work we built a 5-statement Inclusion Index that captures the different dimensions of Inclusion. These include an employees’ sense of belonging, the feeling that they can be their authentic selves at work, their perceived organizational worth as well as their perception of psychological safety.

Employee’s perception of psychological safety is about whether or not they feel safe to learn, contribute, or challenge the status quo in their organization. We measure this with the statement “I feel safe to take risks at ACME” as research shows that the actions and words from members of marginalized groups carry more risk. They are often under greater scrutiny and judged against different standards compared to their majority counterparts. For example, a woman who engages in assertive leadership behavior may be viewed more negatively compared to a man. Her behavior may also not yield the same results.

However, while the word “risk” can accurately describe the experience of employees of underrepresented or marginalized groups, we acknowledge that it can trigger negative connotations, especially in environments that are characterized by a strong health and safety culture such as the manufacturing, health or construction industry. Therefore, we provide some alternative wording for this question that allows you to capture psychological safety as part of your inclusion outcome without explicitly referring to risk.

Alternative question

Benchmarkable?

I feel safe to take chances at my company

I feel safe to challenge the status quo at my company

I feel safe to experiment with different, potentially better ways to do my work at my company

If I have concerns I feel it is safe to speak up

Remember, if you decide not to include this question as part of your Inclusion outcome, but keep the other questions consistent, you will still be able to use our benchmarks to compare your scores with similar organizations or regions once these are released.

Did this answer your question?