In this product update Diversity Atlas’ Cultural Attaché, Quincy Hall, discusses anonymity in DEI work and surveys. 

Is Survey Anonymity Good or Bad?

It depends on who you ask. At Diversity Atlas, we believe anonymity means better data because the participant is more likely to answer hard questions truthfully on an anonymous survey, and we also believe anonymity means more data, in that when we offer an opt-in survey that asks ‘sensitive questions’, the guarantee of anonymity lifts response rates.

But what of the administrators who are implementing our survey in their workplace? Many, in the interests of the best data imaginable, really want the ability to link responses to internal HR records. As a company, we understand this, but the fact remains that if staff don’t care to offer insights as to their sexuality and religion to their employer, but want to at least be counted, then anonymity becomes paramount. 

So, organisations looking to create a diversity profile by way of self-ID across a range of demographics, attributes and cultural markers (from sexuality and gender to religion and disability status) must first consider whether to deploy such a survey from within their own HR systems and thus be able to identify the participants either directly through unique ID markers or by granular breakdowns of the data, or whether to engage a third-party data collector who can ensure anonymity. 

The very nature of the survey questions, several of which could be termed ‘sensitive information’ urges our customers towards anonymity as the preferred option, alongside our inclusive datasets, which guarantee the most high-resolution self-ID possible.

There have been hundreds of studies that touch on this very topic, including examinations of drug use in teenagers (ie: will anonymity lead to better data?) and studies of paper surveys vs. electronic surveys. 

Survey theory is potentially thousands of years old. According to legend, in the 16^th century BC, the first King of Athens, King Cecrops, ran a census count that involved the Athenians dropping in one stone for each person. (Note the anonymity feature.)

The absence of established and long-tested research on a survey platform like ours is however notable; this is because the Diversity Atlas survey is a world first, given the breadth and depth of the self-ID datasets. The comparison between anonymity and identification should also factor in the nature of the questions delivered and the answers offered by the Diversity Atlas survey. 

A simple quote by Chantrelle Nielsen examining best practice in diversity data gathering published recently in Harvard Business Review sums up the above sentiment:

“Default to anonymity and aggregation. There is more to be learned by examining the relationship between sales and marketing as a whole than there is by examining the relationship between James in sales and Elliott in marketing. Analytics initiatives are not the place for satisfying personal curiosity.”

Harvard Business Review

The article is also a great introduction to the argument, which encapsulates our company approach:

“[Participants in the survey] are not technically anonymous because the [company] is typically receiving … data that indicates the employee’s business unit, tenure etc …Depending on the number of these demographic questions, this approach can generate … concern and scepticism regarding confidentiality… [One way] is to rely on an independent third party to administer the survey and analyze the results, with the understanding that the company may have aggregate information about respondents’ demographics, but not individual information that might allow them to identify respondents.”  

(ibid.)

One peer-reviewed paper that we certainly took notice of (Robertson, R.E., Tran, F.W., Lewark, L.N. et al. Estimates of Non-Heterosexual Prevalence: The Roles of Anonymity and Privacy in Survey Methodology. Arch Sex Behav 47, 1069–1084 (2018) states at the outset:

“Focusing specifically on sexual orientation prevalence—a measure that is sensitive to the pressures of heteronormativity—the present study was conducted to examine the variability in U.S. estimates of non-heterosexual identity prevalence and to determine how comfortable people are with answering questions about their sexual orientation when asked through commonly used survey modes. We found that estimates of non-heterosexual prevalence in the U.S. increased as the privacy and anonymity of the survey increased.”

(ibid.)

Further, a different study found that:

“Corroboration of positive results, unintended outcomes, and counter-intuitive results by a third party will carry more influence with administrators than will the reports, analysis, and suggestions of diversity management initiatives provided by trainers and consultants,”

(ibid.)

and also that:

“To those who are optimistic, the path toward better understanding and knowl­edge about the impact of diversity management initiatives is clear. It points to the need for rigorous research with better designs, and theoretical frameworks, expanding the composition of research samples beyond African-American and white groups, study­ing diversity management programs over a period of time, and using more sophisti­cated statistical analysis.”

(ibid.)

Although dating back to the year 2000, the message is as relevant today, and it is in this spirit that we developed Diversity Atlas. 

Note: There are also studies that suggest encouragement can overcome the drop in response rates when offering a non-anonymous survey through monetary and non-monetary incentives, though we do not recommend such practices.

It is true to say that there also exists academic research that argues the opposite, but none refers specifically to the many identity parameters we survey, such as sexuality, religion and disability.  For instance, a paper from 1977 looking at anonymous vs. identified survey responses of USA school teachers found that: “In a two-by-two factorial design, anonymity and social setting had no appreciable effect on response…”. The paper concluded with: “Respondents receiving the questionnaire at their school address had a higher return rate than those receiving it at home.”

Our Findings

Our own experiences at Diversity Atlas support the above hypotheses, and below I will list our most pertinent findings, observations and responses.

It is however important to note that our datasets are the largest commercially available globally, so we cannot offer a direct comparison. For most companies, their staff have never been asked these questions, in this way, before. For example, we have 7,900+ religion/worldview options, including 140+ specific ‘Catholic’ denominations, and it is difficult to compare that level of self-ID data with other surveys wherein ‘Christian’ may be the only option. 

Some findings from our surveys are as follows:

• When asking staff about their sexuality, religion, disability status and so on, we have found that staff are more likely to answer these questions if assured anonymity from a third-party data gatherer than they would if asked by internal HR departments. Studies show that ‘sensitive information’ instantly drops participation rates, whether that be about sexuality or drug use, but anonymity encourages greater self-ID, and many of our customers have reported greater response rates for our survey than from their own efforts in previous years. Finance, law, tech and healthcare industries have all reported similar results. 
• The degree to which results vary (HR vs. third party) varies from country to country, industry to industry, function to function, and fluctuates between levels of hierarchy (e.g. a senior manager entrenched in a position may be more inclined to answer questions as opposed to a lower-level / new staff member). There is no steadfast rule, with the only obvious ‘pattern’ being the increase in responses from an anonymous third party, as reported by our customers across several sectors when looking at previous attempts to gather data.
• It also relies on the level of trust the staff person has for their company and its leaders, that they have engaged a reputable third party who will protect their anonymity. The communications strategy for this is integral. 
• There is an interesting global cultural shift. On one hand, people are wanting to self-ID, to be seen and heard, counted and they want to be free to identify in the manner of their own choosing with regard to identity markers such as gender, sexuality, worldview and so on. But on the other hand, their data and identity privacy is paramount. They don’t want their identity stolen, sold or leveraged for nefarious interests; they want to choose who knows. Prima facie, the two wants counter each other, and it is the guarantee and delivery of anonymity that provides the link between ‘whole-self identity’ and ‘privacy’. 

In the end, we believe keeping the participants’ identities anonymous is the right thing to do when collecting sensitive information as this will allow the participant to answer truthfully and produce the most in-depth diversity data possible.