The Surprising Risks Associated With Crowdsourced Salary Data

by Angelo Kostopoulos on December 12, 2016

By Al DuPree on November 29, 2016

Ask any human resources manager how they determine employee salaries and you’ll soon find out just how complex salary data has become in recent years.

In the past, HR consulting companies gathered data from a broad cross section of employers and then sold the reports to human resources managers to help them make informed decisions. Employers that could afford it had access to a handful of expensive but reliable sources for compensation-related data; employees had to make do with the federal government’s annual Bureau of Labor Statistics reports and whispers around the water cooler.

Today, thanks to the sharing economy, a multitude of crowdsourced salary data websites collect data by requesting anonymous information from site visitors about salaries, bonuses, and company reviews. Sites like Glassdoor.com, PayScale.com, and Salary.com all report on millions of anonymous user contributions and aggregate that data with everything from geography and industry, to job title and corporate function.

To the untrained eye, these crowdsourced salary data sites offer an embarrassment of riches for human resources managers — there is data as far as the eye can see, and it’s not restricted to the folks that can write big checks. However, employers who dig a little deeper into data volume and availability will soon find out that crowdsourced salary data doesn’t always live up to the hype. Serious questions remain regarding the accuracy of the data; the best way to use it; and the best way to adjust your rates when the data doesn’t quite match up with internal practices.

There are two critical drawbacks all employers should understand when considering the pros and cons of using crowdsourced salary data:

Crowdsourced Salary Data Isn’t Consistently Cataloged

Consider how frustrating it is when you are collaborating on a filing project and a colleague uses one name for a file, and you use another name. Even though you’re working on the same project, it’s impossible to achieve your goal because you’re using different names for the same thing. And that’s exactly what happens when crowdsourced salary data websites catalog data at mass scale without using consistent or universal job titles or organizational roles.

Consider the example in Table 1 below, which compares three independent sources: GlassDoor.com; the Bureau of Labor Statistics (BLS); and the HRA-NCA Compensation Survey, an annual survey conducted by an HR professional organization in Washington DC.

Please note that this data is representative of what is available on the top crowdsourced salary data platforms. The data presented is not meant to single out any sources as either more or less accurate, more or less representative, higher or lower quality, or any other such qualitative measure.

The differences in labeling alone can lead to significant confusion and often an inability to draw meaningful conclusions. The first job title, Software Developer, Applications, exists in all three surveys in very different ways. GlassDoor.com reports the Average Salary while BLS and HRA-NCA both report the Median Salary. Comparisons between GlassDoor.com and any other survey are therefore difficult, if not impossible.

BLS and HRA-NCA have significant differences as well. HRA-NCA breaks out the Software Developer, Applications, into five different levels, while BLS does not.

The second job title, Software Engineer, only exists in GlassDoor.com. One could speculate that the Software Developer, Applications (Level III) from HRA-NCA, the Software Developer, Applications, from BLS, and the Software Engineer from GlassDoor.com are probably referring to very similar jobs. But that’s all it would be:  speculation.

Method and Sample Size Heavily Impact Accuracy

Most statistical methodologies begin with a simple premise of random sampling, or choosing samples at random to make it equally likely that any particular individual item will be selected as part of the sample. When a using a random sampling, statisticians apply certain adjustments to account for errors because it’s generally accepted that some level of error will exist. And this is just where crowdsourced salary data breaks down in reliability:

• Large consulting firms often sample on the basis of previous client relationships, which is most certainly not random
• Crowdsourced salary data websites gather data from individuals who for one reason or another self-select into the sample, which could affect the data they provide (perhaps they’re looking for a job or wondering whether they are being paid fairly)

Not only are crowdsourced salary data websites not working with a random sampling of data, but they often reach conclusions based on very small sample sizes. For example, GlassDoor.com, PayScale.com, and Salary.com’s sample sizes are quite small relative to the entire population. Comparatively, BLS offers the best overall sample size while HRA-NCA stratifies the data according to levels supplied by its survey respondents.

---

“Assuming your budget can support it, we recommend using at least three reliable survey sources for most pay decisions.”

---

The Best Alternative to Crowdsourced Salary Data Sources

Considering the average company spends forty to eighty percent of its gross revenue on salaries and benefits, it’s more important than ever to ask whether or not crowdsourced salary data holds true statistical value. After all, aggregating errors of only one or even one-half percent can result in significant errors in budget estimates. And those same errors, when taken individually, might result in the loss of a key contributor to a competitor when an employee finds significant differences between what they make and what they’ve found on a crowdsourced salary data site.

Depending on the sector of your business, you may spend between 40 to 80 percent of gross revenues on employee salaries and benefits combined. Salaries alone can account for 18 to 52 percent of your operating budget.

-Society for Human Resources Managers

Instead of attempting to navigate these disparate data sources, the most strategic and stable approach to setting employee pay rates is to secure a high-quality salary survey from a reputable source. While there is an associated cost with most surveys, they vary dramatically according to your market, so you can often secure a survey at your budget if you shop around.

To get started, look for recommendations from local chapters of professional human resource associations, like the Society for Human Resource Management (SHRM) and WorldatWork. Then consider the following elements when interviewing a survey partner:

1. Is this data sufficiently granular in my market? In trying to cover the entire country, many large companies are unable to include large sample sizes from any one market. Be sure that your survey contains a large sample size from your particular market for the jobs that interest you.
2. Is my industry well represented?  It’s clear that companies operating in the same markets, often competing for the same talent, aren’t always subject to the same labor market conditions. Be sure that your survey contains a reasonable representation of your industry.
3. Is this a higher quality survey? The most effective surveys will provide enough information for the user to make a determination as to its usability, such as:

• Average or median as central measures, but also 25^th and 75^th percentiles
• Enough breakdowns by scoping parameters (company size, location, revenue, etc.)
• Sufficient descriptions for each job to ascertain general job function, some sense of the level, which correlates to years of experience; some sense of the supervisory responsibilities, if any, and some sense of the educational qualifications, if any.

Having identified your base survey(s), selectively compare what salary ranges they indicate for jobs that have been deemed critical by senior management. The rationale for this is twofold:

1. Account for and explain all survey data differences. These are, after all, your basis for how you intend to approach the labor market. Any significant differences in source data should be addressed first.
2. Compare your survey data to publicly available survey data. While crowdsourced salary data isn’t reliable enough to provide the basis for all of your budgeting decisions, reviewing available data will help you prepare for counter offers from prospective employees and address existing employees who feel that their compensation is lagging the market.

Which leaves us with a much clearer picture of the purpose of crowdsourced salary data: it’s valuable information to help you compare and refine your decision-making process, but it’s simply not statistically relevant enough to single-handedly support your salary decisions. The more informed and prepared you are, the better decisions you can make, and high-quality salary surveys from a reputable source are still the best informants in the industry.

Do you want to make the most informed and strategic salary decisions?

Introducing Advantage 360, the advanced algorithm technology that will help you understand how your HR practices will help you gain an advantage in your sector.

About the authors:

This article, courtesy of Al DuPree, COO of AKRON,Inc and Jennifer Barbee, VP, Marketing. AKRON, Inc. is an HR data analytics company located in Washington, DC.  AKRON is a leader in providing substantiated, empirical data to empower HR departments in their plight for organizational excellence and talent retention. Since 2004, AKRON has been the administrator of the HRA-NCA Compensation and Benefits surveys.