Can a Machine Predict Work Outcomes? Screening in the Age of Artificial Intelligence

Imagine this scenario: you’re working in HR at an organization hiring for an open position. By the application deadline, you’ve received 500 resumes, varying in formatting, clarity, level of detail, and relevance to the job. You’re given a week to screen these applications and forward 25 to the hiring manager. Assuming you’re busy and can only spend half of your time reviewing resumes, this gives you, on average, 2.4 minutes for each candidate.

Although this scenario may seem daunting, it’s familiar for many recruiters. Facing large volumes of applicants and pressure for efficiency, they may rely on a handful of cues or heuristics when screening resumes, increasing the likelihood of biased decisions and overlooking qualified candidates. To avoid these issues, organizations increasingly use recruitment software that automatically scans resumes for keywords specified by a hiring manager. Although these approaches may accelerate the screening process, they are limited in their ability to detect the context in which keywords are used and are rarely based on research. This means that the choice of keywords could still be influenced by hiring managers’ biases and may not effectively predict performance outcomes.

Recently, Sajjadiani and colleagues (2019) demonstrated that machine learning techniques can address these issues and provide a more systematic approach to evaluating applicants’ work histories. Specifically, they identified variables that have been empirically shown to predict job performance and turnover and trained computer algorithms to score job applications on these variables. These scores were shown to predict applicants’ future work outcomes and lower the risk of adverse impact, resulting in fairer hiring ratios across minority and majority groups.

Machine Learning, AI, and Algorithms… Oh, My!

To understand the study, it can be useful to have an idea of what exactly machine learning is. A subset of artificial intelligence, machine learning (ML) is a set of techniques through which a computer, provided with data, “learns” to optimize an algorithm (i.e., a set of rules to follow to solve a problem) through trial and error to make it as accurate as possible. ML is particularly useful when analyzing text data. It can identify patterns of words that belong to categories or consider the context or intent of a text.

Applying ML to Applicant Screening

It is this type of text-based ML that Sajjadiani and colleagues used to try to improve the screening process. They recognized that the issues with current screening approaches reflected a lack of consensus on how to predict work outcomes using information from applicants’ work history. In other words, organizations know that information about applicants’ previous jobs is important; they just don’t yet know how to organize it into meaningful categories that can predict applicants’ future success.

To address this issue, the authors first turned to research. They identified three variables that have been shown to predict job performance and turnover: work experience relevance, tenure history, and applicant attributions of previous turnover. Then, they used ML to score job applications on these predictors. To assess work experience relevance, they trained an algorithm to match applicants’ self-reported previous job titles and descriptions to occupations in the Occupational Information Network (O*NET). This large database contains information about job titles, job descriptions, and the required knowledge, skills, abilities, and other attributes (KSAOs) for various occupations. The algorithm then created a score representing the match between the KSAOs required in applicants’ previous jobs and those required for the advertised position. Tenure history was calculated by taking the average difference between applicants’ previous job tenures and the median tenures for the matching O*NET occupations. To evaluate applicants’ attributions for leaving previous jobs, the researchers trained another algorithm to categorize applicants’ self-reported reasons for leaving into four types of attributions: involuntary, avoiding bad jobs, approaching better jobs, or other reasons.

To evaluate their algorithms, the researchers used data from 16,071 job applications for teaching positions in a school district, including information about candidates’ work histories and turnover attributions. Using their algorithms’ predictor scores, the researchers compiled a list of recommended hires and compared it to the actual hires made by the district. Notably, they found little overlap between the two.

The researchers used job performance and turnover data collected by the district to test their algorithm’s ability to predict outcomes. They found that, on average, individuals on the algorithm-recommended list performed better and showed less turnover. They also found that while both methods showed a low risk of adverse impact for gender, the ML approach slightly lowered the risk for race.

Is ML the Solution to Better Screening?

These findings suggest that using ML to score job applications on variables known to predict desirable work outcomes can be a useful screening approach that could improve hiring decisions. Compared to conventional methods, it uses techniques that are better attuned to context in text-based data, slightly reduces the risk of adverse impact, and is grounded in research, increasing its ability to predict applicants’ future work outcomes.

Although the results of this study are convincing, certain limitations of ML prevent it from being a miracle solution to better screening. One limitation is that ML techniques require a lot of data to develop valid and generalizable algorithms. For instance, the algorithms described in this study were only developed and tested on teachers. Before being used in other contexts, they would have to be refined, trained with more data, and re-evaluated for validity.

Another concern is that ML algorithms can only be as unbiased as the training data they are given. This means that using these algorithms to screen applicants could actually reproduce and perpetuate structural biases. For example, Amazon recently scrapped an attempt to use ML to develop a recruitment tool because its algorithm became biased against women after being trained on predominantly male resumes. Taken together, the potential benefits and limitations of this new research suggest that ML has the potential to significantly enhance the screening process, but only to the extent that it is done right.

Reference

Sajjadiani, S., Sojourner, A. J., Kammeyer-Mueller, J. D., & Mykerezi, E. (2019). Using machine learning to translate applicant work history into predictors of performance and turnover. Journal of Applied Psychology, 104(10), 1207–1225. https://doi.org/10.1037/apl0000405