Curious Conversations, a research podcast

Music

Travis

When we talk about the future of work, a lot of the conversation generally focuses on artificial intelligence possibly taking over the jobs of humans.

But what role might artificial intelligence play, or even currently be playing, in deciding who gets those jobs in the first place? With so much of the hiring process having already migrated to online formats, I'm curious how much artificial intelligence is already being used in those settings. Should we expect for that to increase in the future? How do its results compare to a more traditional format? And perhaps most importantly, do I need to start preparing to interview with Johnny Five?

Thankfully, Virginia Tech's Lewis Hickman was willing to share his insights with me on this very topic. Lewis is an assistant professor of industrial organizational psychology in the Department of Psychology. His research focuses on the intersection of technology and work with an emphasis on applications of machine learning and artificial intelligence to the organizational science of practice. A lot of Lewis's current research involves looking at artificial intelligence and seeing how effective it is in assessing automated video interviews for job applicants.

So, we chatted a little about how artificial intelligence is currently being used in the hiring process, a little about how it's historically been used, and how he thinks it may be used in the future. We also talked a little about the upsides and the downsides of eliminating that human -to -human interaction, both when it comes to the hiring process and just interacting with people in general, which I believe led to the first reference of South Park on the podcast.

I'm Travis Williams and this is Virginia Tech's Curious Conversations.

Travis

 I'm curious right off the bat, how is artificial intelligence being utilized during the interview process?

Louis

Oh, well, it's being used by some organizations, not all organizations. I thought initially your question was going to be how's it being used in hiring in general? Because for around 20 years, people have been using some form of a computer. It's generous to call it artificial intelligence, especially 10, 20 years ago to screen people's resumes in larger organizations. But in terms of interviews, there are organizations and I think it's something like 98 % to Fortune 100 and probably pretty large proportion of Fortune 500 companies at some, for some jobs in some units are automatically scoring interviews with artificial intelligence. And this could mean a variety of different things. I think the most robust and valid way that people are using that is to score question level responses and try to replicate human ratings on behaviorally anchored rating scales, which are a method of scoring interviews that enhances reliability, reduces bias, tries to make it more fair and predictive of job performance.

There are also AI interviews that are scoring people on their big five traits, for example, extraversion, agreeableness, conscientiousness, emotional stability, openness, usually on behavior across the entire interview. So a little bit different approach. There have been some people doing some research on like, oh, can we have like AI agents like interacting with people?

At least so far, that's not what's going on. It's just using the AI to score. Today, usually only your spoken response. Historically, which means three years ago in the AI world, there were companies also analyzing nonverbal and paravirbal cues. So your facial expressions, how your voice sounds, are you talking really fast? Are you talking really slow? And using that also as part of the evaluation.

But most everybody's moved away from that now because there's a lot of legal concerns around using nonverbal and paraverbal cues.

Travis

So I guess I should back that up then and ask, prior to using it during the interview process, did we use some sort of, I guess, loosely called artificial intelligence to, I guess, qualify candidates?

Louia

Yes. So kind of when the job started to be posted on the internet, organizations started getting a lot more applications for job openings. And this creates pressure. You have only so many HR people and hiring managers, but now you've got a lot more work to do, right? Bigger stacks of resumes to sort through. And so it was around 20 years ago that some of the first systems for automatically screening, I would say, resumes emerged. And these were pretty dumb tools. That's why I said it was generous to call them artificial intelligence. Because usually they were just looking for a few keywords. And then if those keywords were there, you passed and you got onto the next stage. If they weren't, it just cut your resume out of the pool. For years, we've known that these more simplistic systems can be beaten by candidates. For example, you can insert a bunch of white text and really small font on your resume and put in all the buzzwords and keywords. And then suddenly you're passing those filters. Today, some of the resume tools are more sophisticated and that they're looking at the job opening, seeing what knowledge and skills are required, and then checking your resume and trying to extract that and evaluate the match between the two. But I'm sure there's still people out there using keyword -based approaches, unfortunately, because it just takes time for people to go, oh, OK, we should use this new tool as opposed to the one we're familiar with.

Travis

So now we're using it, or I guess hiring companies or hiring folks that hire people, are using it to...not necessarily interview people, but to evaluate the interview process.

Louis

Yeah, so separate from AI scored interviews, there has been, well, even before the pandemic, but during the pandemic, the technology really exploded in use. Online interviews, right? And those could be two way like we're doing right now. There's two of us on the call. But those can also be one way interviews where you just sit down at your computer, some questions come up on the screen, and then you record your response to the question.

Then you get your next question, record your response. And so that's just called an asynchronous interview or one -way interview, right? Done on your own, whenever it's convenient for you. Really convenient for the hiring managers too. Say you've got a bunch of applicants and you want to give more of them the opportunity to interview than you normally would. You can use that one -way interview to give a lot of people an opportunity to interview. And then you can review that interview at your convenience, just like they can record it at their convenience.

Instead of having humans review the interview and rate it, you could have AI review the interview and rate it. And again, AI, I like to joke that there was just nothing intelligent about artificial intelligence in general, because it's just statistics. That's all that's going on in these systems is they're using statistics to say, well, the response contains this. That means it's a four. It's a two, whatever it might be.

Travis

The story that was in Virginia Tech News, I guess, recently, you evaluated some of this and I'm curious, what did you find? Like how good a job did some of these systems do in evaluating candidates?

Louis

Yeah, and in that paper we trained on data that I collected, machine learning models, it's a subclass of artificial intelligence as a field, trained models to predict the big five traits. So we trained that to both predict self -reported big five traits so people complete their little...

I tend to be very disorganized, strongly agree to strongly disagree. And we also trained models where we had several people review the interview, rate their big five traits, and then we're trying to replicate the average of those multiple humans. When it came to replicating the self -reports, it basically didn't work at all. I think there are several reasons for that. A big one is that we tend to have pretty positive perceptions of ourselves. That's a psychological mechanism that helps us be healthy individuals.

If you have a negative view of yourself, it's hard to be happy and productive and present and aware. If you have a positive view of yourself, those things are a lot easier. So we tend to have positive views of ourselves that maybe aren't so accurate. In addition, the self -reports are hard to replicate because if I'm completing a self -report and I'm judging, how disorganized do I tend to be, my frame of reference as a professor at Virginia Tech, probably different from the frame of reference that a undergraduate student is using for what means organized or disorganized. And that's different from what, say, a construction worker is using for their level of organization. So we have a lot of differences in subjectivity and interpreting those self -reports. But when it came to the interviewer ratings, the models did a really good job, particularly with extraversion. You tend to be talkative and energetic. They did a good job with conscientiousness. How organized and dependable and achievement striving do people perceive you to be and did a good job with openness How creative and interested in new ideas and learning are you? You didn't do such a good job with agreeableness and emotional stability, but those are also harder for people to judge from a short video You're kind of man you kind of Motivated to appear agreeable. You're motivated to appear emotionally stable in those videos. So there's maybe less to go on for judging those people are evaluating that. So what worked really well for the replicating the human interviewers, there's a company I collaborate with a lot on research that just published a paper really following up on mine, but on the technology that they do. Theirs is scoring people on each question on the competency that the question tries to elicit. And again, works really, really well. We have modern natural language processing, as we've seen with Chad GBT, for example, just does a really good job of, I don't want to say understanding language, but getting a probabilistic representation that comes, looks like understanding, right? And so when we have that now, it's pretty straightforward to replicate these human ratings. And the promise of these technologies is that instead of a tired human who, maybe their son came up into their bed at five in the morning last night, and now they're tired today from that, like me. Maybe you forgot your coffee in the morning, you had a big lunch, you're sleepy after lunch, you're in a good mood, you're in a bad mood. You evaluate people differently depending on whether they're wearing glasses, whether they're young or old. That doesn't happen with the AI. With the AI, we can give only the transcript of the interview. That's all that it has to evaluate you on. And it never gets tired. It's never grumpy.

It never has a bad day. And so it can evaluate people more consistently than humans can, both considering like within person variation and between people. If I rate somebody versus you're rating somebody, Travis, we might rate the same answer differently. But again, the AI never gets tired, always is going to give the exact same score to the exact same response. Unless the model changes, that's really the promise. It doesn't mean that it's fair and unbiased, but it means..that it can be more consistent and reliable. And so if we design it well, we can reduce that reliance on human subjectivity, which can be a random error due to fluctuations in attention. It can also be systematic error due to things like racial and gender bias.

Travis

It sounds like even with us as human beings, like how we view ourselves, maybe that would even fluctuate. So maybe that's part of the self -assessment stuff, not, because there's some days I feel great about me and other days I'm like, not so good. So that could be a day-to-day thing.

Louis

Yeah, you're exactly right. You know, we humans go through a lot of fluctuation in terms of who we are and what we are. And the kind of the hardest thing to do is try to picture what you're thinking would be like in some future state. Say you're about to have a third child like I am. It's kind of impossible to really picture and understand that and what you're going to feel about it. Similarly, it's impossible to go back in time. And remember exactly how you felt and thought at that time, how you would view events. We have the present moment and that's how we can view things and understand things. But those other states of mind are so difficult to access. And so from day to day, from moment to moment, week to week, year to year, we do change. And that's causing changes in how we respond to these scales, whether it's about us or about other people.

Travis

I'm curious, how do you think that artificial intelligence is and will continue to change the hiring and the interview process?

Louis

Yeah, I think we're going well, I mean, we can go on both sides. I think we already have concerns. It's a big concern in education, but also a concern in pre -hire assessment. The people are completing things now using, you know, chat GPT or llama or Claude or any of these large language models that are out there, right? You've Virginia Tech has short essays that undergraduates write as part of their application material. People write cover letters. Maybe they complete something similar to an essay. When they're completing an interview, if it's on their computer, there was this video a person posted online several months ago where in real time they're transcribing what's going on in the interview. It's then putting the questions they're getting asked into Chad GBT and generating answers then they're reading.

So the applicants can benefit from this and right on the one hand it could be Cheating is one way that you could use it on the other hand a way that you could use it Which I encourage my undergrad students to do is take your resume You have those descriptions of activities you did in jobs Take that put it into chat GPT and ask it to revise it for the type of job that you're applying for And often it goes from something that's clearly written by an undergraduate to something that sounds like your God's gift to corporate America. And so it can make a big difference in helping you prepare those materials and improve. Similarly, you could go and practice answering an interview, take that recording of yourself, transcribe it, put it into chat GPT and ask for feedback. And so it can be a coach to help you grow and improve because you're probably still gonna have a face -to -face interview at some point in time in the hiring process unless you're doing a fully remote job, then maybe not. So on the one hand, it could be used for cheating, which is less desirable. On the other hand, it can be used to help people get better at these things and present themselves better. We already engage in impression management on the job and when we're applying for jobs. So this is just another kind of evolution of that. Or it could be a new evolution of faking and fraud and things like that. When it comes to employer use of these technologies, it's only going to grow and it's going to grow a lot. I mean, we've talked, there's already AI for screening resumes, AI for screening interviews. I think that will become just only more common. Additionally, you might now have employers where when they're reaching out to people and encouraging them to apply for jobs, it might not even be a human reaching out to them, right? You have an algorithm that goes and crawls LinkedIn to identify people with relevant skills and experience, maybe working in the same type of job. Then you can have your AI automatically send a message from your LinkedIn account to the person and encourage them to apply for a job at your organization. And that message could be tailored, right? Because the AI can maybe see their profile, know what they're interested in, and say it in a way that's maybe been tested in prior research to be optimal for engaging them and encouraging them to apply. Now we're going to see applications of that also occurring outside of employment that are going to continue to cause issues in society, but that's one that could potentially help organizations hire people. I also know many large organizations are working very hard to try to reduce the amount of human labor that goes into the hiring process. So if you're designing the interview that you're going to use to enter, to evaluate people for a job, maybe AI can help with that instead of taking several hours or even a day of an HR person's time to create an interview guide. Maybe you can get 90 to 95 % of the way they're just using AI, giving it some information about the role and what they're looking for. And then have the human kind of do that last 5 to 10 % of tweaking it and optimizing it to make sure there's not anything strange or unusual in there. What people call hallucinations with LLMs, but they're not hallucinating. They just got the statistics wrong. That's all that it is.

And so seeing a lot of people interested in creating materials around interviews and job postings and onboarding materials. Soon there are going to be LLM agents that people can interact with to ask questions about their organization, whether it's about culture or processes and so on. They're going to be there and available on demand for people when they join jobs. Because what I've seen and what other programmers have seen, you know, the traffic on Stack Overflow, which is a question and answer forum for programming, is down drastically since ChatGPT came out. And it's because you go from, OK, has my question been asked and has somebody answered it acceptably on Stack Overflow to, I'll just ask the exact question I want to ask to these large language models and get the answer right away. So that's so powerful and can be powerful in many other settings, including, you know, asking questions before you get hired, after you get hired, and so on.

Travis

Yeah, this sounds like a lot. And if I was a new graduate, this may very well be, would probably be intimidating to me. So I'm curious when you work with graduate students, undergraduate students, folks that are heading out into the job market, what just very basic advice do you give them related to this topic?

Louis

I mean, it's still relatively rare at least to have like your interview or anything like that assessed by AI. But certainly your resumes are likely getting checked if you're applying for large employers by an AI. So you do want to go and give your resume at a chat GBT or your large language model of choice and say, Hey, here's the job I'm applying for. Here's my resume. Tweak these descriptions of these experiences to better fit the language for this job. And it will sound so much better. I had a student who was having no luck getting job interviews applying for a bunch of jobs. He did this GPT revision of his resume and suddenly it was interviewing places and found a job pretty soon after because it does make such a difference on the quality of resumes and the quality of writing because it's finding what people expect more or less in the writing. And we're not always so good at guessing that and getting at that. When it comes to then on the job, if you're doing a job where there is writing or programming, these large language models can help you do your job. And you should be using them because you can be better. Boy, the time I save programming. And I wonder, I would not be good at programming if I had this in grad school, I think. Because it can be such a time saver and improve what you're up to.

When I write, if I want to get feedback on writing, now I don't have to wait for a human to give me feedback. I can go GPT, critique and this paragraph and give me feedback. And then, you know, some of the feedback's good. Some of the feedback's bad, just like when a human gives you feedback. And then you can kind of pick and choose what you're going to do so that I think that the death of human labor is vastly overblown. I don't think that's happening any time soon, but there is a big opportunity to be a human AI collaboration and hybrid and enhance you and your work with these AI tools and increase your productivity and success in your career through doing so.

Travis

Yeah, I think that that all sounds like great advice to me. It sounds like it's a tool and you should use it to make yourself a better human. You can. Last kind of thing I'm curious about, have you or will you, are you interested in researching this topic when it comes to like, online dating apps? Is that a field where this is being used that you know of?

Louis

It's a funny question because I know people are. I think South Park, which to tell you, I go from earlier, I was talking to people about philosophy and now I'm mentioning South Park. South Park did an episode where the various characters in South Park were responding to all the text messages from their significant others using large language models and having like more relationship satisfaction from their significant other than they were having prior to doing that. But then when they get face to face and their significant other started talking about what they were talking about over text, they're like, what? What are you? Why do you say that? You know, so it's definitely possibly useful. I I've considered I'm hoping social psychologists are doing this research on you know, the AI girlfriends, which is just really concerning to me because they don't, there's no understanding from these chatbots. It's a good acting job that looks like understanding. And so to let yourself develop an emotional connection with that or feel like that's important to you is concerning, right? It is not real connection. It cannot be real connection just based on how these systems work.

But yeah, I would definitely imagine based on that South Park episode that people are certainly using it to interact with people maybe more effectively and with less effort than it took previously. And that's, yeah, I mean, we're right in addition to dating apps all over social media, Reddit, everywhere. There's so much content now that is LLM generated. It just can really amplify and accelerate the disinformation and misinformation and influence campaigns from corporations, from artists, from, you know, countries and their governments. So it's that there are unfortunately a lot of negative potential uses of these tools in addition to the opportunities they pose for making our giving us time to work on the things that we can do and, and benefit society, like any tool, right? They have the good and the bad.

Travis

Thanks to Lewis for sharing his expertise related to artificial intelligence and the hiring process. If you or someone you know would make for a great curious conversation, email me at traviskw at vt .edu. I'm Travis Williams and this has been Virginia Tech's Curious Conversations.