Interviewing in the Age of AI

In February, Anthropic caused a stir by posting a message on its job descriptions asking applicants not to use AI in the application process.

The statements read “We want to understand your personal interest in Anthropic without mediation through an AI system, and we also want to evaluate your non-AI-assisted communication skills”. Ironic, really, given that Anthropic is one of the world’s highest-profile generative AI firms and encourages its employees to make use of AI on the job.

Irony aside, though, the story poses lots of questions about how AI fits into the jobseeking process.

Anthropic’s regardless of whether or not you’ll end up using AI when working there, they want people to be genuinely motivated to work for the company – without getting AI to generically summarise this motivation – and to be strong communicators in their own right, without needing to rely on AI.

Other businesses have warned that AI can be used to engineer the interview process so that jobs are offered to people who are incapable of doing them. Last year, a Capterra survey of UK jobseekers found that 29% had used AI to generate answers to interview questions, and that 27% had used it to complete job application tasks or assignments. 

Where else might AI creep into an interview process, though? And perhaps more importantly, given most engineers will use AI tools on the job, does it even matter if you use AI when applying?

CVs

Writing a CV is one of the first and most arduous tasks that anyone looking for a new job will carry out. In our view, this is something where, done correctly, AI can add real value.

CVs have always been a bit problematic for recruiters who believe in fair and equitable hiring – giving everyone an even chance of a job based on their ability and experience, without letting superficial factors sway them. A CV is an information-sharing tool, but everyone knows that some people are better than others at making a CV stand out. Those people might gain a slight edge over others at capturing the attention of time-restricted recruiters and hiring managers.

AI can be something of a leveller here. To demonstrate, we asked ChatGPT to write two versions of the same CV for a fictional mid-level software developer, but to make one version better-optimised to present information clearly and grab the reader’s attention.

Here’s the professional summary from the basic version:

And here’s the optimised version:

Both are fairly generic – which is something anyone using generative AI for anything needs to be on guard for – but it’s interesting that the optimised version has tweaked the same information to include personal attributes (“Innovative and detail-oriented”) and even bolded some of the specific phrases that a recruiter scanning a CV quickly might be on the lookout for. In other words, using AI to optimise a CV could reduce the chances of certain candidates standing out more than others purely due to how their CV is written (as opposed to the experience it represents). 

That’s as far as we’d suggest going with this, though. Your CV has to be an honest representation of your career, so anything that a genAI program creates for you should be double-checked for hallucinations (and, of course, ensure that whatever info you’re prompting it with to begin with is true). But in terms of refining the final product, there’s nothing wrong with using AI to optimise your CV to give your achievements the best chance possible of standing out in the right way. 

Of course, Anthropic’s wording implies that using AI in this manner isn’t what they’re looking for. The reference to “communication skills” means we can assume that Anthropic wants to hire such strong communicators that they write the “optimised” CV themselves, rather than needing AI to help them get there. 

In our view, though, that’s a little pointless. Firstly, an AI-optimised CV (as opposed to one written entirely by AI) is probably going to be hard to spot anyway. Secondly, plenty of people who aren’t confident communicators – at least in the way that makes a CV stand out – are very good engineers. 

But more to the point, CVs are just the shop window. They’re a tool for getting you an interview – which, in itself, is the real test of your communication skills. 

AI in Interviews

So how is AI changing job interviews – particularly tech ones? 

The tech blogger Kane Narraway – Head of Enterprise Security at Canva – argues that AI is rendering certain parts of the traditional coding interview redundant. 

Specifically, he says, AI will quickly make online coding tests (like Hackerrank) and basic computer science-focused technical interviews redundant. If conducted remotely, interviews like this are too easily answered with AI support. 

However, more complex interviews – like those focusing on architecture – aren’t realistically going to be displaced by AI any time soon, as the questions are typically too involved to be able to discreetly type in to ChatGPT and see what comes out.

The complexity of the interview question seems to have a strong bearing on how easy it is to fake the answer using genAI. 

interviewing.io conducted an experiment into how much of a difference genAI made when used to try to pass technical interviews. It involved asking three types of questions to candidates that were instructed to use ChatGPT to help them answer it: “verbatim” questions taken directly from LeetCode; “modified” versions of LeetCode questions, and “custom” questions that didn’t already exist online.

73% of candidates were able to find the exact answer to the verbatim questions using ChatGPT, as were 67% of the group given modified LeetCode questions. However, only 25% of the custom questions were answered by ChatGPT.

Clearly, in the age of AI, recruiters and hiring managers will need to be able to come up with their own bespoke questions, rather than relying on pre-built question banks. 

So how else is AI shaping the interview experience for those doing the hiring?

For Recruiters

Anyone who has ever been involved in recruiting knows that there’s a lot of admin involved. What ought to be the bread and butter of talent acquisition – identifying and speaking to candidates – often has to play second fiddle to the needs of logging interview notes, filling out forms, and general organisation.

AI can help significantly with these aspects. However, there is a catch.

Just as much as AI can automate many recruitment tasks, it can also enable jobseekers to mass-apply to jobs. As things stand, the efficiency gains that AI offers to recruiters doesn’t appear to outweigh the extra work created by this mass-application, given that AI can also doctor CVs so that they exactly match job descriptions.

More than that, but the fact that AI can be used to cheat on aptitude tests and in remote technical interviews means that identifying genuinely capable applicants is becoming more difficult.

“It’s much harder to filter candidates to spend time on,” one Engineering Manager told The Pragmatic Engineer. “These tools increase the amount of work recruiters and hiring managers need to do, and it’s harder to do ‘asymmetric’ interviews where the hiring company invests less time, and the candidate needs to invest more time in a task.” 

The big challenge for recruiters is differentiating between authentic answers and knowledge, and that which is AI-generated. As AI tools become faster and more sophisticated, doing so is becoming ever harder. 

The Future of AI in Tech Interviews

So what’s the solution? Narraway proposes several courses of action that he thinks companies can take. These include stopping remote technical interviews altogether, requiring some sort of spyware or other software to ensure that genAI isn’t being used by the interviewee, ignoring the issue, or changing the process to allow access to genAI.

That last option is intriguing. Surely in a world in which engineers will have access to genAI in their day-to-day jobs, it makes sense to assess their ability to use them to problem-solve?  However, as Narraway warns: “This is a huge shift change and would be uncharted territory… This is a huge hiring risk in the short term, as we still need strong coding ability to correct the AI’s failures.”

In the short term, a hybrid approach that encapsulates elements of all these is the only realistic solution. 

“One of the things we can do, however, is change the nature of the interviews themselves,” says Narraway. “Coding interviews today are quite basic, anywhere from FizzBuzz, to building a calculator. 

“With AI assistants, we could expand this 10x and have people build complete applications. I think a single, longer interview (2 hours) that mixes architecture and coding will probably be the way to go.”

As with every innovation, AI is changing fast, and jobseekers and recruiters will need to move quickly themselves in order to adapt. But in our view, in a world in which engineers have access to sophisticated AI tools, it would be crazy to try to keep AI out of the hiring process altogether. 

For more advice on how to connect with innovative companies or engineers in the age of AI, contact Oho’s team today.