The last three months has seen a rapid shift in the employment market. Most of the large tech players have laid off 5% or more of their global workforce, which was unthinkable in the summer of 2022. While the medium to long-term problem of severe skills shortages in Data Science and engineering will continue, for now, the recruiting pendulum has shifted (albeit temporarily) in favour of the hiring company. As more Data Scientists enter the market looking for jobs, you will inevitably see an increase in applications. The heavy lifting in the recruitment process has now shifted from attraction to assessment.
Given the current macroeconomic volatility, and the expectation to do more with less, it’s even more important that you avoid making hiring mistakes.
What GPT 3 and other AI tools mean for hiring Data Scientists/Engineers now and in the future
Just like in every other sector, AI will change recruitment significantly in the next five years. But what does this mean for you if you are hiring and assessing candidates right now?
Setting aside the early hype, it’s far too early to say exactly what Chat GPT3 means for hiring. However, we can be fairly certain about the future direction of travel and also what the immediate effect could be on candidate assessments. Take-home tests, which are standard in most hiring processes for Data Scientists and Engineers could become problematic very quickly.
The education sector runs more tests than probably any other sector, so it might give us some clues about where we are going. Kevin Bryan, a University of Toronto Associate professor, posted on Twitter recently: “You can no longer give take-home exams. I think chat.openai.com may actually spell the end of writing assignments”.
Schools in the USA have already reacted by banning the use of Chat GPT3. Educators are so worried that in many areas, they have stopped giving out take-home essays and tests that were previously completed on home computers and are insisting that essays are completed in school with a pen and paper.
The problem with assessing candidates using take-home coding tests is that Chat GPT3 can already write basic-intermediate levels of code in several languages. It’s also reasonably competent at generating explanations of how the code works. The technology is prone to error and the code needs to be checked by a human, but it’s probably still good enough to score 60-70% in basic coding tests at the more junior end of the spectrum.
Chat GPT4 is just around the corner which means that at some point in the very near future (if not already), many take-home tests are likely to be unreliable when predicting job performance in relation to coding. This is especially true if the tests are the more basic type of coding challenges, or they are generic in nature.
So what can you do now to improve your assessment process?
Back to basics – now is the time to improve interview skills
Most hiring managers in Data Science have never done any formal interview training. One of the most important principles of successful interviewing is making sure the same set of questions are asked in every candidate interview for the same job. If you don’t ask identical questions, how do you compare candidates fairly? As a minimum, there should be a scorecard for each of the main competencies for the job and the question used to assess this should be well thought out. The best resource we have found for this process is the Mark Horstman book, “ The Effective Hiring Manager ”. It will require some work to apply it to Data Science, but it’s worth it.
1 3 2
Check if your technical assessment is up to scratch and relevant to the job you are hiring for
Have you considered if your take-home test could be compromised by an AI tool such as ChatGPT3? Perhaps it’s time to consider replacing your current test with an onsite (if possible), half-day work trial for two or three final stage candidates. This entails sitting the candidate in your team for a day (or half day) and getting them to work with someone you trust on a real-life task or project to see how they perform. With this approach, you can assess technical ability in real-time over the course of a day, without having to worry about whether or not your take-home test results are reliable. And from a culture perspective, you can see how candidates interact with the rest of your team.
Understand Data Science DNA of the person you are interviewing
We define the Data Science DNA of an individual as the key skills, mindset and inclination towards specific types of Data Science or engineering tasks. Some important questions to consider in determining this are – what type of Data Science work does the person like doing and what are they good at? More specifically, what type of Data Scientist or Data Engineer profile are they? How well does this fit the job you are recruiting for and the type of work the role requires?
For example, a Data Scientist with a strong statistics education background will differ significantly to a Data Scientist with more of a machine learning pedigree – even if both list ML and statistics as their main skills.
The Statistics Data Scientist is typically a modeling expert who pays deep attention to the distributional properties of the data sample and applies highly advanced mathematical models. They look at the dataset from the data generating process perspective and work with models that are equations-based.
The Machine Learning Data Scientist is typically an expert at using feature engineering to implement machine learning models that can achieve high levels of prediction accuracy. The main goal of the Machine Learner is to deliver fast and efficient results based on the existing ML frameworks, rather than set up equations-based models from scratch. The mathematical detail is not usually of critical importance to the Machine Learner, and neither is causality. High learning/ prediction accuracy matters most, even if the model is non-interpretable and based on artificially engineered features.
If you can identify these types of differences in the hiring process, then you can predict how the candidate will attempt to create a Data Science solution for a business problem. This will tell you what mindset they apply to the problem and then how they will execute a solution. The results will indicate whether or not they will be successful and a right fit for the role.
In the next issue, I will do a deep-dive into the eight fundamental roles that exist in a corporate Data Science team and define them very clearly.
WHAT DOES THE FUTURE HOLD?
For at least the next three-six months, the heavy lifting in recruitment will be getting the assessment part right. Attracting a reasonable quantity of applicants won’t necessarily be the biggest issue in 2023 – but now is the time to improve your assessment process so that you make the most of your hiring.
There are AI tools specific to every stage of the recruitment process that already exist in the marketplace, and combined with tools such as Chat GPT3, they are set to present both hiring managers and job seekers with both opportunities and challenges.
