I’m involved in evaluating candidates for positions in ML and have been asked how to test their practical skills. We are looking at both potential ML engineers and research scientists. The positions are quite junior, so project management is not something we need to look for. We have a quite broad set of applications going on spanning computer vision, NLP, time series analysis and tabular data.

What I would like to do is to formulate a task (or tasks) that I can use to test the applicants’ practical problem solving abilities. My problem is that the tasks I work with involve a bit sensitive data that I can’t share. Open datasets on the other hand are often already formatted in a “ready-to-model” format with plenty of publicly available solutions, which is usually not the case in real-life projects.

I would like to discuss options that can also serve as a resource for struggling research scientists involved in recruiting. How would you formulate a task that:

• Is solvable within reasonable time before a job interview
• Is not more hardware demanding than that you can solve it in a Collab notebook
• Is not trivially solvable by reading online tutorials
• Shows that the applicant avoid some common pitfall encountered in practice (data leakage, imbalanced datasets, test set peeking)
• Shows that the applicant actually has some practical know-how?