Hello r/MachineLearning,

Disclaimer, I am a data engineer. I hope this post have its place here, I apologies in advanced if not.

What is your experience regarding the industrialization of your ML code ? Who do you collaborate with?

I will start : I’ve worked with data scientists to release in production machine learning pipeline. We have been improving our collaboration over the past year, and thus delivering more efficiently. First by drafting a blue print of a machine learning pipeline, then converging on common tools, finally by sharing our knowledges on our different skills.

Our shared tech stack is mainly: BigQuery, Apache Beam (to distribute the preprocessing), docker, tensorflow/Keras, Apache airflow.

This has lead to data scientists being more autonomous regarding scalability.

I think collaboration is the key.

EDIT : cleaning, removing medium link.