[D] How does your machine learning algorithm are indrustialized ?
Disclaimer, I am a data engineer. I hope this post have its place here, I apologies in advanced if not.
What is your experience regarding the industrialization of your ML code ? Who do you collaborate with?
I will start : I’ve worked with data scientists to release in production machine learning pipeline. We have been improving our collaboration over the past year, and thus delivering more efficiently. First by drafting a blue print of a machine learning pipeline, then converging on common tools, finally by sharing our knowledges on our different skills.
Our shared tech stack is mainly: BigQuery, Apache Beam (to distribute the preprocessing), docker, tensorflow/Keras, Apache airflow.
This has lead to data scientists being more autonomous regarding scalability.
I think collaboration is the key.
EDIT : cleaning, removing medium link.
submitted by /u/schrute_dataeng