https://github.com/JeiKeiLim/simple_distribute_job

I posted earlier that I have built a face detector to blur faces for videos. It was so slow that it takes about 10 hours per each video and I have 160 videos to process….. Which takes about more than two months.

I don’t think I can wait for that long. Thankfully, I have a few machines to run in parallel. And Managing which videos has been done processing which is not and merging all videos to one computer seems too much work.

So I made a simple management job management program for multiple servers to do undone work and upload the resulting video to the main server once it is finished.

Then I realized it could be used for machine learning training on multiple machines or experimentations. I know there are good distributed job management systems but it looks like it requires to install too much. All I needed was a simple job scheduler.

I’m posting this hoping someone finds this useful.

Thank you!

BTW, I’ve been running this with blurring faces on video for 2 days and it still says it will be done in 15 days. So.. I got that going for me which is nice