[P] Not sure if it fits here, I made Simple Distributed Job Scheduler with the only SSH required
https://github.com/JeiKeiLim/simple_distribute_job
I posted earlier that I have built a face detector to blur faces for videos. It was so slow that it takes about 10 hours per each video and I have 160 videos to process….. Which takes about more than two months.
I don’t think I can wait for that long. Thankfully, I have a few machines to run in parallel. And Managing which videos has been done processing which is not and merging all videos to one computer seems too much work.
So I made a simple management job management program for multiple servers to do undone work and upload the resulting video to the main server once it is finished.
Then I realized it could be used for machine learning training on multiple machines or experimentations. I know there are good distributed job management systems but it looks like it requires to install too much. All I needed was a simple job scheduler.
I’m posting this hoping someone finds this useful.
Thank you!
BTW, I’ve been running this with blurring faces on video for 2 days and it still says it will be done in 15 days. So.. I got that going for me which is nice
submitted by /u/workout_JK
[link] [comments]