I was struggling to optimise my vision pipeline on iOS. It turns out the throughput of CoreML is really low on small neural network, namely it is not designed to do inference on large stream of data with a small network.

​

I tried OpenCV DNN out curiosity, was not expecting much, thought I hit the performance ceiling of iOS device, but it did surprised me and increase the throughput by two fold.

​

I also tried running OpenCV DNN on my development computer, and is able to observe similar performance increase. Major deep learning framework seems do not optimise much on CPU inferencing.