int 0×30:

I was recently in need of a clean and simple mechanism for doing realtime blob detection using OpenCV. One technique that would at least get me halfway there would be to use something like OpenFrameworks, which contains a sample project for this purpose. However, when writing applications that have no reason to run on anything other than a particular OS (in my case, OS X), it makes sense to leverage as much of the native OS APIs as possible for performance. In addition, I was also drawn to the possibility of being able to take the video frames and pre-process them with some CoreImage filters before sending them to OpenCV.

The right answer is to use CoreVideo in conjunction with OpenCV to accomplish this task. Unfortunately, I found a number of discussions on the web asking how to accomplish precisely this task — apparently with little success. To be honest, CoreVideo still scares me a bit, but, according to Apple, it’s the best way to deal with video on OS X. I suspect the main reason that I couldn’t find an existing solution to a relatively simple problem is mainly due to the complexity of the CoreVideo APIs compounded with the issue that OpenCV can be challenging to compile for OS X.

Surprisingly, this task is incredibly simple. In less than 200 lines of Objective-C, I’ve got a CoreVideo-based OS X application that does blob detection using OpenCV. My performance concerns were also justified, as it uses ~10% CPU to accomplish the same task the OpenFrameworks does with nearly 80%. The only downside is that even though I’ve been able to inject OpenCV into the CoreVideo pipeline and obtain some great results, there doesn’t appear to be any way to insert CoreImage into this pipeline as well. The benefit of CoreImage filters, of course, is that they are running on the GPU. This turns out to be a double-edged sword however, as the only way to get the pixels back from the GPU (or wherever Apple keeps them), is to copy them back out. And copying is sloooow.