Here at NSHipster, we're all about diving into the darker corners of Objective-C to learn something new about the systems we interact with every day. Often, this means sifting through Apple frameworks or language features (it is, after all, a lot of what it means to work in Objective-C). However, on occasion, it's nice to take a look to the burgeoning landscape of third-party libraries and frameworks (and there are some truly remarkable ones) for a glimpse of what's new and great outside of Cupertino.
This week, we'll be taking a look at one of the most impressive open source projects you'll find: GPUImage. Buckle up, NSHipsters—if you're not careful, you may well end up creating a camera app by the end of the article.
GPUImage is a BSD-licensed iOS library written by Brad Larson that lets you apply GPU-accelerated filters and other effects to images, live camera video, and movies.
GPU vs. CPU
Every iPhone ships with two processors: a CPU, or Central Processing Unit and a GPU, or Graphics Processing Unit. Each processor has its own strengths, and modern chip architecture (like in the iPhone's A4) integrate the CPU and GPU onto the same physical die.
When you write C or Objective-C code in Xcode, you're generating instructions that will be handled almost exclusively by the CPU. The GPU, by contrast, is a specialized chip that is especially well-suited for computation that can be split out into many small, independent operations, such as graphics rendering. The kinds of instructions understood by the GPU are quite different from that of the CPU, and as such, we write this code in a different language: OpenGL (or specifically, OpenGL ES on the iPhone & iPad).
Check out Jeff LaMarche's GLProgram OpenGL ES 2.0 book for a great introduction to OpenGL ES and the rendering pipeline.
Comparing the performance of GPU-based rendering to CPU rendering for something like video, the differences are staggering:
CPU vs. GPU Frame-Rate (Larger FPS is Better)
Calculation
GPU FPS
CPU FPS
Δ
Thresholding ⨉ 1
60.00
4.21
14.3⨉
Thresholding ⨉ 2
33.63
2.36
14.3⨉
Thresholding ⨉ 100
1.45
0.05
28.7⨉
"Oh, so it's like Instagram?"
Let me put it this way:
To put it another way, within GPUImage's APIs lay thousands of camera apps, just waiting for the right combination of filters and a little spark of imagination.
Here's a table of the 125 (!) filters that come with GPUImage:
Color Adjustments
Image Processing
Blending Modes
Visual Effects
Brightness Filter
Exposure Filter
Contrast Filter
Saturation Filter
Gamma Filter
Levels Filter
Color Matrix Filter
RGB Filter
Hue Filter
Tone Curve Filter
Highlight Shadow Filter
Lookup Filter
Amatorka Filter
Miss Etikate Filter
Soft Elegance Filter
Color Invert Filter
Grayscale Filter
Monochrome Filter
False Color Filter
Haze Filter
Sepia Filter
Opacity Filter
Solid Color Generator
Luminance Threshold Filter
Adaptive Threshold Filter
Average Luminance Threshold Filter
Histogram Filter
Histogram Generator
Average Color
Luminosity
Chroma Key Filter
Transform Filter
Crop Filter
Lanczos Resampling Filter
Sharpen Filter
Unsharp Mask Filter
Fast Blur Filter
Single Component Fast Blur Filter
Gaussian Blur Filter
Single Component Gaussian Blur Filter
Gaussian Selective Blur Filter
Gaussian Blur Position Filter
Median Filter
Bilateral Filter
Tilt Shift Filter
Box Blur Filter
3x3 Convolution Filter
Sobel Edge Detection Filter
Threshold Edge Detection Filter
Canny Edge Detection Filter
Harris Corner Detection Filter
Noble Corner Detection Filter
Shi-Tomasi Corner Detection Filter
Non Maximum Suppression Filter
X/Y Derivative Filter
Crosshair Generator
Dilation Filter
RGB Dilation Filter
Erosion Filter
RGB Erosion Filter
Opening Filter
RGB Opening Filter
Closing Filter
RGB Closing Filter
Local Binary Pattern Filter
Low Pass Filter
High Pass Filter
Motion Detector
Hough Transform Line Detector
Line Generator
Motion Blur Filter
Zoom Blur Filter
Chroma Key Blend Filter
Dissolve Blend Filter
Multiply Blend Filter
Add Blend Filter
Subtract Blend Filter
Divide Blend Filter
Overlay Blend Filter
Darken Blend Filter
Lighten Blend Filter
Color Burn Blend Filter
Color Dodge Blend Filter
Screen Blend Filter
Exclusion Blend Filter
Difference Blend Filter
Hard Light Blend Filter
Soft Light Blend Filter
Alpha Blend Filter
Source Over Blend Filter
Color Burn Blend Filter
Color Dodge Blend Filter
Normal Blend Filter
Color Blend Filter
Hue Blend Filter
Saturation Blend Filter
Luminosity Blend Filter
Linear Burn Blend Filter
Poisson Blend Filter
Mask Filter
Pixellate Filter
Polar Pixellate Filter
Polka Dot Filter
Halftone Filter
Crosshatch Filter
Sketch Filter
Threshold Sketch Filter
Toon Filter
Smooth Toon Filter
Emboss Filter
Posterize Filter
Swirl Filter
Bulge Distortion Filter
Pinch Distortion Filter
Stretch Distortion Filter
Sphere Refraction Filter
Glass Sphere Filter
Vignette Filter
Kuwahara Filter
Kuwahara Radius 3 Filter
Perlin Noise Filter
CGAColorspace Filter
Mosaic Filter
JFAVoronoi Filter
Voronoi Consumer Filter
Seriously, the Filter Showcase Example App that comes bundled in the repository could easily retail on the AppStore for $3.99, as-is. Add Twitter integration and a few sound effects, and you could bump that up to a respectable $6.99.
Rendering Pipeline
GPUImage is, at its core, an Objective-C abstraction around a rendering pipeline. Source images from the camera, network, or disk are loaded and manipulated according to a chain of filters, and finally outputted either a view, graphics context, or data stream.
For example, images from the video camera could have a Color Levels filter applied to simulate different types of color blindness and displayed in a live view.
Or, combining various color blending modes, image effects, and adjustments, you could transform still images into something worthy of sharing with your hipster friends (example taken from FilterKit, which is built on GPUImage):
Looking through all of what GPUImage can do, one can't help but get excited. Easy enough to get started immediately (without needing to know anything about OpenGL) yet performant enough to power whatever you dream up. And not just that, but it also comes with a dizzying number of building blocks—all of the color adjustments, blending modes, and visual effects you could ever want (or never knew you needed).
GPUImage is a rare treat for the open source community, and we as Mac & iOS developers are lucky to have it at our disposal. Use it to make something great, and show others the world in a whole new way.