Learning to Synthesize Motion Blur

Summary

Input Images (Average)

Output Image

Input Images (Average)

Output Image

It's hard to capture a sense of movement in a single photograph. Creating compelling motion blur is a problem that has interested me as an artist for many years. I made a crude motion blur app for my first project in an introductory computer science class my freshman year of undergrad. Five years later, I worked with a teammate of mine at Google AI to use machine learning to synthesize much more realistic motion blur, which can be used as a visual effect in photography, cinematography and computer graphics.

Video

Abstract

We present a technique for synthesizing a motion blurred image from a pair of unblurred images captured in succession. To build this system we motivate and design a differentiable "line prediction" layer to be used as part of a neural network architecture, with which we can learn a system to regress from image pairs to motion blurred images that span the capture time of the input image pair. Training this model requires an abundance of data, and so we design and execute a strategy for using frame interpolation techniques to generate a large-scale synthetic dataset of motion blurred images and their respective inputs. We additionally capture a high quality test set of real motion blurred images, synthesized from slow motion videos, with which we evaluate our model against several baseline techniques that can be used to synthesize motion blur. Our model produces higher accuracy output than our baselines, and is significantly faster than baselines with competitive accuracy.