Video + Photography: Mixing Two Forms: Part 1
We’ve all been there. Standing looking out at something we want to capture, but even when you’re using some of the most advanced digital cameras, we all face this question: do you take a video, or a photo — or both?
Most modern DSLR + m.3/4 cameras allow you to take photos while in video mode, but in both (most models anyways) there is a delay, and subsequent shutter/ capture clicks, interrupting the video you’re capturing. You might not think that would matter, and typically it doesn’t — but if you’re shooting video, and the frame just falls where you would like to continue the video — you’ll be missing some frames. Of course if you can time this well, short of having extra batteries, this can work.
The next solution is to use two cameras — this works, but is cumbersome, and time-consuming. A great alternative— if video quality can be compromised is to use a simple camera like a go-pro. But if you’re shooting in the dark, that poses issues.
The other solution is to capture your frames directly from your video file. For a lot of people, this sounds, and typically is a last-ditch effort to get a still frame from a scene. When using this method, it’s important you have your iso + aperture dialed in when shooting (preferably before). There are a few tricks with this, such as making sure you have enough of each color channel to work with (and alpha for exposure adjustments). You won’t have shutter options, you can’t change much — and depending on what file-format you’re shooting + saving your video as, in many dark scenes, there will be artefacts of shading and color inherit to the mp4 file format for example.
Why do we see those color + shading discrepancies? There are at least two of these compression/encoding artefacts that will impact your ability to capture frames within mp4 files.
These are ringing, and ‘mosquito noise’ — two artefacts inherent to the capturing— compression, and encoding of video signals which a camera outputs as H.264 or MPEG-4 + MPEG-2 video files (.mp4, .mov, .m4v, .flv).
These artefacts can also present themselves through other variables (i.e. image compression in JPEG + RAW, CCD capturing in general).
Ringing or echoing, is a more common artefact signifying the halo-ing of sharp-edges of an image or a scene (think defined lines, like a close up of a building on a skyline, zoomed in). In the context of digital-video, ringing is the consequence of decompression of the video file from frame to frame due to a lack of data between frames. A video-file is simply a collection of frames, assembled and/or captured at a set frame rate.
‘mosquito noise’ is similar to ringing and often mistaken, signifying a shimmering on sharp-edges of the scene. It is also an artefact associated with video-frame compression.
MP4 files are really a remarkable thing — they’re compressed containers of image-frames that are also compressed. Within this container, there are algorithmic directives to interrupt and ‘play’ the video + audio frames at a set frame-rate by a player/program that can interrupt and decompress + “play” the file. Using software such as ffmpeg, you can easily extract frames from an mp4 file, at set time-points, or frames where you know there will be a clearer image (i.e not a ‘b-frame’ or a ‘p-frame’ — frames which are essentially predicted based upon the frames previous, in sequence). The container is really remarkable, using predictions based on frames to append together a more or less clear moving image, or video.
Of course using predicted frames is never ideal to capture an image. That’s where the challenge comes in, and additional frame-analyzing+extracting software can come in handy. In general, if you want to use frames from your video capture, make sure you’re using the highest quality compression/encoding format available on your camera. For Canon cameras for instance, if you want above 30fps, AVCHD is typically the way to go with it’s higher bandwith, and 60i capability (depending on the camera). Compared to .mov/.mp4, AVCHD .mts + .m2ts files, they’re still h.264 video-encoded, but the output file also contains Doly Digital AC-3 audio. With a higher frame rate — means more frames, typically allowing easier + more precise frame/image extraction.
We will continue this series with a follow up where we’ll break down using ffmpeg on MacOS running brew, to extract images from mp4 frame files.
-- article by: Alexander Tri