Lessons in smartphone videography: What I learnt from editing a video shot on a phone

I have been obsessed for a while now with the idea of making a short film shot entirely on a smartphone. The versatility that a phone would allow in sheer ease of organising filming schedules is what attracts me the most. There has been a significant amount of interest from amateur / professional filmmakers in the industry along similar lines.

What I did not want to do, however, is to plan a shoot, film it on a smartphone, and then end up with a substandard product. I needed a low-risk project to try out my idea on. Fortunately, the chance presented itself when I got the idea of recording my time at the university’s Graduation Ball: I would record videos of my friends talking about the first time they met me, and what they thought of me. The beauty of this plan was that due to filming times, I would get to put my smartphone camera through its paces in a multitude of lighting conditions and noise environments, and since the video was unscripted, the content of the videos need not be a “good” or a “bad” take. You can watch The First Time…At Grad Ball to see what my effort worked out as. (The second part of my The First Time project was a photo album on Facebook telling the story of the first time I met my friends and what I thought of them.)

I realise that the video itself is pretty much a vanity project. Yet, I felt genuinely happy to do this because for the first time in my life – because of psychological issues I’ve had – I actually feel connected to my friends; that I care about them as individual beings. I wanted to create something to capture the essence of those emotions that I felt. Graduation, even though I’m not graduating yet, is one time people are allowed to be sentimental.

But I digress. The point of this blog post is to document my experience of what I learnt through the process of filming and editing video on a smartphone.

The “Why Now” On Smartphone Filming

Mobile phones have been able to record videos for around six years now, so it’s interesting to note how in general there’s a lot more buzz now about using them in amateur / prosumer contexts. Part of this comes from the fact that while early “smart” mobile phones (think Symbian and their ilk) could record video, the de facto recording format was 3GP / 3G2. Typically recording at QVGA / VGA resolution, the 3GP format allowed compact filesizes necessary for storing video files in a time when phones didn’t have much on-board RAM (to process a video while recording / playback) or storage space (which was often not extensible on such early smartphones).

As you can see in this example Tom and Jerry video, at the amount of compression used on such phones, video quality was poor and often had block distortion artefacts. Furthermore, for storage saving reasons the paired audio format used with the 3GP container was AMR or low-bitrate AAC that results in distorted audio; typically recorded mono channel or doubled-stereo from mono recording.

(Never search “3GP” on YouTube. I did, to find an example, and I got pages after pages of softcore porn. Brrrr.)

Another drawback of early smartphones was that they could not record at 24 fps and above. Phones as recent as my erstwhile 2009-era Nokia 5630 XpressMusic could only record at 15 fps (as demonstrated by this sample video). Such low frame rates added to the jerky, low-quality effect of mobile videos making them unusable for anything beyond sharing and viewing on other mobile devices.

Things started to change around the launch of iPhone 3GS, which came with the capability to record video at 30 fps. Android and Symbian handsets launched around the same time could record at similar frame rates, with some able to do it at 720p and others at 480p. The seed for my desire to film a project on a phone was sowed by this attempt by Nokia in 2010, who commissioned a short film shot entirely on an N8 (starring Dev Patel of Slumdog Millionaire fame as well as Baywatch‘s Pamela Anderson).

Fast forward to 2013, when most smartphones can record 720p video at 30 fps, typically in MP4 format. Significant advances in sensor technology, video processing software, even optical image stabilisation in higher-end phones means that the resulting video shot on phones these days is of much more decent quality. Certainly workable for amateur video projects. I expect, given the rate of progress in the field these days, that such advancements will continue to trickle down to cheaper devices.

Another important aspect of any video production happens to be sound. It may not be the first thing on your mind, but good audio quality is crucial in video recording. Smartphones are getting better at this too, with multiple microphones, noise cancellation, and high-quality AAC audio recording. You can audibly hear the difference that high quality, distortion-free stereo recording can make to a video in this comparison.

The Gear

Now that I’ve covered my reasons for why I think present-generation video recording on smartphones is usable-enough to be used for video projects, I’ll move to my own experience. My video was shot using my Nokia Lumia 620, so your mileage may vary according to what phone you have.

Outdoor shots with natural lighting turned out to be good, no problems there. My primary concern while I was filming was how the camera would perform in low light conditions. These turned out be quite good, surprisingly, for most cases even though they were lit using the LED flash. This is one instance where the video processing software used by your smartphone manufacturer will likely make a difference. I quite like how results on my Lumia didn’t appear to be harshly-lit, as is often the case with LED flashes. The photo gallery below has a sample of shots taken with LED flash that turned out fine.

Paradoxically, the video quality was poorer when there was ambient lighting in indoor night-time shots compared to ones where there was no ambient lighting. This, I’m guessing, is partly due to video processing done by the camera itself, and partly due to the lighting conditions. Keeping this factor in mind could be crucial for any indoor night-time shots you plan to shoot.

Another important factor to keep in mind is that when recording night-time videos, the LED flash works in “lamp” mode that results in a significantly lower coverage area in terms of illumination when compared to “burst” mode used for taking stills.

What this means is that you may need to film closer to the subject when recording a video. Take a look at the photo gallery below for a comparison of illumination when the images used above are thresholded at the same value. (Ideally, I’d have run this comparison across the same scene with different lighting conditions, but this wasn’t a controlled experiment.)

During the filming, the video preview that I saw indicated that the quality of recording was good. Unfortunately, when I had time to play them back on my phone / laptop, a major issue was apparent: there was a lot of stutter in the video, often resulting in frozen video / audio which meant for many of the recordings, I had entire sections of speeches missing! This video sample should demonstrate the problem I’m talking about.

I discovered when investigating this issue is that this was likely caused by a bottleneck with my micro SD card. I had been using a micro SD card I bought four years ago, rated as a class 4 device. If you aren’t aware of this, micro SD cards are rated at different classes based on the read / write speeds they can maintain at a sustained rate. While a class 4 device should technically be able to handle HD video streams, in reality such cards can be much slower.

First, I checked the speed of the phone camera’s focussing / metering capabilities using Sofica’s CamSpeed benchmarking tool; no surprises to report, as it the benchmark showed it was fast enough even in scenes with object motion. I then tested the read / write speed of the micro SD card using AnTuTu benchmark, and found that the read / write speeds were abysmally low – in the range of 0.9 MB/s – rather than the rated 4 MB/s (even though it was from SanDisk, a brand-name manufacturer). Switching the photos / video storage location from my SD card to the phone’s internal memory, and then recording test videos proved that the video lag problem went away. To further test my theory, I swapped out my original SD card for a new class 10 device I bought, and again, the results proved the same thing: with the faster rated card, there were no video lag problems.

This brings up an important issue: don’t skimp on your SD card, get at least a class 10 device! Looking around on Amazon, 4-8 GB class 10 micro SD cards are not that expensive, and the performance premium from the higher class storage is totally warranted for this use case.

(It also brings up an interesting point on whether “Android is slow” can be attributed to apps running off slow SD cards, and Windows Phone’s insistence on not allowing apps to run from SD card. Could be a user experience issue. I’ll follow-up on this in a later blog post.)

The Editing

After I had logged all the videos and was ready for editing, I started with setting up a new project in my video editor of choice, Adobe Premiere Pro CS5.5. This is step where I ran into my first hitch: most mobile phones these days record video at variable bit rates (VBR), and VBR is not supported by Premiere Pro at all! The reasons why phones record in variable bit rates are clear: to save on storage space; and, more importantly, changing the frame rate of video capture to compensate for lighting conditions (higher frame rates in poorer lighting conditions, and vice versa). Premiere Pro supports only a single frame rate across one project, so importing videos shot in VBR results in audio drifting wildly out-of-sync with video.

I was quite shocked to find out this, to be honest. The iPhone is the most popular camera in the world – across all device types, including dedicated cameras – which also records videos in VBR, so to leave such a popular device unsupported makes little sense. With professional recording equipment it’s easy to lock down frame rates but it’s hard to understand why Adobe wasn’t able to devote engineering resources to sort this issue out by 2013. Their decision makes even less sense when you consider Adobe sells a stripped-down version of their professional video editing software under the Adobe Premiere Elements brand. Surely a large portion of consumers who use Elements would need to edit footage shot on phones?

Ultimately, I solved this problem by downloading a trial version of Sony Vegas Pro – which does support multiple frame rates. The settings UI is slightly clunky, but it does allow setting a frame rate you want to hit across the project, and accordingly imported videos in its timeline to ensure audio-video stays in sync. As a non-linear video editor, Sony Vegas Pro does have adequate technical capabilities (in terms of format and project-setting support), although the interface does leave a lot to be desired in comparison to Adobe Premiere Pro.

On a more stylistic note, I could have worked more on this current video that I made to filter the audio for the noisier recording backgrounds. It’s audible, but only if you have a good set of speakers. In the end, I left them without doing this – partly because I didn’t want to spend too much time on this project, and partly to impart what I felt was a more “authentic” touch to the video that fully captured the energy of situation the videos were recorded in.

Naturally, if you’re using a Mac then with iMovie / Final Cut Pro, variable bit rates are supported out of the box. But if you’re editing footage shot on a smartphone on a Windows system, bear in mind that you need to account for additional cost; in terms of acquiring new software, or training time in getting to grips with how to use a software other than you may be accustomed to. The state of affairs for Windows users in this situation is a tad disappointing.

TL;DR

• Regardless of how capable your phone’s camera is, it won’t be able to record video “smoothly” unless you use at least a class 10 micro SD card. Invest in this.
• Adobe Premiere Pro does not support variable bit rate video. The only solution that can handle this on Windows that I’m aware of is Sony Vegas Pro.
• Once you have all the tools necessary in pre-and-post-production phases, smartphone video recording can afford a range of versatility that make them attractive for filming in uncontrolled settings.

Overall, I would say that I enjoyed the process of making a video on a smartphone. I taught me things to be careful about when using such gear, and hopefully, my lessons help out others thinking of similar projects.