Categories
Technology

Lessons in smartphone videography: What I learnt from editing a video shot on a phone

I have been obsessed for a while now with the idea of making a short film shot entirely on a smartphone. The versatility that a phone would allow in sheer ease of organising filming schedules is what attracts me the most. There has been a significant amount of interest from amateur / professional filmmakers in the industry along similar lines.

What I did not want to do, however, is to plan a shoot, film it on a smartphone, and then end up with a substandard product. I needed a low-risk project to try out my idea on. Fortunately, the chance presented itself when I got the idea of recording my time at the university’s Graduation Ball: I would record videos of my friends talking about the first time they met me, and what they thought of me. The beauty of this plan was that due to filming times, I would get to put my smartphone camera through its paces in a multitude of lighting conditions and noise environments, and since the video was unscripted, the content of the videos need not be a “good” or a “bad” take. You can watch The First Time…At Grad Ball to see what my effort worked out as. (The second part of my The First Time project was a photo album on Facebook telling the story of the first time I met my friends and what I thought of them.)

I realise that the video itself is pretty much a vanity project. Yet, I felt genuinely happy to do this because for the first time in my life – because of psychological issues I’ve had – I actually feel connected to my friends; that I care about them as individual beings. I wanted to create something to capture the essence of those emotions that I felt. Graduation, even though I’m not graduating yet, is one time people are allowed to be sentimental.

But I digress. The point of this blog post is to document my experience of what I learnt through the process of filming and editing video on a smartphone.

The “Why Now” On Smartphone Filming

Mobile phones have been able to record videos for around six years now, so it’s interesting to note how in general there’s a lot more buzz now about using them in amateur / prosumer contexts. Part of this comes from the fact that while early “smart” mobile phones (think Symbian and their ilk) could record video, the de facto recording format was 3GP / 3G2. Typically recording at QVGA / VGA resolution, the 3GP format allowed compact filesizes necessary for storing video files in a time when phones didn’t have much on-board RAM (to process a video while recording / playback) or storage space (which was often not extensible on such early smartphones).

As you can see in this example Tom and Jerry video, at the amount of compression used on such phones, video quality was poor and often had block distortion artefacts. Furthermore, for storage saving reasons the paired audio format used with the 3GP container was AMR or low-bitrate AAC that results in distorted audio; typically recorded mono channel or doubled-stereo from mono recording.

(Never search “3GP” on YouTube. I did, to find an example, and I got pages after pages of softcore porn. Brrrr.)

Another drawback of early smartphones was that they could not record at 24 fps and above. Phones as recent as my erstwhile 2009-era Nokia 5630 XpressMusic could only record at 15 fps (as demonstrated by this sample video). Such low frame rates added to the jerky, low-quality effect of mobile videos making them unusable for anything beyond sharing and viewing on other mobile devices.

Things started to change around the launch of iPhone 3GS, which came with the capability to record video at 30 fps. Android and Symbian handsets launched around the same time could record at similar frame rates, with some able to do it at 720p and others at 480p. The seed for my desire to film a project on a phone was sowed by this attempt by Nokia in 2010, who commissioned a short film shot entirely on an N8 (starring Dev Patel of Slumdog Millionaire fame as well as Baywatch‘s Pamela Anderson).

Fast forward to 2013, when most smartphones can record 720p video at 30 fps, typically in MP4 format. Significant advances in sensor technology, video processing software, even optical image stabilisation in higher-end phones means that the resulting video shot on phones these days is of much more decent quality. Certainly workable for amateur video projects. I expect, given the rate of progress in the field these days, that such advancements will continue to trickle down to cheaper devices.

Another important aspect of any video production happens to be sound. It may not be the first thing on your mind, but good audio quality is crucial in video recording. Smartphones are getting better at this too, with multiple microphones, noise cancellation, and high-quality AAC audio recording. You can audibly hear the difference that high quality, distortion-free stereo recording can make to a video in this comparison.

The Gear

Now that I’ve covered my reasons for why I think present-generation video recording on smartphones is usable-enough to be used for video projects, I’ll move to my own experience. My video was shot using my Nokia Lumia 620, so your mileage may vary according to what phone you have.

Outdoor shot
Outdoor shot with natural lighting, good quality.

Outdoor shots with natural lighting turned out to be good, no problems there. My primary concern while I was filming was how the camera would perform in low light conditions. These turned out be quite good, surprisingly, for most cases even though they were lit using the LED flash. This is one instance where the video processing software used by your smartphone manufacturer will likely make a difference. I quite like how results on my Lumia didn’t appear to be harshly-lit, as is often the case with LED flashes. The photo gallery below has a sample of shots taken with LED flash that turned out fine.

Paradoxically, the video quality was poorer when there was ambient lighting in indoor night-time shots compared to ones where there was no ambient lighting. This, I’m guessing, is partly due to video processing done by the camera itself, and partly due to the lighting conditions. Keeping this factor in mind could be crucial for any indoor night-time shots you plan to shoot.

Video capture still night time shot, low quality.
Video capture still night-time shot, low quality.

Another important factor to keep in mind is that when recording night-time videos, the LED flash works in “lamp” mode that results in a significantly lower coverage area in terms of illumination when compared to “burst” mode used for taking stills.

Still shot showing larger illumination coverage area with burst mode flash
Still shot showing larger illumination coverage area with burst mode flash

What this means is that you may need to film closer to the subject when recording a video. Take a look at the photo gallery below for a comparison of illumination when the images used above are thresholded at the same value. (Ideally, I’d have run this comparison across the same scene with different lighting conditions, but this wasn’t a controlled experiment.)

During the filming, the video preview that I saw indicated that the quality of recording was good. Unfortunately, when I had time to play them back on my phone / laptop, a major issue was apparent: there was a lot of stutter in the video, often resulting in frozen video / audio which meant for many of the recordings, I had entire sections of speeches missing! This video sample should demonstrate the problem I’m talking about.

I discovered when investigating this issue is that this was likely caused by a bottleneck with my micro SD card. I had been using a micro SD card I bought four years ago, rated as a class 4 device. If you aren’t aware of this, micro SD cards are rated at different classes based on the read / write speeds they can maintain at a sustained rate. While a class 4 device should technically be able to handle HD video streams, in reality such cards can be much slower.

First, I checked the speed of the phone camera’s focussing / metering capabilities using Sofica’s CamSpeed benchmarking tool; no surprises to report, as it the benchmark showed it was fast enough even in scenes with object motion. I then tested the read / write speed of the micro SD card using AnTuTu benchmark, and found that the read / write speeds were abysmally low – in the range of 0.9 MB/s – rather than the rated 4 MB/s (even though it was from SanDisk, a brand-name manufacturer). Switching the photos / video storage location from my SD card to the phone’s internal memory, and then recording test videos proved that the video lag problem went away. To further test my theory, I swapped out my original SD card for a new class 10 device I bought, and again, the results proved the same thing: with the faster rated card, there were no video lag problems.

This brings up an important issue: don’t skimp on your SD card, get at least a class 10 device! Looking around on Amazon, 4-8 GB class 10 micro SD cards are not that expensive, and the performance premium from the higher class storage is totally warranted for this use case.

(It also brings up an interesting point on whether “Android is slow” can be attributed to apps running off slow SD cards, and Windows Phone’s insistence on not allowing apps to run from SD card. Could be a user experience issue. I’ll follow-up on this in a later blog post.)

The Editing

After I had logged all the videos and was ready for editing, I started with setting up a new project in my video editor of choice, Adobe Premiere Pro CS5.5. This is step where I ran into my first hitch: most mobile phones these days record video at variable bit rates (VBR), and VBR is not supported by Premiere Pro at all! The reasons why phones record in variable bit rates are clear: to save on storage space; and, more importantly, changing the frame rate of video capture to compensate for lighting conditions (higher frame rates in poorer lighting conditions, and vice versa). Premiere Pro supports only a single frame rate across one project, so importing videos shot in VBR results in audio drifting wildly out-of-sync with video.

I was quite shocked to find out this, to be honest. The iPhone is the most popular camera in the world – across all device types, including dedicated cameras – which also records videos in VBR, so to leave such a popular device unsupported makes little sense. With professional recording equipment it’s easy to lock down frame rates but it’s hard to understand why Adobe wasn’t able to devote engineering resources to sort this issue out by 2013. Their decision makes even less sense when you consider Adobe sells a stripped-down version of their professional video editing software under the Adobe Premiere Elements brand. Surely a large portion of consumers who use Elements would need to edit footage shot on phones?

Sony Vegas Pro was the only software that could process videos from my phone
Sony Vegas Pro was the only software that could process videos from my phone

Ultimately, I solved this problem by downloading a trial version of Sony Vegas Pro – which does support multiple frame rates. The settings UI is slightly clunky, but it does allow setting a frame rate you want to hit across the project, and accordingly imported videos in its timeline to ensure audio-video stays in sync. As a non-linear video editor, Sony Vegas Pro does have adequate technical capabilities (in terms of format and project-setting support), although the interface does leave a lot to be desired in comparison to Adobe Premiere Pro.

On a more stylistic note, I could have worked more on this current video that I made to filter the audio for the noisier recording backgrounds. It’s audible, but only if you have a good set of speakers. In the end, I left them without doing this – partly because I didn’t want to spend too much time on this project, and partly to impart what I felt was a more “authentic” touch to the video that fully captured the energy of situation the videos were recorded in.

Naturally, if you’re using a Mac then with iMovie / Final Cut Pro, variable bit rates are supported out of the box. But if you’re editing footage shot on a smartphone on a Windows system, bear in mind that you need to account for additional cost; in terms of acquiring new software, or training time in getting to grips with how to use a software other than you may be accustomed to. The state of affairs for Windows users in this situation is a tad disappointing.

TL;DR

  • Regardless of how capable your phone’s camera is, it won’t be able to record video “smoothly” unless you use at least a class 10 micro SD card. Invest in this.
  • Adobe Premiere Pro does not support variable bit rate video. The only solution that can handle this on Windows that I’m aware of is Sony Vegas Pro.
  • Once you have all the tools necessary in pre-and-post-production phases, smartphone video recording can afford a range of versatility that make them attractive for filming in uncontrolled settings.

Overall, I would say that I enjoyed the process of making a video on a smartphone. I taught me things to be careful about when using such gear, and hopefully, my lessons help out others thinking of similar projects.

Categories
Personal Reflections

So, I guess I’m not graduating with everyone else

I can’t believe this is happening.

My university released its results for this academic year and I’ve just found out that I’m not graduating this year. I need to resit two modules in August as part of the late summer resits the university offers, which means that the exam board will not be able to ratify my results in time for the graduation ceremony in July.

In June, I was called by the department to discuss “my case”. I was told in that meeting that I had the following options:

  • Take a leave of absence from university on medical grounds, take exams later, and defer my graduation until next year.
  • Sit for the June exams as normal, but in doing so acknowledge that I was going against the department’s advice.

I do accept my personal responsibility for why this is happening. I chose the latter option, and the gamble didn’t pay off.

***

It’s not hard to pinpoint where my academic career went off the rails. Earlier this year, I talked at length about the mental health issues that I have been facing for a considerable period. That was back in April, at the beginning of Easter break. I was still struggling to get back on my feet when I wrote that piece, as I was ramping up my dose of prescribed antidepressants at the time. By mid-May, I finally felt better, felt that the situation was under control.

The problem I faced was that I’d gone into a freefall for the period from February to May. I had a pile of work to chew through and not enough time.

I felt paralysed. I felt claustrophobic. My primary concern was my dissertation. I’d sunk a lot of effort into it at the start of the academic year, and it was a project that I was proud to be doing because it was innovative, it was an idea that I came up with myself, and something that I felt passionate about. I planned to use the Easter break to catch up on my project, and I emailed my dissertation supervisor for guidance at the time. I never heard back.

I looked up to my dissertation supervisor as role model in academic research, almost as a father-figure for how he guided me through the year. (And my counsellor was someone that I looked up to as a mother-figure too.) It wasn’t until that blog post I published on depression that I felt comfortable to talk to him about what I was going through. I was ashamed to admit it, and I was scared whether he’d consider me less capable for it. In fact, he was quite understanding and supportive. It reassured me the tiniest bit to know that he had my back.

So when I didn’t get an email response back during Easter break from him, paranoia kicked in. I thought my supervisor was mad or disappointed with me, for letting him down. I felt shame and guilt for not performing to what I thought was the best of my abilities. Faced with technical challenges, I just plain gave up on working on the project for weeks.

The truth was for more mundane: the email that I sent simply got buried in the flood of emails that academics usually get. I could have resolved the situation simply by pinging him again. When I finally felt stable, I tried to get on track with getting in touch with people in my department. I’d been regularly attending counselling sessions and having medical reviews of my case, and I went with this evidence to them.

The university does have procedures in place for considering extenuating circumstances. Unfortunately, the process of applying for one and the evidence being considered is a long-winded process, and decisions can take weeks – if not months – to be reached. By the time I applied I started getting decisions back on coursework and dissertation deadline extensions, it was already a week to go for my exams. It literally came down to the wire. I remember frantically panicking in the library because I’d supplied all the evidence, and yet only found out half an hour before the normal dissertation deadline whether my circumstances would be considered. I almost had a panic attack.

And then…I just got a week extra, when I was finally getting back on track but felt I needed more time to catch up during all those lost weeks. I didn’t realistically think I’d get an extra three months, but what I wanted was to have my dissertation deadline stretch into the summer before the exam board met so that I could focus on preparing for my exams. That, apparently, wasn’t possible because of logistical reasons.

I could have appealed the decision my department took directly with the university. However, due to the procedure laid down for this, it would have meant waiting for 3-4 months before I found out the outcome. And if I lost the appeal, well, I’d be up shit creek without a paddle for having failed my dissertation. So I decided not to, and rushed through with wrapping up my project.

I redirected all my efforts into getting the project, and did have a working demo in time. My project was to create a real-time gaze tracking system that worked with standard webcam video on mobile phones. (Kinda like what the Samsung Galaxy S4 has, but more advanced because they aren’t doing “true” gaze tracking.) Admittedly, partly due to the time schedule I was on I didn’t have time to sort out bugs in upstream open source projects that I was using, so I didn’t hit all of the targets that I set out to achieve…but it was at least the best of a worst case effort.

During the same time, I was asked to attend a review by the department. Based on the evidence that I had provided so far to them, all of the mental health concerns that were raised during my medical reviews, their recommendation was I take a leave of absence. It was quite a frank and friendly chat, actually. I remember sitting in that room and my overpowering urge was that I just wanted this whole episode of university to be done and over with. Taking deferred assessment would mean I would be paying tuition fees for another year where I would have jack to do except for kick tyres until (possibly) June 2014, as I would have to wait until then to take exams. And most of all, I just wanted to graduate with my friends. I decided to take the decision to plunge ahead with exams this year. Honestly, I didn’t even care what degree classification I would get. I just wanted to get this done with.

In retrospect, it’s easy to say that I should have taken the “out” when it was offered to me. It’s also easy to be bitter and start blaming external factors. Yes, I wish there was better communication between different arms of the university. Yes, I would have fucking loved it if they made an exception to the rules to suit my schedule. Yes, I wish some of the communication and the decision-making process was quicker, so that I could make informed decisions faster without needing to go through long spells of anxiety.

But the truth is that, at the end of the day, it was my decision to ignore the advice and go ahead. I may not have been in the best state of mind when I made the decision, but it is something that I need to live with. Similarly, I was the one who let go and effectively went AWOL from my course. I can’t “take responsibility” for my depression because I didn’t choose to be this way, but ultimately that’s what happened and affected my performance.

I’m actually quite impressed and glad with all the support that I have received from my university. Throughout the process, the people whom I have spoken to have been understanding; constrained by what they could do according to the rules, but understanding nonetheless – and to me, while I was going through a hard time, that meant a lot. I also have to thank the Students’ Union for excellent advice and support throughout the process. Every step of the way when I faltered, it was reassuring to know that it just wasn’t my friends but the university too which had my back.

As things stand now, I’m taking resits in August. Resits are capped at pass-mark cutoff under our university’s regulations, which means my overall degree classification will probably suffer in the process since it’s weighted heavily towards performance in the final year. I will have a transcript by September, when the final exam board meets to ratify results, but I won’t be awarded a degree scroll until next year’s graduation ceremony in June.

***

What next? Honestly, I have no idea. I guess I need to focus on the resits for now. I probably need to do a second degree to make up for what I know will be shit result on this one, whenever I find that out three months later. Which will be made harder by the fact that I have a shit result, so my options may be limited, unless my extenuating circumstances are taken into consideration. And the PhD offer that I could have had is definitely off the cards now.

A big confidence boost in the recent weeks has been that I have been making good progress with job applications. I have made it to the final round for every single application I have made, with only two of them giving a final decision so far; rejecting my application, but with valuable feedback nonetheless. And I’m currently awaiting outcomes from multiple companies, with more confirmed interviews lined up in the future. The first thing that I did after I finally felt I was in control of things was to get on with the job hunt, and finding approval from such external agencies feels good. I have my fingers crossed that through the interview process and/or in case I get offer, my extenuating circumstances would be something they are willing to consider – and that I can move on from higher education.

Failing which…I have no idea what degree to study next or where to study it. It kinda excites and scares me.

***

Academic stress has not been my only concern over the past few months. I’ve had my own problems to deal with, with getting better and more often than not it has felt like gasping for breath. Times when I feel better, times when I feel worse. Besides that, I’ve had friends going through personal problems too during the period which I was helping them deal with. I won’t go into details because it concerns the private lives of other people, but it was something that added to the stress and anxiety nevertheless. I absolutely don’t regret it though; I would be there for them if I had to do it again. What I have realised this year is that I’ve finally learnt not be a selfish little piece of shit and sometimes put others’ concerns before my own, because that’s just what friends do. And in return, I’ve had friends who’ve stuck with me through some pretty dark times. I’m glad for that.

Throughout the months when I battled worsening depression, I never wanted my friends to know the true extent of how shit I was feeling on a day-to-day basis. I’d tell them what was bothering me, but almost in a flippant, nonchalant way. I like to present an image of “yes, I’ve got my shit together” because the alternative of actually showing how bad I was a rabbit hole that I didn’t want to go down. Despite their best intentions and what people say, the truth is that nobody likes being around a depressive at their worst – especially if they have never been through anything like it. Like I said in my earlier blog post on depression, healing is a long-term process and if you don’t make efforts to achieve that goal, you’re well and truly fucked. I like to think “I’m not bothered by what’s happening, I’ll just focus on the moment” because if I do think it should bother me, then it does actually bother me and I feel worse. It’s a vicious feedback loop.

Looking back at the past few months, I’d say that the support that I’ve had from friends has massively helped me recover. Because I’ve seen other people dealing with similar issues, I’m glad I was able to push myself to remain socially active and not give up on daily life. It’s given me something to wake up to every day.

***

University of Surrey graduation ceremony

I recently acted as an extra in a film shot by the university’s alumni department about the university’s graduation ceremony. I got to wear a robe and a hat, and I made a mental note to amend my robe hire order because I found out through that shoot that I got my hat measurements wrong. I’d only just made my graduation ceremony bookings earlier this week. And now, I won’t have to bother with that at least until next year.

This is a video that is intended to be played at every graduation ceremony for the next two years.

Irony, thou art a cruel bitch.

***

“Life was so much simpler ten years ago when I knew everything.” – Paul Carr