Technology – Page 4

I have been obsessed for a while now with the idea of making a short film shot entirely on a smartphone. The versatility that a phone would allow in sheer ease of organising filming schedules is what attracts me the most. There has been a significant amount of interest from amateur / professional filmmakers in the industry along similar lines.

What I did not want to do, however, is to plan a shoot, film it on a smartphone, and then end up with a substandard product. I needed a low-risk project to try out my idea on. Fortunately, the chance presented itself when I got the idea of recording my time at the university’s Graduation Ball: I would record videos of my friends talking about the first time they met me, and what they thought of me. The beauty of this plan was that due to filming times, I would get to put my smartphone camera through its paces in a multitude of lighting conditions and noise environments, and since the video was unscripted, the content of the videos need not be a “good” or a “bad” take. You can watch The First Time…At Grad Ball to see what my effort worked out as. (The second part of my The First Time project was a photo album on Facebook telling the story of the first time I met my friends and what I thought of them.)

I realise that the video itself is pretty much a vanity project. Yet, I felt genuinely happy to do this because for the first time in my life – because of psychological issues I’ve had – I actually feel connected to my friends; that I care about them as individual beings. I wanted to create something to capture the essence of those emotions that I felt. Graduation, even though I’m not graduating yet, is one time people are allowed to be sentimental.

But I digress. The point of this blog post is to document my experience of what I learnt through the process of filming and editing video on a smartphone.

The “Why Now” On Smartphone Filming

Mobile phones have been able to record videos for around six years now, so it’s interesting to note how in general there’s a lot more buzz now about using them in amateur / prosumer contexts. Part of this comes from the fact that while early “smart” mobile phones (think Symbian and their ilk) could record video, the de facto recording format was 3GP / 3G2. Typically recording at QVGA / VGA resolution, the 3GP format allowed compact filesizes necessary for storing video files in a time when phones didn’t have much on-board RAM (to process a video while recording / playback) or storage space (which was often not extensible on such early smartphones).

As you can see in this example Tom and Jerry video, at the amount of compression used on such phones, video quality was poor and often had block distortion artefacts. Furthermore, for storage saving reasons the paired audio format used with the 3GP container was AMR or low-bitrate AAC that results in distorted audio; typically recorded mono channel or doubled-stereo from mono recording.

(Never search “3GP” on YouTube. I did, to find an example, and I got pages after pages of softcore porn. Brrrr.)

Another drawback of early smartphones was that they could not record at 24 fps and above. Phones as recent as my erstwhile 2009-era Nokia 5630 XpressMusic could only record at 15 fps (as demonstrated by this sample video). Such low frame rates added to the jerky, low-quality effect of mobile videos making them unusable for anything beyond sharing and viewing on other mobile devices.

Things started to change around the launch of iPhone 3GS, which came with the capability to record video at 30 fps. Android and Symbian handsets launched around the same time could record at similar frame rates, with some able to do it at 720p and others at 480p. The seed for my desire to film a project on a phone was sowed by this attempt by Nokia in 2010, who commissioned a short film shot entirely on an N8 (starring Dev Patel of Slumdog Millionaire fame as well as Baywatch‘s Pamela Anderson).

Fast forward to 2013, when most smartphones can record 720p video at 30 fps, typically in MP4 format. Significant advances in sensor technology, video processing software, even optical image stabilisation in higher-end phones means that the resulting video shot on phones these days is of much more decent quality. Certainly workable for amateur video projects. I expect, given the rate of progress in the field these days, that such advancements will continue to trickle down to cheaper devices.

Another important aspect of any video production happens to be sound. It may not be the first thing on your mind, but good audio quality is crucial in video recording. Smartphones are getting better at this too, with multiple microphones, noise cancellation, and high-quality AAC audio recording. You can audibly hear the difference that high quality, distortion-free stereo recording can make to a video in this comparison.

The Gear

Now that I’ve covered my reasons for why I think present-generation video recording on smartphones is usable-enough to be used for video projects, I’ll move to my own experience. My video was shot using my Nokia Lumia 620, so your mileage may vary according to what phone you have.

Outdoor shots with natural lighting turned out to be good, no problems there. My primary concern while I was filming was how the camera would perform in low light conditions. These turned out be quite good, surprisingly, for most cases even though they were lit using the LED flash. This is one instance where the video processing software used by your smartphone manufacturer will likely make a difference. I quite like how results on my Lumia didn’t appear to be harshly-lit, as is often the case with LED flashes. The photo gallery below has a sample of shots taken with LED flash that turned out fine.

Paradoxically, the video quality was poorer when there was ambient lighting in indoor night-time shots compared to ones where there was no ambient lighting. This, I’m guessing, is partly due to video processing done by the camera itself, and partly due to the lighting conditions. Keeping this factor in mind could be crucial for any indoor night-time shots you plan to shoot.

Video capture still night time shot, low quality. — Video capture still night-time shot, low quality.

Another important factor to keep in mind is that when recording night-time videos, the LED flash works in “lamp” mode that results in a significantly lower coverage area in terms of illumination when compared to “burst” mode used for taking stills.

Still shot showing larger illumination coverage area with burst mode flash

What this means is that you may need to film closer to the subject when recording a video. Take a look at the photo gallery below for a comparison of illumination when the images used above are thresholded at the same value. (Ideally, I’d have run this comparison across the same scene with different lighting conditions, but this wasn’t a controlled experiment.)

During the filming, the video preview that I saw indicated that the quality of recording was good. Unfortunately, when I had time to play them back on my phone / laptop, a major issue was apparent: there was a lot of stutter in the video, often resulting in frozen video / audio which meant for many of the recordings, I had entire sections of speeches missing! This video sample should demonstrate the problem I’m talking about.

I discovered when investigating this issue is that this was likely caused by a bottleneck with my micro SD card. I had been using a micro SD card I bought four years ago, rated as a class 4 device. If you aren’t aware of this, micro SD cards are rated at different classes based on the read / write speeds they can maintain at a sustained rate. While a class 4 device should technically be able to handle HD video streams, in reality such cards can be much slower.

First, I checked the speed of the phone camera’s focussing / metering capabilities using Sofica’s CamSpeed benchmarking tool; no surprises to report, as it the benchmark showed it was fast enough even in scenes with object motion. I then tested the read / write speed of the micro SD card using AnTuTu benchmark, and found that the read / write speeds were abysmally low – in the range of 0.9 MB/s – rather than the rated 4 MB/s (even though it was from SanDisk, a brand-name manufacturer). Switching the photos / video storage location from my SD card to the phone’s internal memory, and then recording test videos proved that the video lag problem went away. To further test my theory, I swapped out my original SD card for a new class 10 device I bought, and again, the results proved the same thing: with the faster rated card, there were no video lag problems.

This brings up an important issue: don’t skimp on your SD card, get at least a class 10 device! Looking around on Amazon, 4-8 GB class 10 micro SD cards are not that expensive, and the performance premium from the higher class storage is totally warranted for this use case.

(It also brings up an interesting point on whether “Android is slow” can be attributed to apps running off slow SD cards, and Windows Phone’s insistence on not allowing apps to run from SD card. Could be a user experience issue. I’ll follow-up on this in a later blog post.)

The Editing

After I had logged all the videos and was ready for editing, I started with setting up a new project in my video editor of choice, Adobe Premiere Pro CS5.5. This is step where I ran into my first hitch: most mobile phones these days record video at variable bit rates (VBR), and VBR is not supported by Premiere Pro at all! The reasons why phones record in variable bit rates are clear: to save on storage space; and, more importantly, changing the frame rate of video capture to compensate for lighting conditions (higher frame rates in poorer lighting conditions, and vice versa). Premiere Pro supports only a single frame rate across one project, so importing videos shot in VBR results in audio drifting wildly out-of-sync with video.

I was quite shocked to find out this, to be honest. The iPhone is the most popular camera in the world – across all device types, including dedicated cameras – which also records videos in VBR, so to leave such a popular device unsupported makes little sense. With professional recording equipment it’s easy to lock down frame rates but it’s hard to understand why Adobe wasn’t able to devote engineering resources to sort this issue out by 2013. Their decision makes even less sense when you consider Adobe sells a stripped-down version of their professional video editing software under the Adobe Premiere Elements brand. Surely a large portion of consumers who use Elements would need to edit footage shot on phones?

Sony Vegas Pro was the only software that could process videos from my phone

Ultimately, I solved this problem by downloading a trial version of Sony Vegas Pro – which does support multiple frame rates. The settings UI is slightly clunky, but it does allow setting a frame rate you want to hit across the project, and accordingly imported videos in its timeline to ensure audio-video stays in sync. As a non-linear video editor, Sony Vegas Pro does have adequate technical capabilities (in terms of format and project-setting support), although the interface does leave a lot to be desired in comparison to Adobe Premiere Pro.

On a more stylistic note, I could have worked more on this current video that I made to filter the audio for the noisier recording backgrounds. It’s audible, but only if you have a good set of speakers. In the end, I left them without doing this – partly because I didn’t want to spend too much time on this project, and partly to impart what I felt was a more “authentic” touch to the video that fully captured the energy of situation the videos were recorded in.

Naturally, if you’re using a Mac then with iMovie / Final Cut Pro, variable bit rates are supported out of the box. But if you’re editing footage shot on a smartphone on a Windows system, bear in mind that you need to account for additional cost; in terms of acquiring new software, or training time in getting to grips with how to use a software other than you may be accustomed to. The state of affairs for Windows users in this situation is a tad disappointing.

TL;DR

Regardless of how capable your phone’s camera is, it won’t be able to record video “smoothly” unless you use at least a class 10 micro SD card. Invest in this.
Adobe Premiere Pro does not support variable bit rate video. The only solution that can handle this on Windows that I’m aware of is Sony Vegas Pro.
Once you have all the tools necessary in pre-and-post-production phases, smartphone video recording can afford a range of versatility that make them attractive for filming in uncontrolled settings.

Overall, I would say that I enjoyed the process of making a video on a smartphone. I taught me things to be careful about when using such gear, and hopefully, my lessons help out others thinking of similar projects.

I love discovering new music to listen to, but what I don’t like is reading reviews on music blogs or magazines to find them. Reading through reviews rather than listening to discover new stuff feels so 20th century to me. Music is emotional for me – as it is with many people – and I’d rather get a feel for something new myself to find whether I like it, rather than just reading words on a page.

(After I find an artist I like, though, I do read up about their history and creative process on music blogs. That’s what they are good for: no matter how obscure an artist, you’re likely to find someone who has interviewed them.)

My go-to source for music recommendations used to be Last.fm. I still scrobble to my profile – partly for keeping track of my music tastes over time, mostly in the vain hope that some day I will be able to unlock this data in a usable form.

Last.fm remains a one-of-a-kind service for archiving music tastes; there simply hasn’t been any replacement over the years for a service like this. Ever since they stopped on-demand playback of tracks though (due to licensing issues; it was simply financially unviable for them to offer it) the platform has been less than worthless for new music discovery. Using Last.fm radio – while good at making automated recommendations based on listening history – is quite cumbersome to use as a standalone application. As it stands now, Last.fm’s discovery mechanism is primarily focussed around surfacing recommendations based on similar tracks / albums…which makes the process of going through all of them a chore.

***

I have been a Spotify customer for four years now (first, Spotify Unlimited, and now Spotify Premium). It’s every music fanatic’s wet dream – an almost-limitless library of songs, available on desktop, web, mobile, offline that syncs everywhere, and for the artists holding out from adding their works to their library, it’s easy to add local MP3s to your collection.

We Are Hunted's automated playlists — We Are Hunted’s automated playlists

For over a year now, I was using the We Are Hunted Spotify app to source my music recommendations from. The neat idea behind We Are Hunted was that it scoured the web for signals of what music people were listening to – rather than relying merely on charts such as Billboard – and automatically created digests of what the hottest tracks in each genre were. I loved it, because for people like me who are into indie / alternative emerging music, it surfaced finds that may not even yet be climbing on any official charts yet…but thanks to Spotify’s vast tie-ups with independent labels could still be found in its library. There were times when I discovered music so underground that they weren’t available anywhere other than Spotify, or perhaps, Soundcloud (not even YouTube!). Many of my “What I Have Been Listening To” recommendations came about this way. Every month, We Are Hunted would publish an automatically-generated playlist of the hottest music and make it available through Spotify. I looked forward to this day every month with the same kind of excitement one would for a magazine issue.

Then, it all ended when We Are Hunted was acquired by Twitter. Functionally, it does the same thing that it used to – in a far more cumbersome interface. Now, I need to login to Twitter Music using my Twitter and Spotify accounts, and play the tracks through that interface. There’s no easy way to export the list as a playlist any more, as the obvious intention here is to lock people in to playing music through Twitter’s own interface and / or music apps.

Herein lies the problem with every other music discovery service I have ever seen: they expect the user to play part of the curation process either by selecting ‘channels’ or ‘users’ to follow (such as Hype Machine, 8tracks, and others), or by basing recommendations on individual albums / tracks / artists / playlists (such as Last.fm and countless Spotify apps). The downside, as I see it, is that either way it requires effort on part of the user to constantly prune lists of whom to follow, or in the second scenario to refresh the recommendations manually. This is a cumbersome process! We Are Hunted was doing something extremely unique with the service they were offering. It’s a service useful enough for me that I’d pay for something like it on top of my Spotify subscription fees. (The closest replacement that I have (recently) stumbled upon is Tunigo. It’s still not a perfect replacement though for We Are Hunted as its top-of-genre lists count absolutes rather than ‘rising’ tracks like WAH used to.)

***

Spotify's new Discover feature in its web player — Spotify’s new Discover feature in its web player

The need for such a service or an app would be obviated if Spotify itself offered an equivalent. It has moved somewhat in this direction with the launch of its new Discover feature.

Apps such as We Are Hunted and Tunigo are good, but what I really want is a leanback, interaction-free music recommendation engine – much like Last.fm’s radio feature – that simply melts into the background and throws up one new track after another. Spotify does something similar with its Radio feature, on the surface, but in my experience it isn’t truly a radio service. Instead of having a native recommendation engine, Spotify integrates with The Echo Nest’s music discovery APIs to power its radio functionality – and the way this seems to work is that it generates a static playlist based on a genre, artist, track, or playlist that is queued for playback. The reason why I say this is that whenever I play Spotify radio, it seems to continue repeating the same tracks over and over again. Supposedly, using the thumbs up / down buttons is meant to help its recommendation engine ‘get better’ but all a thumbs down seems to do is to skip to next track, and the thumbs up button to add the track to a ‘Liked from radio’ playlist.

Spotify’s API primarily seems to revolve around playlists – and I can see how for people whose music experience revolves around playlists, e.g., many friends I have on Spotify, this is the perfect model of curation for them. However, for anyone – like me – who wants more robust and automated tools for an interaction-free music experience, it falls short of expectations.

My point is that without a good recommendation engine backing up its radio feature, Spotify is leaving money on the table in terms of user experience. It has such a deep library of music, and yet, discovering it is still primarily driven by last-decade methods such as collecting albums and playlists. I expected much better from a service that is supposedly the future of music.

***

UPDATE: Twitter launched a Spotify app that allows playlists to be exported. Solves many of the problems I listed here! I’m glad Twitter realised its #music feature was crippled without supporting an external ecosystem.

UPDATE 2: Twitter Music is dead.