Firefox's Gecko Media Plugin & EME Architecture

For rendering audio and video Firefox typically uses either the operating system's audio/video codecs or bundled software codec libraries, but for DRM video playback (like Netflix, Amazon Prime Video, and the like) and WebRTC video calls using baseline H.264 video, Firefox relies on Gecko Media Plugins, or GMPs for short.

This blog post describes the architecture of the Gecko Media Plugin system in Firefox, and the major class/objects involved, as it looked in June 2019.

For DRM video Firefox relies upon Google's Widevine Content Decryption Module, a dynamic shared library downloaded at runtime. Although this plugin doesn't conform to the GMP ABI, we provide an adapter to allow it to be run through the GMP system. We use the same Widevine CDM plugin that Chrome uses.

For decode and encode of H.264 streams for WebRTC, Firefox uses OpenH264, which is provided by Cisco. This plugin implements the GMP ABI.

These two plugins are downloaded at runtime from Google's and Cisco's servers, and installed in the user's Firefox profile directory.

We also ship a ClearKey CDM, which is the baseline decryption scheme required by the Encrypted Media Extensions specification. This mimics interface which the Widevine CDM implements, and is used in our EME regression tests. It's bundled with the rest of Firefox, and lives in the Firefox install directory.

The objects involved in running GMPs are spread over three processes; the main (AKA parent) process, the sandboxed content process where we run JavaScript and load web pages, and the sandboxed GMP process, which only runs GMPs.

You can view a Diagram of Firefox's Gecko Media Plugin online, or download a PDF version of Firefox's Gecko Media Plugin architecture.

The main facade to the GMP system is the GeckoMediaPluginService. Clients use the GeckoMediaPluginService to instantiate IPDL actors connecting their client to the GMP process, and to configure the service. In general, most operations which involve IPC to the GMPs/CDMs should happen on the GMP thread, as the GMP related protocols are processed on that thread.

mozIGeckoMediaPluginService can be used on the main thread by JavaScript, but the main-thread accessible methods proxy their work to the GMP thread.

How GMPs are downloaded and installed

The Firefox front end code which manages GMPs is the GMPProvider. This is a JavaScript object, running in the front end code in the main process. On startup if any existing GMPs are already downloaded and installed, this calls mozIGeckoMediaPluginService.addPluginDir() with the path to the GMP's location on disk. Gecko's C++ code then knows about the GMP. The GeckoMediaPluginService then parses the metadata file in that GMP's directory, and creates and stores a GMPParent for that plugin. At this stage the GMPParent is like a template, which stores the metadata describing how to start a plugin of this type. When we come to instantiate a plugin, we'll clone the template GMPParent into a new instance, and load a child process to run the plugin using the cloned GMPParent.

Shortly after the browser starts up (usually within 60 seconds), the GMPProvider will decide whether it should check for new GMP updates. The GMPProvider will check for updates if either it has not checked in the past 24 hours, or if the browser has been updated since last time it checked. If the GMPProvider decides to check for updates, it will poll Mozilla's Addons Update Server. This will return an update.xml file which lists the current GMPs for that particular Firefox version/platform, and the URLs from which to download those plugins. The plugins are hosted by third parties (Cisco and Google), not on Mozilla's servers. Mozilla only hosts the manifest describing where to download them from.

If the GMPs in the update.xml file are different to what is installed, Firefox will update its GMPs to match the update.xml file from AUS. Firefox will download and verify the new GMP, uninstall the old GMP, install the new GMP, and then add the new GMP's path to the mozIGeckoMediaPluginService. The objects that do this are the GMPDownloader and the GMPInstallManager, which are JavaScript modules in the front end code as well.

Note Firefox will take action to ensure its installed GMPs matches whatever is specified in the update.xml file. So if a version of a GMP which is older than what is installed is specified in the update.xml file, Firefox will uninstall the newer version, and download and install the older version. This is to allow a GMP update to be rolled back if a problem is detected with the newer GMP version.

If the AUS server can't be contacted, and no GMPs are installed, Firefox has the URLs of GMPs baked in, and will use those URLs to download the GMPs.

On startup, the GMPProvider also calls mozIGeckoMediaPluginService.addPluginDir() for the ClearKey CDM, passing in its path in the Firefox install directory.

How EME plugins are started in Firefox

The lifecycle for Widevine and ClearKey CDM begins in the content process with content JavaScript calling Navigator.requestMediaKeySystemAccess(). Script passes in a set of MediaKeySystemConfig, and these are passed forward to the MediaKeySystemAccessManager. The MediaKeySystemAccessManager figures out a supported configuration, and if it finds one, returns a MediaKeySystemAccess from which content JavaScript can instantiate a MediaKeys object. 

Once script calls MediaKeySystemAccess.createMediaKeys(), we begin the process of instantiating the plugin. We create a MediaKeys object and a ChromiumCDMProxy object, and call Init() on the proxy. The initialization is asynchronous, so we return a promise to content JavaScript and on success we'll resolve the promise with the MediaKeys instance which can talk to the CDM in the GMP process.

To create a new CDM, ChromiumCDMProxy::Init() calls GeckoMediaPluginService::GetCDM(). This runs in the content process, but since the content process is sandboxed, we can't create a new child process to run the CDM there and then. As we're in the content process, the GeckoMediaPluginService instance we're talking to is a GeckoMediaPluginServiceChild. This calls over to the parent process to retrieve a GMPContentParent bridge. GMPContentParent acts like the GMPParent in the content process. GeckoMediaPluginServiceChild::GetContentParent() retrieves the bridge, and sends a LaunchGMPForNodeId() message to instantiate the plugin in the parent process.

In the non multi-process Firefox case, we still call GeckoMediaPluginService::GetContentParent(), but we end up running GeckoMediaPluginServiceParent::GetContentParent(), which can just instantiate the plugin directly.

When the parent process receives a LaunchGMPForNodeId() message, the GMPServiceParent runs through its list of GMPParents to see if there's one matching the parameters passed over. We check to see if there's an instance from the same NodeId, and if so use that. The NodeId is a hash of the origin requesting the plugin, combined with the top level browsing origin, plus salt. This ensures GMPs from different origins always end up running in different processes, and GMPs running in the same origin run in the same process.

If we don't find an active GMPParent running the requested NodeId, we'll make a copy of a GMPParent matching the parameters, and call LoadProcess() on the new instance. This creates a GMPProcessParent object, which in turn uses GeckoChildProcessHost to run a command line to start the child GMP process. The command line passed to the newly spawned child process causes the GMPProcessChild to run, which creates and initializes the GMPChild, setting up the IPC connection between GMP and Main processes.

The GMPChild delegates most of the business of loading the GMP to the GMPLoader. The GMPLoader opens the plugin library from disk, and starts the Sandbox using the SandboxStarter, which has a different implementation for every platform. Once the sandbox is started, the GMPLoader uses a GMPAdapter parameter to adapt whatever binary interface the plugin exports (the Widevine C API for example) to the match the GMP API. We use the adapter to call into the plugin to instantiate an instance of the CDM. For OpenH264 we simply use a PassThroughAdapter, since the plugin implements the GMP API.

If all that succeeded, we'll send a message reporting success to the parent process, which in turn reports success to the content process, which resolves the JavaScript promise returned by MediaKeySystemAccess.createMediaKeys() with the MediaKeys object, which is now setup to talk to a CDM instance.

Once content JavaScript has a MediaKeys object, it can set it on an HTMLMediaElement using HTMLMediaElement.setMediaKeys().

The MediaKeys object encapsulates the ChromiumCDMProxy, which proxies commands sent to the CDM into calls to ChromiumCDMParent on the GMP thread.

How EME playback works

There are two main cases that we care about here; encrypted content being encountered before a MediaKeys is set on the HTMLMediaElement, or after. Note that the CDM is only usable to the media pipeline once it's been associated with a media element by script calling HTMLMediaElement.setMediaKeys().

If we detect encrypted media streams in the MediaFormatReader's pipeline, and we don't have a CDMProxy, the pipeline will move into a "waiting for keys" state, and not resume playback until content JS has set a MediaKeys on the HTMLMediaElement. Setting a MediaKeys on the HTMLMediaElement causes the encapsulated ChromiumCDMProxy to bubble down past MediaDecoder, through the layers until it ends up on the MediaFormatReader, and the EMEDecoderModule.

Once we've got a CDMProxy pushed down to the MediaFormatReader level, we can use the PDMFactory to create a decoder which can process encrypted samples. The PDMFactory will use the EMEDecoderModule to create the EME MediaDataDecoders, which process the encrypted samples.

The EME MediaDataDecoders talk directly to the ChromiumCDMParent, which they get from the ChromiumCDMProxy on initialization. The ChromiumCDMParent is the IPDL parent actor for communicating with CDMs.

All calls to the ChromiumCDMParent should be made on the GMP thread. Indeed, one of the primary jobs of the ChromiumCDMProxy is to proxy calls made by the MediaKeys on the main thread to the GMP thread so that commands can be sent to the CDM via off main thread IPC.

Any callbacks from the CDM in the GMP process are made onto the ChromiumCDMChild object, and they're sent via PChromiumCDM IPC over to ChromiumCDMParent in the content process. If they're bound for the main thread (i.e. the MediaKeys or MediaKeySession objects), the ChromiumCDMCallbackProxy ensures they're proxied to the main thread.

Before the EME MediaDataDecoders submit samples to the CDM, they first ensure that the samples have a key with which to decrypt the samples. This is achieved by a SamplesWaitingForKey object. We keep a copy in the content process of what keyIds the CDM has reported are usable in the CDMCaps object. The information stored in the CDMCaps about which keys are usable is mirrored in the JavaScript exposed MediaKeySystemStatusMap object.

The MediaDataDecoder's decode operation is asynchronous, and the SamplesWaitingForKey object delays decode operations until the CDM has reported that the keys that the sample requires for decryption are usable. Before sending a sample to the CDM, the EME MediaDataDecoders check with the SamplesWaitingForKey, which looks up in the CDMCaps whether the CDM has reported that the sample's keyId is usable. If not, the SamplesWaitingForKey registers with the CDMCaps for a callback once the key becomes usable. This stalls the decode pipeline until content JavaScript has negotiated a license for the media.

Content JavaScript negotiates licenses by receiving messages from the CDM on the MediaKeySession object, and forwarding those messages on to the license server, and forwarding the response from the license server back to the CDM via the MediaKeySession.update() function. These messages are in turn proxied by the ChromiumCDMProxy to the GMP thread, and result in a call to ChromiumCDMParent and thus an IPC message to the GMP process, and a function call into the CDM there. If the license server sends a valid license, the CDM will report the keyId as usable via a key statuses changed callback.

Once the key becomes usable, the SamplesWaitingForKey gets a callback, and the EME MediaDataDecoder will submit the sample for processing by the CDM and the pipeline unblocks. 

EME on Android

EME on Android is similar in terms of the EME DOM binding and integration with the MediaFormatReader and friends, but it uses a MediaDrmCDMProxy instead of a ChromiumCDMProxy. The MediaDrmCDMProxy doesn't talk to the GMP subsystem, and instead uses the Android platform's inbuilt Widevine APIs to process encrypted samples.

How WebRTC uses OpenH264

WebRTC uses OpenH264 for encode and decode of baseline H.264 streams. It doesn't need all the DRM stuff, so it talks to the OpenH264 GMP via the PGMPVideoDecoder and PGMPVideoEncoder protocols.

The child actors GMPVideoDecoderChild and GMPVideoEncoderChild talk to OpenH264, which conforms to the GMP API.

OpenH264 is not used by Firefox for playback of H264 content inside regular <video>, though there is still a GMPVideoDecoder MediaDataDecoder in the tree should this ever be desired.

How GMP shutdown works

Shutdown is confusing, because there are three processes involved. When the destructor of the MediaKeys object in the content process is run (possibly because it's been cycle or garbage collected), it calls CDMProxy::Shutdown(), which calls through to ChromiumCDMParent::Shutdown(), which cancels pending decrypt/decode operations, and sends a Destroy message to the ChromiumCDMChild.

In the GMP process, ChromiumCDMChild::RecvDestroy() shuts down and deletes the CDM instance, and sends a __delete__ message back to the ChromiumCDMParent in the content process.

In the content process, ChromiumCDMParent::Recv__delete__() calls GMPContentParent::ChromiumCDMDestroyed(), which calls CloseIfUnused(). The GMPContentParent tracks the living protocol actors for this plugin instance in this content process, and CloseIfUnused() checks if they're all shutdown. If so, we unlink the GMPContentParent from the GeckoMediaPluginServiceChild (which is PGMPContent protocol's manager), and close the GMPContentParent instance. This shuts down the bridge between the content and GMP processes.

This causes the GMPContentChild in the GMP process to be removed from the GMPChild in GMPChild::GMPContentChildActorDestroy(). This sends a GMPContentChildDestroyed message to GMPParent in the main process.

In the main process, GMPParent::RecvPGMPContentChildDestroyed() checks if all actors on its side are destroyed (i.e. if all content processes' bridges to this GMP process are shutdown), and will shutdown the child process if so. Otherwise we'll check again the next time one of the GMPContentParents shuts down. 

Note there are a few places where we use GMPContentParent::CloseBlocker. This stops us from shutting down the child process when there are no active actors, but we still need the process alive. This is useful for keeping the child alive in the time between operations, for example after we've retrieved the GMPContentParent, but before we've created the ChromiumCDM (or some other) protocol actor.

How crash reporting works for EME CDMs

Crash handling for EME CDMs is confusing for the same reason as shutdown; because there are three processes involved. It's tricky because the crash is first reported in the parent process, but we need state from the content process in order to identify which tabs need to show the crash reporter notification box.

We receive a GMPParent::ActorDestroy() callback in the main process with aWhy==AbnormalShutdown. We get the crash dump ID, and dispatch a task to run GMPNotifyObservers() on the main thread. This collects some details, including the pluginID, and dispatches an observer service notification "gmp-plugin-crash".  A JavaScript module ContentCrashHandlers.jsm observes this notification, and rebroadcasts it to the content processes.

JavaScript in every content process observes the rebroadcast, and calls mozIGeckoMediaPluginService::RunPluginCrashCallbacks(), passing in the plugin ID. Each content process' GeckoMediaPluginService then goes through its list of GMPCrashHelpers, and finds those which match the pluginID. We then dispatch a PluginCrashed event at the window that the GMPCrashHelper reports as the current window owning the plugin. This is then handled by PluginChild.jsm, which sends a message to cause the crash reporter notification bar to show.

GMP crash reporting for WebRTC

Unfortunately, the code paths for WebRTC handling crashes is slightly different, due to their window being owned by PeerConnection. They don't use GMPCrashHelpers, they have PeerConnection help find the target window to dispatch PluginCrashed to.

Quick start: Profiling local builds of Firefox for Android and GeckoView_example

Getting building and profiling Firefox for Android or GeckoView_example is relatively easy if you know how, so here's my quickstart guide.

See also, the official GeckoView documentation.

First, ensure you run ./mach boostrap, and select "4. GeckoView/Firefox for Android".

Here's the mozconfig I'm using (Ubuntu 18.04):

ac_add_options --enable-optimize
ac_add_options --disable-debug
ac_add_options --enable-release
ac_add_options --disable-tests
mk_add_options AUTOCLOBBER=1
ac_add_options --enable-debug-symbols
# With the following compiler toolchain:
CC="/home/chris/.mozbuild/clang/bin/clang"
CXX="/home/chris/.mozbuild/clang/bin/clang++"
export CC="/home/chris/.mozbuild/clang/bin/clang -fcolor-diagnostics"
export CXX="/home/chris/.mozbuild/clang/bin/clang++ -fcolor-diagnostics"
ac_add_options --with-ccache=/usr/bin/ccache
mk_add_options 'export RUSTC_WRAPPER=sccache'
# Build GeckoView/Firefox for Android:
ac_add_options --enable-application=mobile/android
# Work around issues with mozbuild not finding the exact JDK that works.
# See also https://bugzilla.mozilla.org/show_bug.cgi?id=1451447#c7
ac_add_options --with-java-bin-path=/usr/lib/jvm/java-8-openjdk-amd64/bin
# With the following Android NDK:
ac_add_options --with-android-ndk="/home/chris/.mozbuild/android-ndk-r17b"
ac_add_options --with-android-min-sdk=16
ac_add_options --target=arm-linux-androideabi

A noteworthy item in there is "--with-java-bin-path". I've had trouble on Ubuntu with the system default Java not being the right version. This helps.

Note that if you're profiling, you really want to be doing a release build. The behaviour of release is different from an optimized build.

If you're debuging, you probably need --enable-debug. For details of how to debug, see GeckoView Debugging Native Code in Android Studio.

To build, package, and install Firefox for Android (Fennec) on your Android device, run:

./mach build && ./mach package && ./mach install 

Note that you need to do the package step after every build. Once you've installed, you can start Firefox on a given URL with:

./mach run --url https://www.youtube.com/watch?v=dQw4w9WgXcQ

For testing and profiling GeckoView, the easiest option is to run the GeckoView_example app. To build and install this, run:

./mach build && ./mach package && ./mach android build-geckoview_example && ./mach android install-geckoview_example

To run GeckoView_example, opening a URL:

adb shell am start -a android.intent.action.MAIN -c android.intent.category.LAUNCHER -n org.mozilla.geckoview_example/org.mozilla.geckoview_example.GeckoViewActivity -d 'https://www.youtube.com/watch?v=dQw4w9WgXcQ'

If you want to set environment variables, for example to turn on MOZ_LOGs, run like so:

adb shell am start -a android.intent.action.MAIN -c android.intent.category.LAUNCHER -n org.mozilla.geckoview_example/org.mozilla.geckoview_example.GeckoViewActivity -d 'https://www.youtube.com/watch?v=dQw4w9WgXcQ' --es env0 MOZ_LOG=MediaSource:5

Note if you want to create more than one environment variable, each one needs to be numberd, i.e. `--es env0 FOO=BAR env1 BAZ=FUZ`, and so on. Also note that you do not put quotes around environment variables here. That is, use `--es env0 FOO=BAR`, do not use `--es env0 FOO="BAR"`.

MOZ_LOGs go to adb logcat. To setup an output stream that reads specific MOZ_LOGs:

adb logcat | grep MediaSource

This stays open, printing logs until you terminate with CTRL+C. If you want to exit at the end of the logs buffered, pass -d. i.e.:

adb logcat -d > log_file.txt

Apparently you can pass a logtag filterspec to `adb logcat` to have it filter for you, but I never figured the syntax out.

To clear logcat's buffered logs:

adb logcat --clear

This is useful if you're prinf-debugging something via logcat, and want to clear the decks before each run.

Other useful commands...

To terminate a running GeckoView_example:

adb shell am force-stop org.mozilla.geckoview_example

To list all packages on your device related to Mozilla:

adb shell pm list packages mozilla

To uninstall a GeckoView_example:

adb uninstall org.mozilla.geckoview_example && adb uninstall org.mozilla.geckoview_example.test

Note that this also uninstalls the GeckoView test app. Sometimes you may find you need to uninstall both apps before you can re-install. I think this is related to different versions of adb interacting.

To get the Android version on your device:

adb shell getprop ro.build.version.release

To simulate typing text:

adb shell input text "your text"

To profile a GeckoView_example session, you need to download the latest Firefox Desktop Nightly build, and install the Firefox Profiler add-on. Note that the Firefox Profiler Documentation is pretty good, so I'll only cover the highlights.

Once you've got Firefox Desktop Nightly and the Firefox Profiler add-on installed, start up your GeckoView_example app and URL you want to profile, and in Firefox Nightly Desktop open about:debugging. Click "Connect" to attach to the device you want to profile on, and then click "Profile Performance".

If you're profiling media playback, you want to add "Media" to the custom thread names under the "Threads" settings.

Since you're profiling a local build, you want to open the "Local build" settings, and ensure you add the path to your object directory.

Once you're configured, press "Start recording", do the thing in GeckoView_example you're profiling, and then hit "Stop and grab the recording".

The profile will open in a new tab in the browser. Sometimes I've noticed that the profiler hangs at "Waiting for symbol tables for library libxul.so". Just reloading the page seems to resovle this normally.

I find the Firefox Profiler very straightforward to use. The Flame Graph view can be particularly enlightening to see where threads are spending time.

Unfortunately the Firefox profiler can't symbollocate Java call stacks. Java calls usually show up as hex addresses, sometimes on the far side of an AndroidBridge C++ call.

On Android >= P you can use Simpleperf to capture profiles with both native and JIT'd Java call stacks. Andrew Creskey has instructions on how to use Simpleperf with GeckoView_example.

On learning Go and a comparison with Rust

I spoke at the AKL Rust Meetup last month (slides) about my side project doing data mining in Rust. There were a number of engineers from Movio there who use Go, and I've been keen for a while to learn Go and compare it with Rust and Python for my data mining side projects, so that inspired me to knuckle down and learn Go.

Go is super simple. I was able to learn the important points in a couple of evenings by reading GoByExample, and I very quickly had an implementation of the FPGrowth algorithm in Go up and running. For reference, I also have implementations of FPGrowth in Rust, Python, Java and C++. 

As a language, Go is very simple. The language lacks many of the higher level constructs of other modern languages, but the lack of these make it very easy to learn, straightforward to use, and easy to read and understand. It feels similar to Python. There's little hidden functionality; you can't overload operators for example, and there's no generics or macros, so the implementation for everything has to be rewritten for every type. This gets tedious, but it does at least mean the implementation for everything is simple and explicit, the code right in front of you.

I also really miss the functional constructs that are built into many other languages, like mapping a function over a sequence, filter, any, all, etc. With Go, you need to reimplement these yourself, and because there's no generics (yet), you need to do it for every type you want to use these on. The lack of generics is also painful when writing custom containers.

Not being able to key a map with a struct containing a slice was a nuisance for my problem domain; I ended up having to write a custom tree-set data structure due to this; though it was very easy to write thanks to in built maps. Whereas Rust, or even Java, has traits/functions you can implement to ensure things can be hashed.

The package management for Go feels a bit tacked on; requiring all Go projects to be in a GO_PATH seems a consequence of not having a tool the equal of Rust's Cargo coupled with something like crates.io.

And Go's design decision to use the case of a symbol's first letter to express whether that symbol is public or private is annoying. I have a long standing habit of using foo as the name for a single instance of type Foo, but that pattern doesn't work in Go. The consequence of this design choice is it leads programmers to using lots of non-descriptive names for things. Like single letter variable names. Or the dreaded myFoo.

The memory model of Go is simple, and again I think the simplicity is a strength of the language. Go uses escape analysis to determine whether a value escapes outside of a scope, and moves such values to the heap if so. Go also dynamically grows goroutines' stacks, so there's no stack overflow. Go is garbage collected, so you don't have to worry about deallocating things.

I found that thinking of values as being on the heap or stack wasn't a helpful mental model with Go. Once I started to think of variables as references to values and values being shared when I took the address (via the & operator), the memory model clicked.

I think Go's simple memory model and syntax make it a good candidate as a language to teach to beginner programmers, more so than Rust.

The build times are impressively fast, particularly on an incremental build. After the initial build of my project, I was getting build times to fast to perceive on my 2015 13" MBP, which is impressive. Rust has vastly slower build time.

The error messages produced by the Go compiler were very spartan. The Rust compiler produces very helpful error messages, and in general I think Rust is leading here.

Go has a very easy to use profile package which you can embed in your Go program. Combined with GraphViz, it produces simple CPU utilization graphs like this one:

CPU profile graph produced by Go's "profile" package and GraphViz.

Having an easy to use profiler bundled with your app is a huge plus. As we've seen with Firefox, this makes it easy for your users to send you profiles of their workloads on their own hardware. The graph visualization is also very simple to understand.

The fact that Go lacks the ability to mark variables/parameters as immutable is mind-boggling to me. Given the language designers came from C, I'm surprised by this. I've written enough multi-threaded and large system code to know the value of restricting what can mess with your state.

Goroutines are pretty lightweight and neat. You can also use them to make a simple "generator" object; spawn a goroutine to do your stateful computation, and yield each result by pushing it into a channel. The consumer can block on receiving the next value by receiving on the channel, and the producer will block when it pushes into a channel that's not yet been received on. Note you could do this with Rust too, but you'd have to spawn an OS thread to do this, which is more heavy weight than a goroutine, which are basically userspace threads.

Rust's Rayon parallelism crate is simply awesome, and using that I was able to easily and effectively parallelize my Rust FPGrowth implementation using Rayon's parallel-iterators. As best as I can tell, Go doesn't have anything on par with Rayon for parallelism. Go's goroutines are great for lightweight concurrency, but they don't make it as easy as using's Rayon's par_iter() to trivially parallelize a loop. Note, parallelism is not concurrency.

All of my attempts to parallelize my Go FPGrowth implementation as naively as I'd parallelized my Rust+Rayon implementation resulted in a slower Go program. In order to parallelize FPGrowth in Go, I'd have to do something complicated, though I'm sure channels and goroutines would make that easier than in a traditional language like Java or C++.

Go would really benefit from something like Rayon, but unfortunately due to Go's lack of immutability and a borrow checker, it's not safe to naively parallelize arbitrary loops like it is in Rust. So Rust wins on parallelism. Both languages are strong on concurrency, but Rust pulls ahead due to its safety features and Rayon.

Comparing Rust to Go is inevitable... Go to me feels like the spiritual successor to C, whereas Rust is the successor to C++.

I feel that Rust has a learning curve, and before you're over the hump, it can be hard to appreciate the benefits of the constraints Rust enforces. For Go, you get over that hump a lot sooner. Whereas with Rust, you get over that hump a lot later, but the heights you reach after are much higher.

Overall, I think Rust is superior, but if I'd learned Go first I'd probably be quite happy with Go.

Firefox Media Playback Team Review Policy

Reviews form a central part of how we at Mozilla ensure engineering diligence. Prompt, yet thorough, reviews are also a critical component in maintaining team velocity and productivity. Reviews are also one of the primary ways that a distributed organization like Mozilla does its mentoring and development of team members.

So given how important reviews are, it pays to be deliberate about what you're aiming for.

The senior members of the Firefox Media Playback team met in Auckland in August 2016 to codify the roadmap, vision, and policy for the team, and and one of the things we agreed upon was our review policy.

The policy has served us well, as I think we've demonstrated with all we've achieved, so I'm sharing it here in the hope that it inspires others.

• Having fast reviews is a core value of the media team.
• Review should be complete by end of next business day.
• One patch for one logical scope or change. Don't cram everything into one patch!
• Do not fix a problem, fix the cause. Workarounds are typically bad. Look at the big picture and find the cause.
• We should strive for a review to be clear. In all cases it should be clear what the next course of action is.
• Reviews are there to keep bad code out of the tree.
• Bad code tends to bring out bad reviews.
• Commit message should describe what the commit does and why. It should describe the old bad behaviour, and the new good behaviour, and why the change needs to be made.
• R+ means I don’t want to see it again. Maybe with comments that must be addressed before landing.
• R- means I do want to see it again, with a list of things to fix.
• R canceled means we’re not going to review this.
• Anyone on the media team should be expected to complete a follow up bug.
• It’s not OK for a reviewer to ask a test to be split out from a changeset, provided the test is related to the commit. By the time a patch gets to review, splitting the test out doesn’t create value, just stop-energy.
• Review request. If response is slow, ping or email for a reminder, otherwise find another reviewer.
• Don’t be afraid to ask when the review will come. The reply to “when” can be “is it urgent?”
• Everyone should feel comfortable pointing out flaws/bugs as a “drive by”.
• Give people as much responsibility as they can handle.
• Reviewers should make it clear what they haven’t reviewed.
• American English spelling, for comments and code.
• Enforce Mozilla coding style, and encourage auto formatters, like `./mach clang-format`.
• Use reviewboard. Except when you can’t, like security patches.

Not every bit of code you write needs to be optimal

It's easy to fall into the trap of obsessing about performance and try to micro-optimize every little detail in the code you're writing. Or reviewing for that matter. Most of the time, this just adds complexity and is a waste of effort.

If a piece of code only runs a few (or even a few hundred) times a second, a few nanoseconds per invocation won't make a significant difference. Chances are the performance wins you'll gain by micro optimizing such code won't show up on a profile.

Given that, what should you do instead? Code is read and edited much more than it is written, so optimize for readability, and maintainability.

If you find yourself wondering whether a piece of code is making your program slow, one of the first things you should do is fire up a profiler, and measure it. Or add telemetry to report how long your function takes in the wild. Then you can stop guessing, and start doing science.

If data shows that your code is slow, by all means optimize it. But if not, you can get more impact out of your time by directing your efforts elsewhere.

How to install Ubuntu 17.04 on Dell XPS 15 9550

I had some trouble installing Ubuntu 17.04 to dual-boot with Windows 10 on my Dell XPS 15 9550, so documenting here in case it helps others...

Once I got Ubuntu installed, it runs well. I'm using the NVIDIA proprietary driver, and I've had no major issues with hardware yet so far.

Most of the installation hurdles for me were caused by Ubuntu not being able to see the disk drive while it was operating in Raid mode, and UEFI/Secure Boot seemed to block the install somehow.

The trick to getting past these hurdles was to set Windows to boot into Safe Mode and then switch the disk drive to AHCI and disable UEFI in the BIOS before booting back into Windows in Safe Mode, and then switching Windows back to non-Safe Mode.

I found rcasero's notes on installing Ubuntu on Dell XPS 15 9560 useful.

Detailed steps to install...

1. (If your Windows partition is encrypted, print out a copy of your BitLocker key. You'll need to enter this on boot after changing anything in your BIOS.
2. Boot into Windows 10.
3. I also needed to resize my main Windows partition from inside Windows; the Ubuntu installer seemed unable to cope with resizing my encrypted Windows partition for some reason. You can resize your main Windows partition using Windows' "Create or edit Disk Partitions" tool.
4. Configure Windows to boot into safe mode: Press Win+R and run msconfig.exe > Boot > Safe Mode. Reboot.
5. Press the F12 key while the BIOS splash screen comes up. Just repeatedly pressing it while the machine is booting seems to be the most reliable tactic.
6. In the BIOS menu, BIOS Setup > System Configuration > SATA Operation, change "RAID On" to "AHCI".
7. In the BIOS menu, disable Secure Boot.
8. Reboot into Windows. You'll need to enter your BitLocker key to unlock the drive since the BIOS changed. Windows will boot into Safe Mode. If you don't have your Windows install set to boot into Safe Mode, you'll get a BSOD.
9. Once you've booted into Windows Safe Mode, you can configure Windows to boot in normal (non-Safe Mode) with msconfig.exe > Boot > Safe Mode again.
10. Reboot with your Ubuntu USB Live Disk inserted, and press F12 while booting to select to boot from the Live USB disk.
11. The rest of the install Just Worked.
12. Once you've installed Ubuntu, for better reliability and performance, enable the proprietary GPU drivers, in System Settings > Software and Updates > Additional Drivers. I enabled the NVIDIA and Intel drivers.
13. I've found the touchpad often registers clicks while I'm typing. Turning off System Settings > Mouse and Touchpad > "Tap to click" fixed this and gives pretty good touchpad behaviour.
14. Firefox by default has its hardware accelerated layers disabled, but force-enabling it seems to work fine. Open "about:config" in Firefox, and toggle "layers.acceleration.force-enabled" to true. Restart Firefox.

Firefox video playback's skip-to-next-keyframe behavior

One of the quirks of Firefox's video playback stack is our skip-to-next-keyframe behavior. The purpose of this blog post is to document the tradeoffs skip-to-next-keyframe makes.

The fundamental question that skip-to-next-keyframe answers is, "what do we do when the video stream decode can't keep up with the playback speed?

Video playback is a classic producer/consumer problem. You need to ensure that your audio and video stream decoders produce decoded samples at a rate no less that the rate at which the audio/video streams need to be rendered. You also don't want to produce decoded samples at a rate too much greater than the consumption rate, else you'll waste memory.

For example, if we're running on a low end PC, playing a 30 frames per second video, and the CPU is so slow that it can only decode an average of 10 frames per second, we're not going to be able to display all video frames.

This is also complicated by our video stack's legacy threading model. Our first video decoding implementation did the decoding of video and audio streams in the same thread. We assumed that we were using software decoding, because we were supporting Ogg/Theora/Vorbis, and later WebM/VP8/Vorbis, which are only commonly available in software.

The pseudo code for our "decode thread" used to go something like this:
 
while (!AudioDecodeFinished() || !VideoDecodeFinished()) {
  if (!HaveEnoughAudioDecoded()) {
    DecodeSomeAudio();
  }
  if (!HaveEnoughVideoDecoded()) {
    DecodeSomeVideo();
  }
  if (HaveLotsOfAudioDecoded() && HaveLotsOfVideoDecoded()) {
    SleepUntilRunningLowOnDecodedData();
  }
}
 
This was an unfortunate design, but it certainly made some parts of our code much simpler and easier to write.

We've recently refactored our code, so it no longer looks like this, but for some of the older backends that we support (Ogg, WebM, and MP4 using GStreamer on Linux), the pseudocode is still effectively (but not explicitly or obviously) this. MP4 on Windows, MacOSX, and Android in Firefox 36 and later now decode asynchronously, so we are not limited to decoding only on one thread.

The consequence of decoding audio and video on the same thread only really bites on low end hardware. I have an old Lenovo x131e netbook, which on some videos can take 400ms to decode a Theora keyframe. Since we use the same thread to decode audio as video, if we don't have at least 400ms of audio already decoded while we're decoding such a frame, we'll get an "audio underrun". This is where we don't have enough audio decoded to keep up with playback, and so we end up glitching the audio stream. This sounds is very jarring to the listener.

Humans are very sensitive to sound; the audio stream glitching is much more jarring to a human observer than dropping a few video frames. The tradeoff we made was to sacrifice the video stream playback in order to not glitch the audio stream playback. This is where skip-to-next-keyframe comes in.

With skip-to-next-keyframe, our pseudo code becomes:

while (!AudioDecodeFinished() || !VideoDecodeFinished()) {
  if (!HaveEnoughAudioDecoded()) {
    DecodeSomeAudio();
  }
  if (!HaveEnoughVideoDecoded()) {
    bool skipToNextKeyframe =
      (AmountOfDecodedAudio < LowAudioThreshold()) ||
       HaveRunOutOfDecodedVideoFrames();
    DecodeSomeVideo(skipToNextKeyframe);
  }
  if (HaveLotsOfAudioDecoded() && HaveLotsOfVideoDecoded()) {
    SleepUntilRunningLowOnDecodedData();
  }
}

We also monitor how long a video frame decode takes, and if a decode takes longer than the low-audio-threshold, we increase the low-audio-threshold.

If we pass a true value for skipToNextKeyframe to the decoder, it is supposed to give up and skip its decode up to the next keyframe. That is, don't try to decode anything between now and the next keyframe.

Video frames are typically encoded as a sequence of full images (called "key frames", "reference frames", or  I-frames in H.264) and then some number of frames which are "diffs" from the key frame (P-Frames in H.264 speak). (H.264 also has B-frames which are a combination of diffs of frames frames both before and after the current frame, which can lead the encoded stream to be muxed out-of-order).

The idea here is that we deliberately drop video frames in the hope that we give time back to the audio decode, so we are less likely to get audio glitches.

Our implementation of this idea is not particularly good.

Often on low end Windows machines playing HD videos without hardware accelerated video decoding, you'll get a run of say half a second of video decoded, and then we'll skip everything up to the next keyframe (a couple of seconds), before playing another half a second, and then skipping again, ad nasuem, giving a slightly weird experience. Or in the extreme, you can end up with only getting the keyframes decoded, or even no frames if we can't get the keyframes decoded in time. Or if it works well enough, you can still get a couple of audio glitches at the start of playback until the low-audio-threshold adapts to a large enough value, and then playback is smooth.

The FirefoxOS MediaOmxReader also never implemented skip-to-next-keyframe correctly, our behavior there is particularly bad. This is compounded by the fact that FirefoxOS typically runs on lower end hardware anyway. The MediaOmxReader doesn't actually skip decode to the next keyframe, it decodes to the next keyframe. This will cause the video decode to hog the decode thread for even longer; this will give the audio decode even less time, which is the exact opposite of what you want to do. What they should do is skip the demux of video up to the next keyframe, but if I recall correctly there was bugs in the Android platform's video decoder library that FirefoxOS is based on that caused this to be unreliable.

All these issues occur because we share the same thread for audio and video decoding. This year we invested some time refactoring our video playback stack to be asynchronous. This enables backends that support it to do their decoding asynchronously, on another own thread. So since audio decodes on a separate thread to video, we should have glitch-free audio even when the video decode can't keep up, even without engaging skip-to-next-keyframe. We still need to do something like skipping the video decode when the video decode is falling behind, but it can probably engage less aggressively.

I did a quick test the other day on a low end Windows 8.0 tablet with an Atom Z2760 CPU with skip-to-next-keyframe disabled and async decoding enabled, and although the video decode falls behind and gets out of sync with audio (frames are rendered late) we never glitched audio.

So I think it's time to revisit our skip-to-next-keyframe logic, since we don't need to sacrifice video decode to ensure that audio playback doesn't glitch.

When using async decoding we still need some mechanism like skip-to-next-keyframe to ensure that when the video decode falls behind it can catch up. The existing logic to engage skip-to-next-keyframe also performs that role, but often we enter skip-to-next-keyframe and start dropping frames when video decode actually could keep up if we just gave it a chance. This often happens when switching streams during MSE playback.

Now that we have async decoding, we should experiment with modifying the HaveRunOutOfDecodedVideoFrames() logic to be more lenient, to avoid unnecessary frame drops during MSE playback. One idea would be to only engage skip-to-next-keyframe if we've missed several frames. We need to experiment on low end hardware.

How to prefetch video/audio files for uninterrupted playback in HTML5 video/audio

Sometimes when you're playing a media file using an HTML5 <video> or <audio> element, or with WebAudio, you really want to be sure that the whole audio/video file is totally downloaded before you start playing it. For example, you may be writing a game, and you want to be sure all your sound effects are preloaded, so there's no delay between your animations and your sound effects while the network downloads the remainder of the file.

So how can you be sure a media resource is fully downloaded before beginning playing it? You could wait for the "canplaythrough" event to fire on all your media elements, but that event is not fired correctly by Chrome.

A more reliable solution is to prefetch the video/audio file using XHR/AJAX requests, and play the video/audio from a Blob URI.

Here's a simple snippet of a JS file that downloads a file using XHR. The function accepts callbacks to return results.

function prefetch_file(url,
                       fetched_callback,
                       progress_callback,
                       error_callback) {
  var xhr = new XMLHttpRequest();
  xhr.open("GET", url, true);
  xhr.responseType = "blob";

  xhr.addEventListener("load", function () {
    if (xhr.status === 200) {
      var URL = window.URL || window.webkitURL;
      var blob_url = URL.createObjectURL(xhr.response);
      fetched_callback(blob_url);
    } else {
      error_callback();
    }
  }, false);

  var prev_pc = 0;
  xhr.addEventListener("progress", function(event) {
    if (event.lengthComputable) {
      var pc = Math.round((event.loaded / event.total) * 100);
      if (pc != prev_pc) {
        prev_pc = pc;
        progress_callback(pc);
      }
    }
  });
  xhr.send();
}

When the file is successfully downloaded, the fetched_callback is called with an argument which is the blob URI. You can simply set this as the src of an audio or video element and can then play the fully-downloaded resource. You can also set the same blob URI as the src of multiple audio/video elements, and the downloaded data won't be re-downloaded or duplicated/copied in memory.

There's also a progress_callback that's called with a percentage complete parameter as the file is downloaded, and an error_callback that's called when the download fails.

For a working demo: prefetching a video file before playback using HTML5 demo

Why does the HTML fullscreen API ask for approval after entering fullscreen, rather than before?

The HTML fullscreen API is a little different from other JS APIs that require permission, in that it doesn't ask permission before entering fullscreen, it asks forgiveness *after* entering fullscreen.

Firefox's fullscreen approval dialog, which asks "forgiveness" rather than permission.

The rationale for having our fullscreen API implementation ask forgiveness rather than request permission is to make it easier on script authors.

When the original API was designed, we had a number of HTML/JS APIs like the geolocation API that would ask permission. The user was prompted to approve, deny, or ignore the request, though they could re-retrieve the request later from an icon in the URL bar to approve the request at a later time.

Geolocation approval dialog, from Dive Into HTML's geolocation example.

The problem with this design for script authors is that they can't tell if the user has ignored the approval request, or is just about to go back and approve it by bringing up the geolocation door-hanger again.

This model of requesting permission has been seen to cause problems for web apps in the wild using the geolocation API. Often if a user ignores the geolocation permission request, the web app doesn't work right, and if you approve the request some time later, the site often doesn't start working correctly. The app just doesn't know if it should throw up a warning, or if it's about to be granted permission.

So the original developers of the fullscreen spec (Robert O'Callahan, and later I and others were involved), opted to solve this problem by having our implementation ask forgiveness. Once you've entered fullscreen, the user is asked to confirm the action.

This forces the user to approve or deny the request immediately, and this means that script will immediately know whether fullscreen was engaged, so script will know whether it needs to take its fallback path or not.

Note that the specification for requestFullscreen() defines that most of the requestFullscreen() algorithm should run asynchronously, so there is scope to change the fullscreen approval dialog to being a permission request before entering fullscreen instead if future maintainers, or other implementors/browser, wish to do so.

What does the H.264/avc1 codecs parameters for video/mp4 mime types mean?

The HTMLMediaElement.canPlayType() API enables you to query what video formats a user agent can play. For "video/mp4", the container for H.264/AAC, you can specify a "codecs" parameter that denotes the the H264 profile and level. Firefox doesn't currently handle MP4 codecs parameter very well, so I took it upon myself to figure out what the codecs parameters mean for H.264.

According to RFC6381 The 'Codecs' and 'Profiles' Parameters for "Bucket" Media Types, codecs paremeters for H.264 are contained in the "avc1" sample entry, and are is represented as follows:

avc1.PPCCLL

That is, the string "avc1." (or "avc2.", I'm not sure what the difference is yet), followed by 3 bytes represented in hex without the "0x" prefix, where the bytes represent the following:

PP = profile_idc
CC = constraint_set flags
LL = level_idc

These fields are defined in in Annex 1 of ITU-T H.264 and ISO/IEC 14496-10:2012  twinned standards. ITU-T H.264 can be downloaded for free.

profile_idc defines the H.264 profile. ITU-T H.264 doesn't have a single table listing what the different profile_idc values mean, but handily, Microsoft defines an eAVEncH264VProfile enumeration on the decimal values of the profile_idc in Codecapi.h (available on Win7):

enum eAVEncH264VProfile {
  eAVEncH264VProfile_unknown                    = 0,
  eAVEncH264VProfile_Simple                     = 66,
  eAVEncH264VProfile_Base                       = 66,
  eAVEncH264VProfile_Main                       = 77,
  eAVEncH264VProfile_High                       = 100,
  eAVEncH264VProfile_422                        = 122,
  eAVEncH264VProfile_High10                     = 110,
  eAVEncH264VProfile_444                        = 144,
  eAVEncH264VProfile_Extended                   = 88,
  eAVEncH264VProfile_ScalableBase               = 83,
  eAVEncH264VProfile_ScalableHigh               = 86,
  eAVEncH264VProfile_MultiviewHigh              = 118,
  eAVEncH264VProfile_StereoHigh                 = 128,
  eAVEncH264VProfile_ConstrainedBase            = 256,
  eAVEncH264VProfile_UCConstrainedHigh          = 257,
  eAVEncH264VProfile_UCScalableConstrainedBase  = 258,
  eAVEncH264VProfile_UCScalableConstrainedHigh  = 259
};

So for example, avc1.4D401E has a profile_idc of 0x4D, which is 77 in decimal, so it's main profile.

constraint_set flags are encoded as a bit flags 6 bitfields named constraint_set0_flag through to constraint_set5_flag. The meaning of a constraint_setN_flag being set depends on the profile being represented. The bits are stored in with constraint_set0_flag in the high bit, and so there are two padding 0s in the low bits. So for example, avc1.4D401E, the contraint_set flags are 0x40, so that's 01000000b, thus this means constraint_set1_flag is set.

The level number, level_idc, is defined as fixed point, from 1 to 5.2, with an oddball 1b level, as defined in ITU-T H.264 in Table A-1 - Level Limits. The level_idc is encoded in the codecs parameter as 10 times the level number, i.e. level 5.1 is represented as decimal 51, or 0x33 hex. So continuing our example avc1.4D401E, the level_idc is 1E, or 30 decimal, so level 3.0.

Level 1b is an oddball encoding, and is encoded as 11 decimal with the constraint_set3_flag equal to 1. If the constraint_set3_flag is not equal to 1, level 1.1 is encoded.

Microsoft also conveniently define this in the eAVEncH264VLevel enumeration.

enum eAVEncH264VLevel {
  eAVEncH264VLevel1    = 10,
  eAVEncH264VLevel1_b  = 11,
  eAVEncH264VLevel1_1  = 11,
  eAVEncH264VLevel1_2  = 12,
  eAVEncH264VLevel1_3  = 13,
  eAVEncH264VLevel2    = 20,
  eAVEncH264VLevel2_1  = 21,
  eAVEncH264VLevel2_2  = 22,
  eAVEncH264VLevel3    = 30,
  eAVEncH264VLevel3_1  = 31,
  eAVEncH264VLevel3_2  = 32,
  eAVEncH264VLevel4    = 40,
  eAVEncH264VLevel4_1  = 41,
  eAVEncH264VLevel4_2  = 42,
  eAVEncH264VLevel5    = 50,
  eAVEncH264VLevel5_1  = 51,
  eAVEncH264VLevel5_2  = 51
}

Mozilla at the New Zealand Programming Contest

On Saturday Mozilla sponsored the Auckland site of the New Zealand Programming Contest. We supplied t-shirts for the participants, competed in the competition, and we provided pizza for dinner afterwards.

Word must have got out that we were giving away swag, as the contest had double the normal participants than usual, around 120 people, and we ran out of t-shirts!

I've been wanting to do this for a while. I learned a lot about coding during training for the programming contest while I was at university, so I think it's a great way to encourage the next generation to hone their skills.

We're also looking for interns to join us over the summer, so I also took the opportunity to make a plug for our 2013 Mozilla Auckland Internship intake. I think it's a good way to get targeted exposure to the types of people we want to hire too.

We did well in the competition too, largely thanks to Edwin Flores, who formerly represented Australasia at the ACM Programming Contest world finals a few years back. Go Team!

Hardware accelerated H.264 decoding landed in Firefox on Windows Vista and later

I have been hard at work getting our H.264 support on Windows Vista and later hardware accelerated using DXVA2. I'm happy to say that this finally landed in Firefox 23 Nightly builds. With this patch we'll use the GPU to accelerate H.264 video decoding when possible on Windows, which greatly reduces our CPU and power usage.

If you spot a bug in H.264 video in Firefox Nightly builds, please file a bug in Core: Audio/Video. Bonus points if you toggle the pref "media.windows-media-foundation.use-dxva" to false, reload the video, and tell me whether the bug still happens!

Notes:

1. H.264/AAC/MP3 support on Windows 7 and later is shipping in Release builds in Firefox 21 next week, Vista gets it in Firefox 22.
2. Edwin Flores is working on our H.264/AAC/MP3 support on Mac and Linux.
3. I'm currently working on getting MP3 support for Windows XP using DirectShow, it will probably land next cycle (Firefox 24).

Reducing Windows' background CPU load while building Firefox

If you're building Firefox on Windows 8 like I am you might want to tweak the following settings to reduce the OS' background CPU load while you're building Firefox (some of these settings may also be applicable to Windows 7, but I haven't test this):

1. Add your src/object directory to Windows Defender's list of locations excluded from real time scans. I realized that the "Antimalware Service Executable" was using up to 40% CPU utilization during builds before I did this. You can add your src/objdir to the exclude list using the UI at: Windows Defender > Settings > Excluded files and locations.
2. Remove your src/objdir from Windows' list of locations to be indexed. I actually did the inverse, and removed my home directory (which my src/objdir was inside) from the list of indexable locations and re-added the specific subdirectories in my home dir that I wanted indexed (Documents, etc) without re-adding my src/objdir.

Update, 11 July 2014: Recently the "Antimalware Service Execuable" started hogging CPU again while building, so I added MSVC's cl.exe, and link.exe, to the list of "Excluded Processes" in Windows Defender > Settings, and that reduced "Antimalware Service Execuable"'s CPU usage while building.

H.264/AAC/MP3 support now enabled by default in Firefox Nightlies on Windows 7 and later

Support for playing H.264/AAC in MP4 and support for playing MP3 audio files in HTML5 <video> and <audio> elements has now been switched on by default for Windows 7 and later in the Firefox Nightly channel. You no longer need to set the pref to enable it. Please test MP4/MP3 support in Firefox Nightly builds, and file bugs in the "Core:: Video/Audio" component in Bugzilla.

Assuming no catastrophic bugs are found, H.264/AAC/MP3 support in Firefox on Windows 7 and later should ship in Firefox 22 21, which is scheduled to be released around June 25 May 14.

Edit 25 Feb 2013: Corrected target release, it's Firefox 21, not Firefox 22.

HTML5 video playbackRate and Ogg chaining support landed in Firefox

Paul Adenot has recently landed patches in Firefox to enable the playbackRate attribute on HTML5 <audio> and <video> elements.

This is a cool feature that I've been looking forward to for a while; it means users can speed up playback of videos (and audio) so that you can for example watch presentations sped up and only slow down for the interesting bits. Currently Firefox's <video> controls don't have support for playbackRate, but it is accessible from JavaScript, and hopefully we'll get support added to our built-in controls soon.

Paul has also finally landed support for Ogg chaining. This has been strongly desired for quite some time by community members, and the final patch also had contributions from "oneman" (David Richards), who also was a strong advocate for this feature.

We decided to reduce the scope of our chaining implementation in order to make it easier and quicker to implement. We targeted the features most desired by internet radio providers, and so we only support chaining in Ogg Vorbis and Opus audio files and we disable seeking in chained files.

Thanks Paul and David for working on these features!

Experimental H.264,AAC, and MP3 support in Firefox Nightly builds on Windows 7 and later

As the Internet has already discovered, recently I landed patches to add a Windows Media Foundation playback backend for Firefox. This is preff'd off by default.

This allows playback of H.264 video and AAC audio in MP4 and M4A files, and MP3 audio files in HTML5 <audio> and <video> elements in Firefox on Windows 7 and later.

To test MP4/MP3 playback, download the latest Firefox Nightly Build, and toggle the pref "media.windows-media-foundation.enabled" to "true" in about:config.

There are a few bugs that I'm aware of, which is why this landed preff'd off by default, but if you spot any bugs, please file a bug in Firefox's "Core :: Video/Audio" component.

Enabling external monitor on Lenovo W530 with Nvidia Discrete Graphics and Ubuntu Linux 12.04

I've recently acquired a Lenovo W530 laptop. It's nice. Running Ubuntu 12.04 and with 32GB of RAM and an SSD it builds Firefox with cold/empty caches in 14 minutes flat, which is pretty good!

It was a trial figuring out how to get the Nvidia graphics card and a second monitor working however. The current Ubuntu stable nvidia-current package doesn't include support for the Quadro K1000M chipset, and the official Nvidia drivers available from Nvidia.com didn't work for me, and left my libGL.so's missing when I uninstalled them.

I eventually figured out the steps to get a compatible Nvidia driver installed and displaying a second monitor:

1. Install the "X Updates" team's PPA, and install the latest Nvidia drivers, which supports the Quadro K1000M. In the terminal enter:

sudo add-apt-repository ppa:ubuntu-x-swat/x-updates
sudo apt-get update
sudo apt-get install nvidia-current

2. Reboot. At the Lenovo BIOS screen press "Enter" to interrupt normal startup, F1 to enter BIOS setup utility. Change Config > Display > Graphics Device to Discrete Graphics. This means the hardware will attempt to use only the Nvidia graphics card, not the Intel integrated graphics. Press F10 to save and boot the computer.

3. Once you've booted up and logged in, to setup multiple monitors you'll need to run nvidia-settings from the terminal as root:

sudo nvidia-settings

Setup your displays in the "X Server Display Configuration" tab (you may need to "Detect Displays" first, and you probably want "TwinView" configuration). Once you've configured your displays, make sure you "Save to X Configuration File". When I saved my config file the path save was blank. If this happens to you, enter: /etc/X11/xorg.conf

3. (Updated 13 October 2012) Once you've booted up and logged in, you can use Ubuntu's "Displays" application to easily configure your secondary displays. Note that if you use nvidia-settings to write an xorg.conf (as I previously suggested) X won't detect when you unplug a monitor, and so when you unplug a monitor X may open windows on the monitor which is no longer connected, i.e. where you can't see them, which can be very inconvenient!

4. To get your hardware brightness keys working (i..e Fn+F8/F9), edit the /etc/X11/xorg.conf file as root:

sudo gedit /etc/X11/xorg.conf 

...and add the following line to the "Device" section for the "Quadro K1000M":

    Option         "RegistryDwords" "EnableBrightnessControl=1"

Log out/log back in and then your brightness keys should work!

4. (Updated 13 October 2012) To get your display brightness keys working (i..e Fn+F8/F9) you need to tell the Nvidia graphics driver to enable the brightness controls.

These days X automatically configures itself, so you can't just edit the xorg.conf file, you instead need to add a section to a file in /usr/share/X11/xorg.conf.d/ and X will include that section in the configuration that it automatically generates.

So to get the screen brightness keys working with your Nvidia graphics card, create a file in the xorg.conf.d directory, e.g:

sudo gedit /usr/share/X11/xorg.conf.d/10-nvidia-brightness.conf

Paste the following into the file:

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "Quadro K1000M"
    Option         "RegistryDwords" "EnableBrightnessControl=1"
EndSection

Log out and log back in, or reboot, or simply kill X with CTRL+ALT+BACKSPACE, or ALT+PrtSc+K, and your brightness keys should now work!

Caveat: I've only had a second monitor working on my VGA output, I'm looking forward to my mini-DVI adapter arriving so I can have a cripser DVI output. Hopefully that will Just Work....

Update 13 October 2012: a second monitor Just Worked with my mini-DVI adapter. I was able to get two external displays running off a docking station. Possibly the docking station could drive more, I haven't felt the need, three displays is enough for me!

And unfortunately my laptop refuses to boot with Discrete Graphics enabled, the Nvidia driver installed, and with virtualization enhancements enabled in the BIOS. Apparently it is possible with the help of the BumbleBee project however.

Update 13 October 2012: I was able to boot with virtualization enhancements enabled in BIOS if I specified the nox2apic kernel command parameter. I haven't done enough research to figure out whether disabling x2apic is a good idea or not.