I spoke at GOTO Amsterdam a few weeks ago. I was really thrilled to be back in the Netherlands after so many years! So thanks to Sergi Mansilla, who curated the HTML5 track, and the organisation in general for bringing me there!
The talk wasn’t recorded, but I made a screencast just in case you really want to listen to me. I am also posting the outline/notes I wrote, and they differ in places because I don’t read them during the talk (I don’t even have them handy) and I sometimes went a bit off topic, but that’s the beauty of improvisation!
Here are the slides, and the slides source code just in case you wanted it too.
On to the notes-expect some MASSIVE GIFs and amazingly clever photomanipulation! tee hee hee!
Invest in the future: build for the web!
Hey all and thanks for coming! My name is Soledad Penadés (@supersole in Twitter) and I am an Apps Engineer at Mozilla. But some time before that… I worked on Android apps.
They sold us this “dream” that you could earn a living with Android apps. And of course, who wouldn’t want to earn enough money building things they loved? So I did this sort of market research and tried to figure out what I wanted to do. The first thing I tried to build was an audio tool, but it was too slow and too glitchy. I kept wanting to do something which had to do with graphics or audio but I didn’t want to build games; they need things such as a story, audio, artwork, etc, and I didn’t think I could do all of them, but I still didn’t know what to build.
One day I was walking around in London and I saw something really cool. I wanted to take a picture but I realised I had forgotten my camera. I thought: well maybe I could take the picture with the phone instead! But it was really freezing, so I was wearing gloves. And although you can buy gloves with conductive tips nowadays, I didn’t know that back then, so I had to take out the glove in order to touch the screen and take the picture. Also, the interface for selecting different options was really slow to use, and it was specially painful when standing there in the cold!
Photography apps
So I had this idea: what if I built a camera app that was better than the stock camera app?
It would…
be less clunky than stock – it would be faster to navigate the options
use the hardware buttons – so you wouldn’t need to touch the screen to take pictures
silent – I was so tired of the shutter sound because it would scare my sister’s cats away each time I tried to take a picture of them
And that was the first photography app I built. Shortly afterwards I decided I wanted to play a little bit more with the Camera API and thought that I could maybe build an app that would apply filters to images. At that time Instagram didn’t exist for Android yet, and the existing filter apps were about taking a picture and applying postprocessing afterwards, so the process wasn’t as fluid and playful as I would have liked it to be.
I then built this realtime effects app that would take the live stream off the camera and process the image in real time.
Good feedback, but…
The apps were generally well welcomed and the feedback was mostly positive, but still there were lots of hardware issues that I couldn’t test because I either didn’t have the phones or couldn’t buy them either. Some people were using weird Chinese built rebranded phones and it was really difficult to find out where the error actually was-was it the OS version, was it the extra rebranded stuff, was it the actual phone? And even if I wanted to buy the phones, sometimes it was impossible because they were exclusive to a certain geography, so my only solution would have been buying them from eBay.
On the other hand, people liked the filters from my app and they started to ask for ports to their favourite operating system. iOS? Blackberry? (yes, Blackberry was still a thing back then). Even desktop platforms: Windows? Mac OS? That was driving me crazy because I really liked being liked and appreciated the positive feedback, but there was just no way I could please them. I was running Linux, for whatever it’s worth!
Interactive picture books start-up
At some point I decided to start working for others again, so I ended up joining a start-up that wanted to enable authors make great interactive picture books. There was work going on in two fronts: first the Mac OS authoring environment, which was the tool for the authors and where they could assemble their text, images, and possibly sounds and animations. This would then be saved into an XML based format that could in turn be read and ‘executed’ by the second ‘front’, which was the engine. There were two versions of the engine: iOS and Android.
Layout was DIFFICULT
I was just working on the Android side, and there we probably spent like 70% of our time fighting the operating system to make things look and behave the way we wanted them to work. The worst part was that there was no preview (specially when using Custom Views) or that the preview differed from the actual results. You then had to push the app to the device, and because it was really rich in graphics it would took so long that when it finished, one minute or so later, you had totally forgotten what you wanted to look at. You’d ask yourself: what was that thing I was working on? I can’t remember. Oh well, I guess I’ll just have another coffee, or something. Ironically, for complex layouts -specially layouts with lots of text- we actually had to use an embedded WebView because it was just faster to layout using HTML and CSS than styling “the Android way”.
Another big issue was that native animations were quite limited. The books had transitions between pages, and often within each page too, and we wanted those to be smooth. But the native animations didn’t offer many parameters, and felt really mechanical. Plus the elements we tried to animate often ended up being really janky! We weren’t sure of where the problem was. Was it our code? Was it the way we were animating? If we were using Android’s native animations, should we have garbage collection issues? And since we weren’t sure at all, we had to spend inordinate amounts of time in the profiler, trying to trace where did those slowdowns come from. Still, jank continued to be there! Plus weird graphical glitches, and we kept finding cargo cult tricks in Stack Overflow and applying them out of desperation, but of course, the jank was still there!
And what about testing… LOL about testing to be quite honest. I would just say that at some point we just half joked about creating a kickstarter to try and get some Google engineer to make it possible to test asynchronous code for reals, because no matter how many testing frameworks and solutions we tried, none would work.
Bad habits, deeply ingrained
There were other issues which were less about the tech and more about the process and the mindset. The biggest issue was that “different output sizes” was treated as an exception and not as the norm. So when a project was initially released for the iPad 1, the designs only considered those dimensions and aspect ratios, which made it really hard to port to variable sized environments such as Android.
For example, the graphic design would call for an overlap on top of the menu that would be sized exactly for the iPad 1, so using that same overlay in an Android tablet, which had a wider screen aspect ratio size, ended up stretching the image in a bad way.
Imagine when the retina iPads were introduced. It was fun fun fun.
One day I woke up and wondered…
why is this not HTML+JS+CSS?
So I asked my boss, which politely replied that he had been experimenting a lot with it, but…
… the web is not ready yet…
… you can’t have smooth animations and audio in the browser…
I hadn’t actually tried to build the engine in HTML5, so what could I argue against that? So I just said “OK” and continued work.
Contracting for local newspaper
We built this app for a local newspaper which was, pretty much, a glorified RSS reader.
With the exception that the layout kept getting more and more complex, and even then we had lots of embedded WebViews for more complicated things and for rendering the articles (which were, basically, the HTML pulled raw from the RSS feeds) and embedded JavaScript for things such as adding favourites and bookmarks.
This app kept getting new features and more complex layout until we reached this point in which we reached the maximum limit of nested views. But–and here’s the funny thing– NOT ALL the devices exhibited that error.
We ended up reaching that point in which the only solution we could suggest was:
hey, so… what about a total rewrite of this less than two years app?
One day I woke up and wondered again…
Why is this not HTML+JS+CSS?
And again I asked my boss. And again I got a similar answer:
… because the web is not ready yet…
… you can’t store offline data…
… you can’t have push notifications…
which only mildly convinced me, so I muttered a meh “A-ha” and kept working on the thing.
One day I was updating the company website
And opened DevTools to live edit it. And I had an epiphany.
We were recreating browsers again and again because “the web is not ready”, but I had had enough.
I said to myself:
I’m out of this madness, and back to the web.
Back with a vengeance!
Two sides of the fence
But I’d be back after having been to the two sides of the fence. I knew about the upsides and the downsides, and I decided that I wanted to teach people about the upsides and help them take the maximum advantage of them, and work in fixing the downsides.
So…
I joined Mozilla
We want YOU to build for the web
Why build for the web?
Of course you will be asking me why should I bother writing for the Web?
Well, the first reason is that it is the only non proprietary platform. No one owns it, and no one controls it. Compare that to the Windows or Apple platforms, and where sometimes you’re not even allowed to discuss the issues you’re having when developing because you’re under an NDA.
And it’s also the closest to Write Once Run Everywhere you’ll ever get. As a sort of secondary bonus, writing for the web, using standards, gives you a high chance that whatever you build will still work in the future. Compare that to trying to get some old binaries to work in newer platforms–specially when you don’t have access to the source code or to the toolchain that generated those binaries in the past.
Also, fragmentation is not an issue: it’s business as usual. And encouraged. The web is about enabling people to access and work on their data, and if they prefer to use a cheaper phone with dithered screens because they can’t afford to get a super high end device, then we shouldn’t be adding artificial barriers. Working this way, taking into account that your apps might be running on a variety of devices widens your base of customers, and at the same time cuts down on development costs–because you don’t need to rewrite the same app for each of the native platforms that customers might use.
And finally, because it is everywhere–e-books, TV set top boxes, GPS trackers, mobile native via WebKit views, native via PhoneGap, Ludei, AppCelerator, desktop environments (GNOME 3), even Mac OS scripting… so maybe all these people cannot be wrong.
We helped unlock desktop browsers from monopoly
Just as Mozilla helped unlock desktop browsers from monopoly with Firefox OS…
We’re doing the same in mobile with Firefox OS
And doing “the same” means giving power back to developers and enabling them to build awesome cool stuff, which means definining and implementing new JavaScript APIs for accessing features that so far have only been accessible to native code.
New APIs
These are some of the new APIs that are accessible for code running in Firefox OS:
Network Information
Bluetooth
Mobile Connection
Network Stats
TCP Socket
Telephony
WebSMS
WiFi Information
Ambient Light Sensor
Battery Status
Proximity
Device Orientation
Screen Orientation
Vibration
WebFM
Camera
Power Management
FileHandle
Contacts
Device Storage
Settings
Alarm
Simple Push
Web Notifications
Web Activities
WebPayment
Browser
Idle
Permissions
Time/Clock
Most of these APIs are prefixed for now, but that’s because Firefox OS is the testing ground–the place where APIs are not only designed but also put in practice and battle tested with real users, apps and needs. And these APIs are also submitted to the appropriate standards tracks, so that in the future not only Firefox OS, but all browsers can use this specification to implement those features, which in turn gives more power to developers to do more amazing stuff that works everywhere.
Or in other words: our guiding goal is to help shaping standards, not to build a proprietary OS in JS.
Existing APIs
Existing Web APIs which are implemented in most of the browsers do need some love for mobile implementations too. They have to be efficient.
That might mean that the C/C++ code that does something in Firefox Desktop has to be rewritten to take advantage of the ARM processor or the hardware decoders that come with the phone and can do that way faster than a software decoder would. Or maybe read and writes to storage need to be optimised for battery usage.
These are some of those APIs:
WebGL
Web Audio
WebRTC
Geolocation
Pointer Lock
IndexedDB
The conclusion to this is that spending developer efforts in making APIs available and code runnable anywhere has a multiplier effect where way more people benefit than if those developers simply focused their efforts on building native apps.
But there’s still more work to be done…
Over two billion people still don’t have access to the Internet
That’s right, 2,000,000,000+ people still can’t “browse” just like you and me can.
At Mozilla we believe the Internet must be open and accessible, so we are working on fixing this too. We partnered with some manufacturers to make a phone that would run Firefox OS and also be affordable.
$25 phone
This phone is in the same price bracket that the feature phones that are sold in many countries yet runs entirely on Firefox OS and has access to 2.5 EDGE networks. It might look underpowered and “crappy” compared to whatever you have in your pocket right now, but compared to those featurephones this is a massive step forward: updatable and a wider range of apps and services can be accessed because it is, basically, a portable browser, rather than a J2ME device.
Better tooling
We are also aware that the web is so much more than just documents today, and we need to have better tools.
These are some examples of the tools that have been added to Firefox:
Responsive design mode
This allows you to quickly see how your app looks and behaves in differently sized viewports, and you can also store presets to go back to those specific values.
Network + cache inspector
Not only lets you inspect the requests that happen when you load a page in your app, but it also allows you to see the difference in requests when you load the page for the first time compared to when it’s loaded a second time. So you can make sure that static content is correctly cached, and your app is as fast as it can be.
Scratchpad
This is a sort of a notepad where you can both write code with autocompletion and suggestions and also execute it on the context of the page you’re on. This will help you debug and experiment without having to go to an external editor to write your code and then reloading the page.
Canvas inspector
This lets you capture all the drawing requests that go to a canvas context between two requestAnimationFrame calls, and then also lets you replay them so you can see what is being done and can both understand how the drawing is made and if there are any extra or redundant calls that could be optimised.
Shader editor
Just as the JavaScript debugger lets you inspect the code in the current page, the shader editor lets you inspect the code in the current WebGL context. And also allows you to pick any program and edit it with instant results. This is really a great tool for developers using WebGL since being able to test your shaders in the right context makes a huge difference compared to editing them and seeing the results being applied to a cube rotating over a white background.
Web Audio Editor
This is the last addition to the tools, so it’s still a bit early times for this inspector! It will let you inspect the Web Audio context node graph, and if you click on a node it will let you change its parameters on the fly, so again you can get instant feedback.
The graph is also very helpful when trying to debug complex hierarchies that might have been built programmatically and thus aren’t easily understandable just by looking at the code-the graph provides a much better visual insight.
Polyfills and libraries
Despite all the work in implementing APIs and pushing them to be standards, this process moves slowly and sometimes the needs of developers are way ahead of the needs of implementors. Luckily JavaScript is flexible enough that it’s often possible to write polyfills and libraries that will either fill the gap or provide some proof of concept API that could maybe become a native API later on if enough people adopt it.
So there is also a number of libraries that we develop to make writing apps more enjoyable.
Brick
Brick is a carefully curated selection of web components to build the UI of a modern web app. You can write your own elements using web components, and this in turn generates a sort of lightweight DSL where you have things such as the header of the app, the main content with a deck element and several cards on it, and a footer with a tabbar in it. The tabbar is linked to the deck, and so when you click on a given tab, the corresponding card in the deck will become active and be brought to the foreground.
Phonegap + Firefox OS
Despite our efforts to make the web the best platform, some features are still only available to native code unless you use something like PhoneGap-then you can keep writing your app in HTML+JS+CSS.
We want to make it easy for PhoneGap developers and we are working on plugins for Firefox OS, so the apps can be exported to Firefox OS too.
I know this all sounds ironic–building native apps using HTML? But I’d like to quote Brian Leroux, one of the stewards of PhoneGap, who gave a talk here past year:
our ultimate goal is to cease to exist
and I really liked the spirit in that sentence. For some things to change, we sometimes need transition solutions, and PhoneGap is one of those.
localForage
While new APIs such as ServiceWorkers get implemented, developers still need to store content for offline access, and the existing set of APIs was very inconsistent and unpleasant to use, so we built this library-mega-polyfill that provides a unified simple interface while trying to use the most efficient solution under the hood without bothering the developer with decisions. So for example if IndexedDB is available it will try to use that but exposing an API that is way less hostile than IndexedDB, and if IndexedDB is not available, as was the case in most iOS platforms until recently, it will try and use WebSQL if available, and if that doesn’t work either it will try to defer back to good old localStorage, which is the slowest of them all because it’s synchronous, so the whole script blocks until localStorage operations have finished.
// In localStorage, we would do:
localStorage.setItem('key', JSON.stringify('value'));
doSomethingElse();
// With localForage, we use callbacks:
localforage.setItem('key', 'value', doSomethingElse);
// Or we can use Promises:
localforage.setItem('key', 'value').then(doSomethingElse);
Animated_GIF
This is another example of API that maybe should be available in the browser but until it is, JavaScript nowadays is powerful enough that this can work decently fast even if performing lots of arithmetic operations.
var imgs = document.querySelectorAll('img');
var ag = new Animated_GIF();
var animatedImage = document.createElement('img');
ag.setSize(320, 240);
for(var i = 0; i imgs.length; i++) {
ag.addFrame(imgs[i]);
}
ag.getBase64GIF(function(image) {
animatedImage.src = image;
document.body.appendChild(animatedImage);
});
This is an example of the output you can generate: a GIF adding several frames, using dithering and an specific color palette.
Maybe in the future some spec writer will look at this and say oh, this is disgusting, what a poor API design! I will design something so much better and submit it to a standards track!
And I’ll be super happy to hear that.
Yeah, but…
Of course not everything is perfect. Of course there’s still lots of work to do.
Are you missing a feature? Don’t just complain, get involved!
This is not the 90s anymore
You can do so much more than complain because your browser vendor hasn’t updated the browser in years and bugs still plague you left, right and center.
The Web is YOURS, so shape it! Get involved!
Getting involved means you make informed decisions. What works? What doesn’t? Why? And… is there a work around?
Getting involved means your needs are taken into account. You can say We need this feature for our use case…, or also This feature can’t work for this reason… and it will be listened to.
Ultimately it boils down to this:
A W3C editor sitting on their chair on a lonely room will never know about your needs unless you tell them.
Ways to get involved
Always assume good intent, and be respectful. You might have had a bad day trying to build your app, and it isn’t working and you can’t figure out why, but it’s not the job of other people to listen to a parade of hateful swearing. Also if your plea for help apparently goes unnoticed, maybe wait a bit more until someone has the time to pick it and answer you. It’s not that people are unresposive–it’s just they might just be busy!
Find your channel: mailing lists, GitHub repos, IRC, meet ups, the Extensible web summit…. Find the channel that makes you feel more comfortable, and study it for a while-you don’t need to get active from day 1. It’s OK to lurk for a while until you get familiarised with the dynamics of the place. Then…
Ask questions. Questions are incredibly valuable for everyone. Phrasing your question in a way that is clear is an skill that requires development, but will help you actually understand what is it that you do not understand. And being asked a question will provide useful feedback to an implementor: it is a way of letting them know that something in the API is not as clear as they think it is, and that needs improvement.
Try your code in different browsers. Often developers try their code in a more permissive browser and when they release their app to the world they start getting reports that it doesn’t work as well as they expected in other, more restrictive browsers. It’s not that the other browsers are wrong but maybe you didn’t test enough. Or maybe they are-so this is a fantastic opportunity to read the spec and ensure you got it right. Maybe this will highlight other errors in your code?
Try nightlies for preview features. This will allow you to make sure your code keeps working even when new versions of an API are introduced. Maybe the API breaks in some edge cases that only your code uses. Is this in the spec? If it is and it shouldn’t behave this way, you should bring that up. If it’s not, maybe that should be in the spec to make things clearer. You can help here!
For the braver souls, maybe you could even learn to compile your favourite browser! And maybe you can even help fix issues that bother you.
File bugs. If you don’t think you can fix anything yet, but keep seeing things that don’t work as they should, you should file bugs. Often some issues only arise when certain factors fall in place at the same time. There are so many configurations and there’s no way to reproduce them all.
Build things. It’s the only way to put APIs to a test, and because each developer is unique, it’s the only way to try APIs in unique ways.
Break things! It’s OK if things break! It either highlights that you didn’t understand something correctly, because it’s unclear or because you need to give another go to reading the specs, or maybe it’s that you found something that truly didn’t work correctly. Congratulations!
File more bugs! You should not keep those extra bugs to yourself. Share them with implementors and browser vendors and help make the web even better!
For the Web to be ours, it needs everyone’s input.
Let’s build this together.