Saturday, October 29, 2011

Integrating Google Page Speed in your Web site build

Google Page Speed is a great tool to help analyze and optimize the performance of your Web pages. It's actually a family of tools:

- Page Speed Chrome and Firefox browser extensions complement Firebug and analyze the performance of your Web pages right from your Web browser.

- Page Speed Online analyzes the performance of your Web pages too without requiring a browser extension.

- Mod_pagespeed is an Apache HTTPD extension module that optimizes, rewrites and minifies your Web pages on the fly.

- The Page Speed Service is a proxy hosted by Google which optimizes and delivers your Web pages from Google's servers.

- The Page Speed SDK allows you to embed the Page Speed analyzers and optimizers in your own project.

If you're paranoid (don't want your Web site to depend on Google's Page Speed servers) and CPU + memory conscious (don't want to spend resources / $$$ running mod_pagespeed on your HTTP server) you can also run Page Speed on your pages ahead of time when you build your Web site.

It's pretty simple. Here's how I'm doing it on Linux using GNU Automake and Apache HTTPD (I'm sure that can be adapted to other environments):

1. Download and build the Page Speed SDK, like this:
curl -OL
cd page-speed-1.9
make builddir=$build/page-speed-1.9-bin

That'll build two useful command line tools: minify_html_bin, an HTML rewriter / minifier which also minifies inline CSS and Javascript, and jsmin_bin, a Javascript minifier which also works well for CSS.

2. Write the following in your Automake file:
minified = htdocs/index-min.html htdocs/foo-min.js htdocs/bar-min.css

sampledir = ${prefix}
nobase_dist_sample_DATA = ${minified}

SUFFIXES = -min.html -min.js -min.css
    ../page-speed-1.9-bin/minify_html_bin $< $@

    ../page-speed-1.9-bin/jsmin_min < $< > $@

    ../page-speed-1.9-bin/jsmin_min < $< > $@

CLEANFILES = ${minified}

I won't go into the Automake specifics now but these rules will run the Page Speed minifiers on your Javascript, CSS and HTML pages as part of your Make build. I'm using a simple naming convention for the minified files, index-min.html is the minified version of index.html.

3. Add a DirectoryIndex directive to your Apache HTTPD httpd.conf config file:
DirectoryIndex index-min.html index.html

That tells HTTPD to serve /index-min.html (the minified page) instead of the original /index.html page when a user points his browser to

4. Reference foo-min.js and bar-min.css in your HTML pages instead of foo.js or bar.css for example.

5. After making changes to your Web pages, build your site like this:

If you had /index.html, foo.js, and bar.css, you should now have /index-min.html, foo-min.js, bar-min.css, all nicely optimized, rewritten and minified by Page Speed.

To clean up the minified files, run:
make clean

That's it. That little trick should normally save you from 30% to 50% bandwidth, CPU, memory and disk space on the client devices that access your site (particularly useful with resource-constrained mobile devices) and on your HTTP server too (which now serves smaller files).

Hope this helps

Tuesday, October 25, 2011

Recursive Functions of Symbolic Expressions and their Computation by Machine

'Recursive Functions of Symbolic Expressions and their Computation by Machine' is the original paper on Lisp from John Mc Carthy. That paper appeared in Communications of the ACM in April 1960. It is a great short read.

John Mc Carthy passed away yesterday. After Steve Jobs and Dennis Ritchie earlier this month, October has been a sad month for the computing community...

Another good article from Paul Graham, 'The Roots of Lisp' helps explain John Mc Carthy's discoveries with Lisp.

If you don't know Lisp, you should learn it.

I had a brief look at it 15 years ago, but was too impatient and unexperimented to get the point then. I took more time to actually learn it on a summer vacation 2 years ago, then went on to learn Scheme (a Lisp dialect) as I was going through the Structure and Interpretation of Computer Programs classic Computer Science book from MIT.

Believe me. Lisp (or Scheme) will transform you into a different programmer. After that, even if you still have to program in C, C++, Python, or even Java, you're in a different world. You'll see programming through different eyes, and there will be no going back.

Think about a few fun facts... Pretty much everything you manipulate in Lisp is a list of values (Lisp stands for 'list processing'). Values can be numbers, strings, symbols or lists. Data and code are interchangeable and represented in the same way, as lists of values. You can write a Lisp interpreter in half a page of Lisp!

So, if you're working with 'modern' OO programming languages, scripting languages, or functional programming languages, if you're using XML dialects, JSON, DSLs, and you're still struggling to map between documents, data, objects, services, code and configuration... spend the time to learn Lisp. It'll open your eyes.

Thursday, October 20, 2011

Caching Images in a Mobile Web app

Another post on a technique to help make mobile Web apps work offline.

Last time I showed how to use XMLHttpRequest and local storage to download and cache Javascript scripts. Here I'm going to show how to download and cache an image, and then inject it dynamically into your HTML page.

That technique can be particularly useful to optimize an image intensive app (a game or a photo gallery app for example) and allow it to work offline.

First place base64-encoded (as defined in Internet RFC 3548) versions of your images on your Web server. On Mac OS X for example, convert foo.png to a base64-encoded foo.b64 file like this:
$ base64 -w 0 foo.png >foo.b64

Configure your Web server to serve .b64 files as text/plain. If you're using the Apache httpd 2.x Web server -- and if you're not using it, you should :) -- add the following to your httpd.conf server configuration file:
text/plain b64

The reason why we want to convert our images to base64-encoded is that it'll make it much easier to use them in the HTML5 app pages. We're getting to that part now.

Place the following Javascript script in your HTML page, under the <head> element for example:
<script type="text/javascript">
window.appcache = {};

// Get a resource
window.appcache.get = function(url) {
  var doc = localStorage.getItem(url);
  if (doc != null)
    return doc;
  var http = new XMLHttpRequest();"GET", url, false);
  if (http.status != 200)
    return null;
  localStorage.setItem(url, http.responseText);
  return http.responseText;

// Convert an image to a data: URL
window.appcache.img = function(url) {
  var b64 = window.appcache.get(url);
  return 'data:image/png;base64,' + b64;

I'm not showing the error handling code to keep it short, but you get the picture. The get(url) function downloads a resource and caches it in local storage. The img(url) function gets an image resource and converts it to a data: URL. A data: URL (as defined in Internet RFC 2397) allows you to include resource data (here our base64-encoded image data) immediately in the URL itself.

Now, later in your HTML page, say you have an <img/> element like this:
<img id="foo"/>

You can set the cached image into it dynamically, like this:
<script type="text/javascript">
var foo = document.getElementById('foo');
foo.src = window.appcache.img('/foo.b64');

The image src property will recognize the data: URL, the image/png;base64 media type at the beginning of the URL, and read the base64-encoded content as a PNG image.

That's it. With the little trick I've described here you should now be able to:
  • control the download of your images;
  • cache them in local storage for speed and working offline;
  • inject them dynamically into your HTML page as needed.

Hope this helps. In the next few days I'll show how to handle CSS style sheets in a similar way.

Sunday, October 16, 2011

Game Industry Forecast - Rapid Growth and a Changing Market

Interesting study and forecast on the game industry there, as Zynga is preparing for IPO and announced 10 new products this week.

The global game industry will generate $60 billions in revenue for 2011. It's not a surprise, but gaming is the only media business growing right now with a rapid growth driven by mobile games, as games are the leading apps on smartphones and tablets.

The game market seems to be changing rapidly from hardcore gamers (mostly teenage boys) to casual gamers with older households (35+yr old), and women (42% of the gamers now), as browsers and mobile are bringing new populations to casual and social games.

The game technical platforms are also shifting quickly to the Web browser. With technologies like HTML5, Canvas, SVG, and WebGL for example, Web browsers have become the most convenient platform for games, and it's now clear that games in browsers are the future. See this interview of the founder of Electronic Arts for more insight and his thoughts on what's happening.

Clearly, that means challenges ahead for the established game companies which have to adapt quickly. That also means great opportunities for new players to come and disrupt the mobile game market!!

Friday, October 14, 2011

PhoneGap / Apache Callback project accepted in the Apache Incubator

The vote to accept the Apache Callback project in the Apache Software Foundation Incubator was open for the last 72 hours, and that vote just passed earlier today.

Apache Callback is the free open source software evolution of the popular PhoneGap project.

Apache Callback is a platform for building native mobile applications using HTML, CSS and JavaScript. Apache Callback allows Web developers to natively target Apple iOS, Google Android, RIM BlackBerry, Microsoft Windows Phone 7, HP webOS, Nokia Symbian and Samsung Bada with a single codebase. The Callback APIs are based on open Web standards and enable access to native device capabilities, including the device camera, compass, accelerometer, microphone or address book for example, from HTML5 apps.

That's really great news for the mobile app developer community! ... and I'll be watching this project very closely in the next few months!

Dennis Ritchie: The Shoulders Steve Jobs Stood On

Dennis Ritchie: The Shoulders Steve Jobs Stood On.

Another great loss. Dennis Ritchie was the father of the C Programming Language and the UNIX operating system, which pretty much the entire Internet and computer industry run on.

I'm writing this on a Macbook Air running Mac OS X, a UNIX-based system, and programs mostly written in C.

His book "The C Programming Language" was my first Computer Science book. My first serious encounter with software was a 2-week job to teach the C language to a team of programmers. I was just trying to make enough money to go on vacation, but then I got hooked and decided that I wanted to be a C coder... Feeling sad today.

#include <stdio.h>

main() {
  printf("goodbye, world\n");

Thank You Sir. RIP.

Caching Javascript in a Mobile Web app

Earlier this week I promised I'd post a useful technique to reference, pre-fetch, and cache Javascripts, as a follow up to my review of an article on HTMLRocks describing HTML5 techniques for optimizing mobile Web performance.

So, here it goes.

The HTML5Rocks article showed how to follow all the links in a page and pre-fetch and cache their target pages in local storage. That was a good starting point, but for a mobile Web app to really work well offline you also need to fetch and cache the referenced JavaScripts, CSS, and images.

Here's how I do that in my apps.

I place the following utility Javascript at the top of my page under the <head> element:
<script type="text/javascript";gt;
window.appcache = {};

// Get a resource
window.appcache.get = function(url) {
  var doc = localStorage.getItem(url);
  if (doc != null)
    return doc;
  var http = new XMLHttpRequest();"GET", url, false);
  if (http.status != 200)
    return null;
  localStorage.setItem(url, http.responseText);
  return http.responseText;

// Load a script
window.appcache.script = function(url, parent) {
  var e = document.createElement('script');
  e.type = 'text/javascript';
  e.text = window.appcache.get(url);

I'm not showing the error handling code here to keep it short, but you get the picture. The get(url) function downloads a resource and caches it, and the script(url, parent) function gets a resource and creates a script element with it.

Then to reference a Javascript script later in my page, instead of writing the usual:
<script type="text/javascript" src="foo.js"></script>

I write this:
<script type="text/javascript">
window.appcache.script('foo.js', document.head);

If you're loading pages in an <iframe> nested inside a main page as suggested in the HTML5Rocks article, you don't need to repeat the utility script in all your pages. Just have it once in the main page, then in the nested pages just write:
<script type="text/javascript">
window.parent.appcache.script('foo.js', document.head);

Or just set the appcache property on your <iframe> like this:
var nested = document.createElement('iframe');
nested.contentWindow.appcache = window.appcache;

Then you can write the same code everywhere:
<script type="text/javascript">
window.appcache.script('foo.js', document.head);

Hope this helps.

To the folks who've pinged me several times this week asking for this tip: Sorry for the delay but I've been kinda busy... I'll show how to fetch and cache CSS (similar to Javascripts) and images (using a data: url and base64-encoded image content) in the next few posts, probably this weekend.

Sunday, October 9, 2011

Offline Mobile Web Apps - Really...

A few days ago I posted a review of an article on HTMLRocks describing HTML5 techniques for optimizing mobile Web performance, in particular a technique to pre-fetch and cache HTML pages and resources, useful to improve navigation performance and allow the mobile Web app to work offline.

In that review I said:
"What HTML5Rocks doesn't describe is how to fetch and cache all the resources referenced by these pages, like CSS style sheets, JavaScript scripts, or images, and that's a lot more work..."

and concluded:
"My 2c... The techniques described in that HTML5Rocks post are fine to get you started, but implementing them is not so simple. As usual, the devil is in the details."

Since then a number of folks have pinged me to challenge and ask:
"So, how do you fetch the referenced resources? and what are these details?"

Fair question :) I'm going to address it in a series of posts in the next few days. I'll also describe a few additional issues and tricks to make a mobile Web app really work offline, and some solutions with code snippets.

Here's a draft outline of the next few posts:
  • Referencing, fetching and caching Javascript scripts
  • Caching CSS stylesheets
  • Caching images
  • HTML5 W3C application cache vs a DIY cache
  • Monitoring your network connection

Stay tuned...

Friday, October 7, 2011

Google Summer of Code 2011 T-shirt - Thanks Google

Received my Google Summer of Code 2011 T-shirt gift from Google today, as I was a mentor for GSoC @ Apache this year again.

Hoping my students have received theirs too (that may take a little longer as they're in Sri-Lanka).

I hope you guys had a lot fun with your coding projects this summer. I enjoyed working with you! Keep up the good work, and look for opportunities to work on and learn from Open Source again!

Silicon Valley Code Camp 2011 - By Developers, for Developers

Silicon Valley Code Camp 2011 is this weekend in  Los Altos Hills.

The camp is free to attend, run by volunteers, with lots of great speakers and interesting sessions. It's also a great opportunity to network with fellow developers.

Here's my session selection:

HTML5 Game Programming - WebGL Edition
Build a mobile web app with Sencha Touch
Faster Mobile Anyone?
Kids Programming Workshop with Scratch
Hands on jQuery Mobile

AutoMobile: the Next Hot Platform for Mobile Linux
Create a Kinect Powered Personal Robot with Microsoft Robotics Developer Studio
The Ecosystem of Context (exploiting user context to go beyond search)
GPU Accelerated Databases using OpenCL

Thursday, October 6, 2011

HTML5 rocks the mobile Web

Great post about HTML5 techniques for mobile on 'HTML5Rocks'. Not sure I agree with everything it says though.

It describes three HTML5 techniques for optimizing mobile Web performance:
  1. For smooth native-feeling sliding and flipping page transitions, use a translate3d(0,0,0) CSS transform to force the phone's Graphics Processing Unint to kick in and perform hardware-accelerated page compositing.

    3D comes at a price though... Hardware acceleration can quickly drain your phone's battery, and some fonts won't look as nice when composited in 3D on the iPhone for example, so you better choose them carefully.

  2. Manually fetch HTML pages and cache them in HTML5 local storage to speed up page navigations and enable the app to work offline (as you have all your pages stored locally).

    What HTML5Rocks doesn't describe is how to fetch and cache all the resources referenced by these pages, like CSS style sheets, JavaScript scripts, or images, and that's a lot more work...

  3. Listen to the network online/offline events and detect the connection type (Ethernet, Wifi, 3G, Edge) using the navigator.connection.type property.

    Well, the online/offline events are useful but navigator.connection.type is not supported on the iPhone (only on Android), and you can't rely on it anyway as sometimes an overloaded public Wifi will get slower than 3G... What you really needed is a measure of the quality of the end to end network connection in terms of bandwidth and latency, and you can get that by instrumenting your usage of XMLHttpRequest in your client code as well as your server code.

My 2c... The techniques described in that HTML5Rocks post are fine to get you started, but implementing them is not so simple. As usual, the devil is in the details.

Wednesday, October 5, 2011

Stay Hungry. Stay Foolish.

'Stay Hungry. Stay Foolish.' And I have always wished that for myself. - Steve Jobs


Remembering Steve Jobs

Apple's Board of Directors - We are deeply saddened to announce that Steve Jobs passed away today.

Steve's brilliance, passion and energy were the source of countless innovations that enrich and improve all of our lives. The world is immeasurably better because of Steve.

His greatest love was for his wife, Laurene, and his family. Our hearts go out to them and to all who were touched by his extraordinary gifts.

Snow Report - 8 inches at el. 8200

8 inches of fresh snow at elevation 8200 feet today.

Counting down 49 days to opening day, Nov 23.

1st Snow Storm of the Season!

Wednesday - Winter Warning - 100% chance of snow in Squaw Valley & Alpine Meadows - accumulation 3 to 7 inches possible.

The ski season is just a few weeks away. I'm expecting some of my new ski gear to be delivered on Thursday. Hoping to try it soon!

Winter Warning forecast details below, from my NOAA favorite site.

Tuesday, October 4, 2011

DELETE FROM keynote WHERE name = 'marc'

Lots of entertainment @ OpenWorld 2011 with last minute keynote changes.

$ sqlplus
SQL> CONNECT larry/password@openworld:1521 as sysdba
SQL> DELETE FROM keynote WHERE speaker = 'marc'

Speeding up Web browsing on Mac OS X

A few days ago I showed how to configure a local DNS server on Mac OS X to help test locally hosted mobile apps.

You can also use that local DNS server to drastically improve Web browsing performance.

On complex Web pages DNS domain name lookups can introduce significant latency and slow down your Web browsing experience. A local DNS server cache (with minimum latency as it's local) connected to fast public DNS servers can eliminate that bottleneck and speed up Web browsing.

The good news is that it's really easy to set up.

After following the steps from my previous post, just edit your /etc/named.conf file, and insert the following inside the options { ... } section:
forwarders {;;
}; and are the Google public DNS servers.

That's it. Like magic, you now have a super fast local DNS caching server connected to the Google public DNS servers, which are also much faster than your average Internet Service Provider DNS servers or the Root Internet DNS servers used in the default Mac OS X DNS config.

Happy speedy browsing!

Monday, October 3, 2011

Parallel Programming in Haskell

Trying to relax after an unlucky day which started with a fender-bender... So, to take my mind away from today's unfortunate events I'm hacking some Haskell code tonight and playing around with the Haskell Par Monad.

I've been exploring various aspects of parallel programming for a while now, ranging from low level GPGPU (General-Purpose computation on Graphics Programming Units) using NVIDIA's CUDA and OpenCL to parallel functional programming using Haskell (also tried F#) and Monads like the Par Monad.

To understand the Par Monad, read the Parallel Haskell Tutorial presentation from last week's Commercial Users of Functional Programming conference, and try the tutorial code available on github there.

What fascinates me about the Par Monad is how it makes it so natural to bring typical Monad composition (watch this interview for a great intro to Monad composition) to a parallel program.

One day we'll be able to construct a parallel program as easily as assembling (think composing) Lego blocks (think logic components/blocks and Monads as connectors between them).

That day is not far away...

Saturday, October 1, 2011

Building Web APIs @ JavaOne 2011

I'm not presenting at JavaOne this year. I've been too busy at work (working on cloud integration and mobile) to focus on any conference presentations.

Last year Luciano Resende and I presented how to create composite apps out of components deployed on several cloud platforms using Apache open source software. See there for the slides and details.

This year Luciano and Raymond Feng will be giving a presentation on how to implement Web APIs, which should be really interesting!

Title: Building Flexible APIs for Web 2.x/Cloud Applications (25208)
Time: Monday, 12:30 PM
Location: Hilton San Francisco, Imperial Ballroom B

Using an open and simple API is a great way to build a developer ecosystem and grow a user base dramatically for Web 2.x (social/cloud/mobile) applications. But building flexible APIs can be a challenge. This session covers how to build a nice Web API from scratch, using open source stacks.

... and they've told me that they're going to show a great demo too!

The postings on this site are my own and don’t necessarily represent positions, strategies or opinions of my employer IBM.