Sebastien's blog

We Really Don't Know How To Compute!

2012-01-31T00:55:00.000-08:00

One of my new year resolutions was to blog more. It's not working out yet, as I've been too busy the last few weeks. It's already Jan 31 and this is only my second blog entry this year.

I recently came across a fascinating presentation from Gerald Jay Sussman, co-author of the famous MIT computer science text book 'Structure and Interpretation of Computer Programs' and co-inventor of the Scheme programming language.

He claims that we really don't know how to compute, compares computer programs (constrained to rigid designs and difficult to adjust to a new situation) to living organisms (which can be reconfigured to implement new ways to solve a problem) and makes a convincing argument that we need drastically different programming models to approach that level of flexibility.

He then introduces the Propagator Programming Model (work supported in part by the MIT Mind Machine project). A propagator program is built as a network connecting cells and propagators. Cells collect and accumulate information. Propagators are autonomous machines which continuously examine some cells, perform computations on the information from these cells and add the results to other cells.

A propagator program is analogous to an electrical wiring diagram. To extend it and add a new way to approach a problem, you simply add and connect new propagators. Your cells now collect alternate results from different propagators, and you can then decide to merge redundant results, combine partial results, or even exclude contradictory results when some propagators do not work well in a new situation.

This is similar to how human beings resolve problems. We try several approaches, weigh and combine their results, then wire up our brain with the approaches that work well for the next time we face a similar situation.

I couldn't help but see some relation between that propagator model and my recent interests in computer programming models.

Massively Parallel programming
A propagator program is naturally parallel. Each propagator is continually watching its neighbor cells and computing new results as their values change, autonomously and in parallel with other parts of the program.

Functional programming
A propagator is like a pure function that computes results only from its inputs. A result can also be wrapped in a monad to provide information about its premises, relevance or correctness (useful to pick or combine partial results as they accumulate in a cell for example).

Web Component Assembly
The wiring diagram describing a propagator program seems to map really well to an SCA (Service Component Architecture) component assembly wiring diagram. A propagator could easily be realized as a stateless Web component providing a computation service. A cell could be realized as a Web resource accumulating and storing data.

The propagator model also seems like a great candidate to represent programming expressions as networks of connected components, a subject I researched a bit last year, but which would be too long to describe here... perhaps in another blog post.

Anyway, that got me thinking about a fun weekend project. If I find the time (no guarantee as I'm usually busy on weekends between my regular ski race training and family time), I'd like to do a little hacking and experiment with implementing a propagator program as an assembly of SCA components wired together.

How about defining two new cell and propagator SCA component types, perhaps with REST interfaces to allow propagator programs to live on the Web and play with data from some useful REST services out there?

Wouldn't that be fun?

Happy New Year 2012!

2012-01-04T23:24:00.000-08:00

Happy New Year 2012 from the Delfino Family!

January 2nd, a great day in Squaw Valley: shopping for the girls, free skiing for my son, slalom gate training for me.

Just got back to work today after a long vacation. Still trying to figure out the year 2012 resolutions.

We wish you all a happy new year 2012!

White Friday - Ski Season Opening

2011-11-27T20:10:00.000-08:00

Far away from the Black Friday shopping frenzy, I've just opened the 2011-2012 ski season in Squaw Valley with little snow -- only two runs were open with artificial snow -- but beautiful sunny weather!

Here's a first test report of my new Volkl Racetiger racing skis after two full days on them on Wednesday and Friday:

super quick and responsive in dynamic short SL style turns;
still stable at high speed in GS style wider turns;
pretty stiff, grip on packed snow like no other skis;
the Tyrolia race plates I mounted on them are great at absorbing vibrations.

You better stay focused or these skis will quickly send you off course. Race stock slalom skis are not for relaxed casual cruising, but they're really fun!

After two days of free skiing and drills to practice putting the skis on edge and find the perfect clean carved turn (carving is the modern ski technique that allows you to turn without skidding and losing speed) I'm ready for the next step: practice with slalom gates. The physical preparation I did the last few weeks also paid off as I was able to progressively increase speed and draw arc curves almost all day without getting too tired.

I will be back in Tahoe next Saturday! and really hope to to run slalom gates this time. I want to practice a few more tricks next week:

early turn initiation / early edge engagement (when you tilt the ski edges earlier to engage from one turn to the next);
loading a little more weight on the inside ski to really carve with both skis; that one is not so easy for me as a classic/old-school skier used to put all the weight on the outside ski; I almost got into trouble -- flying off course :) -- when I tried that on Friday afternoon.

I'm also hoping that Squaw will open more runs, to be able to go full speed, away from the snowboarding crowds and obstacles... The first races are coming soon in December!

Sergeant Pepper Spraying UC Davis Students - What a Shame!

2011-11-22T22:36:00.000-08:00

As a proud father of a UC Davis student, I'm out of words... What a shame!

America's Cup World Series in San Diego - Streaming Live

2011-11-18T23:53:00.000-08:00

Don't miss the America's Cup World Series in San Diego this weekend, also streaming live on the YouTube America's Cup channel!

My son was lucky to see them last week as they were already there, and he was in San Diego for the weekend for the Perry Junior Sailing regatta.

Here's a summary of today's semi-finals:

Tomorrow at 1PM PST the finals, Oracle Racing vs the Energy Team from France (Yay!) will be streamed live again.

Apache HTTP Server 2.3.15 released!

2011-11-16T01:00:00.000-08:00

Back to blogging after a break from it as I was too busy the last few days... with some great news!

The Apache Software Foundation and the Apache HTTP Server Project have announced the release of Version 2.3.15-beta of the Apache HTTP Server (also known as "Apache" or "HTTPD"). This version is the 4th, and likely final, beta release before the general availability of the 2.4 release, with lots of interesting new features, described there.

The following will be particularly useful if you're using HTTPD server to run apps in the cloud:

Rate limiting and request timeout control, to protect your server against misbehaving clients;
Improvements to the HTTP proxy, in particular with clustering, load balancing, and failover;
A better multi-threaded processing module (called Event MPM) capable of handling more HTTP connections, typically kept alive and open between requests by Web browsers, with less threads;
Performance improvements, focusing on latency and request / response turnover.

So, with HTTPD 2.3.15 you get a more robust and faster HTTP server with load balancing for horizontal scaling in the cloud! I've been playing with the 2.3.* code for some time, and it has been working really well for me. It'll be great to finally get a 2.4 GA release with all these new cool features! Hopefully soon...

For more info, config examples and performance data, see this presentation from Jim Jagielski.

Integrating Google Page Speed in your Web site build

2011-10-29T17:20:00.000-07:00

Google Page Speed is a great tool to help analyze and optimize the performance of your Web pages. It's actually a family of tools:

- Page Speed Chrome and Firefox browser extensions complement Firebug and analyze the performance of your Web pages right from your Web browser.

- Page Speed Online analyzes the performance of your Web pages too without requiring a browser extension.

- Mod_pagespeed is an Apache HTTPD extension module that optimizes, rewrites and minifies your Web pages on the fly.

- The Page Speed Service is a proxy hosted by Google which optimizes and delivers your Web pages from Google's servers.

- The Page Speed SDK allows you to embed the Page Speed analyzers and optimizers in your own project.

If you're paranoid (don't want your Web site to depend on Google's Page Speed servers) and CPU + memory conscious (don't want to spend resources / $$$ running mod_pagespeed on your HTTP server) you can also run Page Speed on your pages ahead of time when you build your Web site.

It's pretty simple. Here's how I'm doing it on Linux using GNU Automake and Apache HTTPD (I'm sure that can be adapted to other environments):

1. Download and build the Page Speed SDK, like this:
build=`pwd`
curl -OL https://dl-ssl.google.com/page-speed/sdk/current/page-speed-sdk.zip
unzip page-speed-sdk.zip
cd page-speed-1.9
make builddir=$build/page-speed-1.9-bin

That'll build two useful command line tools: minify_html_bin, an HTML rewriter / minifier which also minifies inline CSS and Javascript, and jsmin_bin, a Javascript minifier which also works well for CSS.

2. Write the following in your Automake Makefile.am file:
minified = htdocs/index-min.html htdocs/foo-min.js htdocs/bar-min.css

sampledir = ${prefix}
nobase_dist_sample_DATA = ${minified}

SUFFIXES = -min.html -min.js -min.css
.html-min.html:
    ../page-speed-1.9-bin/minify_html_bin $< $@

.js-min.js:
    ../page-speed-1.9-bin/jsmin_min < $< > $@

.css-min.css:
    ../page-speed-1.9-bin/jsmin_min < $< > $@

CLEANFILES = ${minified}

I won't go into the Automake specifics now but these rules will run the Page Speed minifiers on your Javascript, CSS and HTML pages as part of your Make build. I'm using a simple naming convention for the minified files, index-min.html is the minified version of index.html.

3. Add a DirectoryIndex directive to your Apache HTTPD httpd.conf config file:
DirectoryIndex index-min.html index.html

That tells HTTPD to serve /index-min.html (the minified page) instead of the original /index.html page when a user points his browser to http://yourwebsite.com/.

4. Reference foo-min.js and bar-min.css in your HTML pages instead of foo.js or bar.css for example.

5. After making changes to your Web pages, build your site like this:
make

If you had /index.html, foo.js, and bar.css, you should now have /index-min.html, foo-min.js, bar-min.css, all nicely optimized, rewritten and minified by Page Speed.

To clean up the minified files, run:
make clean

That's it. That little trick should normally save you from 30% to 50% bandwidth, CPU, memory and disk space on the client devices that access your site (particularly useful with resource-constrained mobile devices) and on your HTTP server too (which now serves smaller files).

Hope this helps

Some Assembly Required

2011-10-27T07:56:00.000-07:00

Finally got my new ski racing gear assembled last weekend. I had to try several ski shops in the Bay Area before finding one that knew how to mount Tyrolia race plates and bindings on stock race skis.

Here's the end result:

By the way, it looks like picking Tyrolia over Marker (which are usually paired with the Volkl skis) was the right thing to do, as word on the street is that a number of regional teams have moved off to Look, Tyrolia and others after problems with the Markers. Marker even had to issue a safety recall on their race Comp 12 bindings earlier this year.

So, I'm getting ready for the season. The gear is assembled, I've completed my registration with NASTAR (NAtional STAndard Race program), and I'm progressively increasing the physical preparation. Now we just need some snow!

Recursive Functions of Symbolic Expressions and their Computation by Machine

2011-10-25T02:34:00.000-07:00

'Recursive Functions of Symbolic Expressions and their Computation by Machine' is the original paper on Lisp from John Mc Carthy. That paper appeared in Communications of the ACM in April 1960. It is a great short read.

John Mc Carthy passed away yesterday. After Steve Jobs and Dennis Ritchie earlier this month, October has been a sad month for the computing community...

Another good article from Paul Graham, 'The Roots of Lisp' helps explain John Mc Carthy's discoveries with Lisp.

If you don't know Lisp, you should learn it.

I had a brief look at it 15 years ago, but was too impatient and unexperimented to get the point then. I took more time to actually learn it on a summer vacation 2 years ago, then went on to learn Scheme (a Lisp dialect) as I was going through the Structure and Interpretation of Computer Programs classic Computer Science book from MIT.

Believe me. Lisp (or Scheme) will transform you into a different programmer. After that, even if you still have to program in C, C++, Python, or even Java, you're in a different world. You'll see programming through different eyes, and there will be no going back.

Think about a few fun facts... Pretty much everything you manipulate in Lisp is a list of values (Lisp stands for 'list processing'). Values can be numbers, strings, symbols or lists. Data and code are interchangeable and represented in the same way, as lists of values. You can write a Lisp interpreter in half a page of Lisp!

So, if you're working with 'modern' OO programming languages, scripting languages, or functional programming languages, if you're using XML dialects, JSON, DSLs, and you're still struggling to map between documents, data, objects, services, code and configuration... spend the time to learn Lisp. It'll open your eyes.

Caching Images in a Mobile Web app

2011-10-20T23:20:00.000-07:00

Another post on a technique to help make mobile Web apps work offline.

Last time I showed how to use XMLHttpRequest and local storage to download and cache Javascript scripts. Here I'm going to show how to download and cache an image, and then inject it dynamically into your HTML page.

That technique can be particularly useful to optimize an image intensive app (a game or a photo gallery app for example) and allow it to work offline.

First place base64-encoded (as defined in Internet RFC 3548) versions of your images on your Web server. On Mac OS X for example, convert foo.png to a base64-encoded foo.b64 file like this:
$ base64 -w 0 foo.png >foo.b64

Configure your Web server to serve .b64 files as text/plain. If you're using the Apache httpd 2.x Web server -- and if you're not using it, you should :) -- add the following to your httpd.conf server configuration file:
text/plain b64

The reason why we want to convert our images to base64-encoded is that it'll make it much easier to use them in the HTML5 app pages. We're getting to that part now.

Place the following Javascript script in your HTML page, under the <head> element for example:
<script type="text/javascript">
window.appcache = {};

// Get a resource
window.appcache.get = function(url) {
  var doc = localStorage.getItem(url);
  if (doc != null)
    return doc;
  var http = new XMLHttpRequest();
  http.open("GET", url, false);
  http.send(null);
  if (http.status != 200)
    return null;
  localStorage.setItem(url, http.responseText);
  return http.responseText;
}

// Convert an image to a data: URL
window.appcache.img = function(url) {
  var b64 = window.appcache.get(url);
  return 'data:image/png;base64,' + b64;
}
</script>

I'm not showing the error handling code to keep it short, but you get the picture. The get(url) function downloads a resource and caches it in local storage. The img(url) function gets an image resource and converts it to a data: URL. A data: URL (as defined in Internet RFC 2397) allows you to include resource data (here our base64-encoded image data) immediately in the URL itself.

Now, later in your HTML page, say you have an <img/> element like this:
<img id="foo"/>

You can set the cached image into it dynamically, like this:
<script type="text/javascript">
var foo = document.getElementById('foo');
foo.src = window.appcache.img('/foo.b64');
</script>

The image src property will recognize the data: URL, the image/png;base64 media type at the beginning of the URL, and read the base64-encoded content as a PNG image.

That's it. With the little trick I've described here you should now be able to:

control the download of your images;
cache them in local storage for speed and working offline;
inject them dynamically into your HTML page as needed.

Hope this helps. In the next few days I'll show how to handle CSS style sheets in a similar way.

Game Industry Forecast - Rapid Growth and a Changing Market

2011-10-16T13:57:00.000-07:00

Interesting study and forecast on the game industry there, as Zynga is preparing for IPO and announced 10 new products this week.

The global game industry will generate $60 billions in revenue for 2011. It's not a surprise, but gaming is the only media business growing right now with a rapid growth driven by mobile games, as games are the leading apps on smartphones and tablets.

The game market seems to be changing rapidly from hardcore gamers (mostly teenage boys) to casual gamers with older households (35+yr old), and women (42% of the gamers now), as browsers and mobile are bringing new populations to casual and social games.

The game technical platforms are also shifting quickly to the Web browser. With technologies like HTML5, Canvas, SVG, and WebGL for example, Web browsers have become the most convenient platform for games, and it's now clear that games in browsers are the future. See this interview of the founder of Electronic Arts for more insight and his thoughts on what's happening.

Clearly, that means challenges ahead for the established game companies which have to adapt quickly. That also means great opportunities for new players to come and disrupt the mobile game market!!

PhoneGap / Apache Callback project accepted in the Apache Incubator

2011-10-14T22:29:00.000-07:00

The vote to accept the Apache Callback project in the Apache Software Foundation Incubator was open for the last 72 hours, and that vote just passed earlier today.

Apache Callback is the free open source software evolution of the popular PhoneGap project.

Apache Callback is a platform for building native mobile applications using HTML, CSS and JavaScript. Apache Callback allows Web developers to natively target Apple iOS, Google Android, RIM BlackBerry, Microsoft Windows Phone 7, HP webOS, Nokia Symbian and Samsung Bada with a single codebase. The Callback APIs are based on open Web standards and enable access to native device capabilities, including the device camera, compass, accelerometer, microphone or address book for example, from HTML5 apps.

That's really great news for the mobile app developer community! ... and I'll be watching this project very closely in the next few months!

Dennis Ritchie: The Shoulders Steve Jobs Stood On

2011-10-14T01:06:00.000-07:00

Dennis Ritchie: The Shoulders Steve Jobs Stood On.

Another great loss. Dennis Ritchie was the father of the C Programming Language and the UNIX operating system, which pretty much the entire Internet and computer industry run on.

I'm writing this on a Macbook Air running Mac OS X, a UNIX-based system, and programs mostly written in C.

His book "The C Programming Language" was my first Computer Science book. My first serious encounter with software was a 2-week job to teach the C language to a team of programmers. I was just trying to make enough money to go on vacation, but then I got hooked and decided that I wanted to be a C coder... Feeling sad today.

#include <stdio.h>

main() {
printf("goodbye, world\n");
}

Thank You Sir. RIP.

Caching Javascript in a Mobile Web app

2011-10-14T00:00:00.000-07:00

Earlier this week I promised I'd post a useful technique to reference, pre-fetch, and cache Javascripts, as a follow up to my review of an article on HTMLRocks describing HTML5 techniques for optimizing mobile Web performance.

So, here it goes.

The HTML5Rocks article showed how to follow all the links in a page and pre-fetch and cache their target pages in local storage. That was a good starting point, but for a mobile Web app to really work well offline you also need to fetch and cache the referenced JavaScripts, CSS, and images.

Here's how I do that in my apps.

I place the following utility Javascript at the top of my page under the <head> element:
<script type="text/javascript";gt;
window.appcache = {};

// Get a resource
window.appcache.get = function(url) {
  var doc = localStorage.getItem(url);
  if (doc != null)
    return doc;
  var http = new XMLHttpRequest();
  http.open("GET", url, false);
  http.send(null);
  if (http.status != 200)
    return null;
  localStorage.setItem(url, http.responseText);
  return http.responseText;
}

// Load a script
window.appcache.script = function(url, parent) {
  var e = document.createElement('script');
  e.type = 'text/javascript';
  e.text = window.appcache.get(url);
  parent.appendChild(e);
}
</script>

I'm not showing the error handling code here to keep it short, but you get the picture. The get(url) function downloads a resource and caches it, and the script(url, parent) function gets a resource and creates a script element with it.

Then to reference a Javascript script later in my page, instead of writing the usual:
<script type="text/javascript" src="foo.js"></script>

I write this:
<script type="text/javascript">
window.appcache.script('foo.js', document.head);
</script>

If you're loading pages in an <iframe> nested inside a main page as suggested in the HTML5Rocks article, you don't need to repeat the utility script in all your pages. Just have it once in the main page, then in the nested pages just write:
<script type="text/javascript">
window.parent.appcache.script('foo.js', document.head);
</script>

Or just set the appcache property on your <iframe> like this:
var nested = document.createElement('iframe');
nested.contentWindow.appcache = window.appcache;

Then you can write the same code everywhere:
<script type="text/javascript">
window.appcache.script('foo.js', document.head);
</script>

Hope this helps.

To the folks who've pinged me several times this week asking for this tip: Sorry for the delay but I've been kinda busy... I'll show how to fetch and cache CSS (similar to Javascripts) and images (using a data: url and base64-encoded image content) in the next few posts, probably this weekend.

New Racing Ski Gear Arrived!

2011-10-10T00:46:00.000-07:00

Got my new gear for the upcoming ski season!

I started last year's season cautiously as I was getting back into skiing after a break of a few years, and went crescendo as I was gaining back confidence, faster, steeper -- I do extreme skiing / free riding -- with fun cornice and cliff drops and high speed runs in any snow conditions.

I ski in Squaw Valley (which BTW recently merged with Alpine Meadows). For folks familiar with Squaw, my favorite playgrounds are Palisades, Silverado, KT22, 75 Chute, and the Cornice area on the other side of KT.

Reflecting on the past season, pushing limits further with extreme skiing would probably be dangerous so to satisfy my taste for adrenaline and speed, this season I've decided to get back into Slalom racing. I ran lots of Slalom gates as a kid and was pretty good back then. Like biking, that's something you don't forget, so we'll see... I'm going to train and run gates first, see how it goes, try the Squaw and Alpine Meadows Masters and then if things go well I'd like to try some of the Tahoe region local races.

That's what I had in mind when I selected my new hot ski gear:

Volkl Racetiger SL Skis 'Race Stock' version, stiffer and faster but less forgiving than the normal consumer version. It's one of the two or three best Slalom skis on the market, although a little difficult to find as consumer type ski shops don't carry it. I got a really good deal on last year's model.

Tyrolia FreeFlex Pro 11.0 Bindings. Volkl skis are normally paired with Marker bindings but IMO the Tyrolia are better quality for the same price, plus I couldn't find a good deal on the Marker Comp 12.0. There is a silly macho trend now where most people want bindings with high DIN ratings (that's the tension of the bindings) like 16.0 or even 18.0... I think that's crazy unless you're 200+lbs, and the best way to break a leg. So, I'm going with a DIN 11.0 as I'm lightweight and have already broken enough bones for now... It's dubbed a Junior binding, probably not high enough DIN for the macho types but I don't care as long as I arrive first down the gates :) and with no broken bones.

Tyrolia RDX Race Plate. Goes under the bindings to adjust the flex of the ski and give you a 14mm lift to make it easier to go on edge (with a longer lever arm) and avoid that the boots hit the snow when on edge.

That gear is going to be great for Slalom and carving hard or groomed snow, good in moguls, OK in fresh snow, not so great in powder. These Slalom skis are really skinny compared to the wide fat skis people use for powder now, but I don't care. I used to ski powder with very skinny skis when I was a kid, skied all types of snow with pretty bad skis when I was skiing in the army in France, so now I guess I can ski powder with pretty much anything, any two pieces of wood will do...

The skis have arrived, waiting for the bindings and the plate later this week...

Countdown to season opening: 44 days! In the meantime I'm training for cliff drops from my backyard fence -- just kidding :)

Offline Mobile Web Apps - Really...

2011-10-09T22:17:00.000-07:00

A few days ago I posted a review of an article on HTMLRocks describing HTML5 techniques for optimizing mobile Web performance, in particular a technique to pre-fetch and cache HTML pages and resources, useful to improve navigation performance and allow the mobile Web app to work offline.

In that review I said:
"What HTML5Rocks doesn't describe is how to fetch and cache all the resources referenced by these pages, like CSS style sheets, JavaScript scripts, or images, and that's a lot more work..."

and concluded:
"My 2c... The techniques described in that HTML5Rocks post are fine to get you started, but implementing them is not so simple. As usual, the devil is in the details."

Since then a number of folks have pinged me to challenge and ask:
"So, how do you fetch the referenced resources? and what are these details?"

Fair question :) I'm going to address it in a series of posts in the next few days. I'll also describe a few additional issues and tricks to make a mobile Web app really work offline, and some solutions with code snippets.

Here's a draft outline of the next few posts:

Referencing, fetching and caching Javascript scripts
Caching CSS stylesheets
Caching images
HTML5 W3C application cache vs a DIY cache
Monitoring your network connection

Stay tuned...

Google Summer of Code 2011 T-shirt - Thanks Google

2011-10-07T17:46:00.000-07:00

Received my Google Summer of Code 2011 T-shirt gift from Google today, as I was a mentor for GSoC @ Apache this year again.

Hoping my students have received theirs too (that may take a little longer as they're in Sri-Lanka).

I hope you guys had a lot fun with your coding projects this summer. I enjoyed working with you! Keep up the good work, and look for opportunities to work on and learn from Open Source again!

Silicon Valley Code Camp 2011 - By Developers, for Developers

2011-10-07T09:09:00.000-07:00

Silicon Valley Code Camp 2011 is this weekend in Los Altos Hills.

The camp is free to attend, run by volunteers, with lots of great speakers and interesting sessions. It's also a great opportunity to network with fellow developers.

Here's my session selection:

Saturday
HTML5 Game Programming - WebGL Edition
Build a mobile web app with Sencha Touch
Faster Mobile Anyone?
Kids Programming Workshop with Scratch
Hands on jQuery Mobile

Sunday
AutoMobile: the Next Hot Platform for Mobile Linux
Create a Kinect Powered Personal Robot with Microsoft Robotics Developer Studio
The Ecosystem of Context (exploiting user context to go beyond search)
GPU Accelerated Databases using OpenCL

HTML5 rocks the mobile Web

2011-10-06T23:53:00.000-07:00

Great post about HTML5 techniques for mobile on 'HTML5Rocks'. Not sure I agree with everything it says though.

It describes three HTML5 techniques for optimizing mobile Web performance:

For smooth native-feeling sliding and flipping page transitions, use a translate3d(0,0,0) CSS transform to force the phone's Graphics Processing Unint to kick in and perform hardware-accelerated page compositing.

3D comes at a price though... Hardware acceleration can quickly drain your phone's battery, and some fonts won't look as nice when composited in 3D on the iPhone for example, so you better choose them carefully.

Manually fetch HTML pages and cache them in HTML5 local storage to speed up page navigations and enable the app to work offline (as you have all your pages stored locally).

What HTML5Rocks doesn't describe is how to fetch and cache all the resources referenced by these pages, like CSS style sheets, JavaScript scripts, or images, and that's a lot more work...

Listen to the network online/offline events and detect the connection type (Ethernet, Wifi, 3G, Edge) using the navigator.connection.type property.

Well, the online/offline events are useful but navigator.connection.type is not supported on the iPhone (only on Android), and you can't rely on it anyway as sometimes an overloaded public Wifi will get slower than 3G... What you really needed is a measure of the quality of the end to end network connection in terms of bandwidth and latency, and you can get that by instrumenting your usage of XMLHttpRequest in your client code as well as your server code.

My 2c... The techniques described in that HTML5Rocks post are fine to get you started, but implementing them is not so simple. As usual, the devil is in the details.

Stay Hungry. Stay Foolish.

2011-10-05T23:32:00.000-07:00

'Stay Hungry. Stay Foolish.' And I have always wished that for myself. - Steve Jobs

Inspiring.

Remembering Steve Jobs

2011-10-05T17:49:00.000-07:00

Apple's Board of Directors - We are deeply saddened to announce that Steve Jobs passed away today.

Steve's brilliance, passion and energy were the source of countless innovations that enrich and improve all of our lives. The world is immeasurably better because of Steve.

His greatest love was for his wife, Laurene, and his family. Our hearts go out to them and to all who were touched by his extraordinary gifts.

Snow Report - 8 inches at el. 8200

2011-10-05T16:34:00.000-07:00

8 inches of fresh snow at elevation 8200 feet today.

Counting down 49 days to opening day, Nov 23.

1st Snow Storm of the Season!

2011-10-05T00:12:00.000-07:00

Wednesday - Winter Warning - 100% chance of snow in Squaw Valley & Alpine Meadows - accumulation 3 to 7 inches possible.

The ski season is just a few weeks away. I'm expecting some of my new ski gear to be delivered on Thursday. Hoping to try it soon!

Winter Warning forecast details below, from my NOAA favorite site.

DELETE FROM keynote WHERE name = 'marc'

2011-10-04T22:51:00.000-07:00

Lots of entertainment @ OpenWorld 2011 with last minute keynote changes.

$ sqlplus
SQL> CONNECT larry/password@openworld:1521 as sysdba
SQL> DELETE FROM keynote WHERE speaker = 'marc'
SQL> EXIT
$

Speeding up Web browsing on Mac OS X

2011-10-04T20:53:00.000-07:00

A few days ago I showed how to configure a local DNS server on Mac OS X to help test locally hosted mobile apps.

You can also use that local DNS server to drastically improve Web browsing performance.

On complex Web pages DNS domain name lookups can introduce significant latency and slow down your Web browsing experience. A local DNS server cache (with minimum latency as it's local) connected to fast public DNS servers can eliminate that bottleneck and speed up Web browsing.

The good news is that it's really easy to set up.

After following the steps from my previous post, just edit your /etc/named.conf file, and insert the following inside the options { ... } section:
forwarders {
8.8.8.8;
8.8.4.4;
};

8.8.8.8 and 8.8.4.4 are the Google public DNS servers.

That's it. Like magic, you now have a super fast local DNS caching server connected to the Google public DNS servers, which are also much faster than your average Internet Service Provider DNS servers or the Root Internet DNS servers used in the default Mac OS X DNS config.

Happy speedy browsing!

Parallel Programming in Haskell

2011-10-03T01:14:00.000-07:00

Trying to relax after an unlucky day which started with a fender-bender... So, to take my mind away from today's unfortunate events I'm hacking some Haskell code tonight and playing around with the Haskell Par Monad.

I've been exploring various aspects of parallel programming for a while now, ranging from low level GPGPU (General-Purpose computation on Graphics Programming Units) using NVIDIA's CUDA and OpenCL to parallel functional programming using Haskell (also tried F#) and Monads like the Par Monad.

To understand the Par Monad, read the Parallel Haskell Tutorial presentation from last week's Commercial Users of Functional Programming conference, and try the tutorial code available on github there.

What fascinates me about the Par Monad is how it makes it so natural to bring typical Monad composition (watch this interview for a great intro to Monad composition) to a parallel program.

One day we'll be able to construct a parallel program as easily as assembling (think composing) Lego blocks (think logic components/blocks and Monads as connectors between them).

That day is not far away...

Building Web APIs @ JavaOne 2011

2011-10-01T19:41:00.000-07:00

I'm not presenting at JavaOne this year. I've been too busy at work (working on cloud integration and mobile) to focus on any conference presentations.

Last year Luciano Resende and I presented how to create composite apps out of components deployed on several cloud platforms using Apache open source software. See there for the slides and details.

This year Luciano and Raymond Feng will be giving a presentation on how to implement Web APIs, which should be really interesting!

Title: Building Flexible APIs for Web 2.x/Cloud Applications (25208)
Time: Monday, 12:30 PM
Location: Hilton San Francisco, Imperial Ballroom B

Abstract:
Using an open and simple API is a great way to build a developer ecosystem and grow a user base dramatically for Web 2.x (social/cloud/mobile) applications. But building flexible APIs can be a challenge. This session covers how to build a nice Web API from scratch, using open source stacks.

... and they've told me that they're going to show a great demo too!

Squaw Valley and Alpine Meadows just merged!

2011-09-29T01:37:00.000-07:00

It's going to be the best ski season ever, as Squaw Valley and Alpine Meadows have just announced a merger to offer more than 6000 skiable acres! More details on the announcement here, and there.

Those who know me well already know that I'm a crazy skier and that I go to Squaw almost every weekend, or on big storm powder days... usually just for the day. Yeah, that's a long day when you leave at 4:45am to avoid traffic and people stuck on I80, arrive in Squaw around 8:15am, ski all day, then drive back :)

The other good news is that Squaw season passes purchased before the announcement -- got mine long time ago from my wife and kids for father's day, Can't thank you enough for that! -- will automatically work in Alpine!

So I'll be able to ski Alpine's secret powder stashes (shhhh, they're secret) too this season! Last season broke records with 810 inches of snow in Squaw. I'm hoping for La Nina to bring a ton of powder again this winter.

I also hope that Squaw will open my favorite playgrounds more often and earlier in the morning... Please :)

The Palisades

The Palisades | Squaw Valley from UnofficialNetworks.com on Vimeo.

Silverado

Silverado Tram Face | Best Crashes

A DNS server to test your mobile apps

2011-09-26T00:02:00.000-07:00

Why would you want to set up an Internet Domain Name System server (DNS) on your laptop? Well, it can be handy to serve test domain names to your iPhone and test the mobile apps you're working on, or to use HTTPS on an insecure network with an TLS/SSL certificate you've created for your test domains.

DNS has a reputation to be tricky to set up, so I thought I'd share the steps to do it on Mac OS X. It's easy, when you know what to do.

Say you want to map the supercoolapp.com domain (an example test domain) and all the *.supercoolapp.com sub-domains to IP address 10.1.1.55 for example.

1) In a terminal window run the following commands:
sudo rndc-confgen -a -c /etc/rndc.key
to generate the rndc.key file as it's included by /etc/named.conf.
Named is the domain name server, rndc is the Mac OS name server control utility.

sudo launchctl load -w /System/Library/LaunchDaemons/org.isc.named.plist
to enable and start named.

2) Add the following to the /etc/named.conf file:

zone "supercoolapp.com" {
  type master;
  file "supercoolapp.com.zone";
};

3) Create a /var/named/supercoolapp.com.zone file, containing this:

supercoolapp.com. 7200 IN SOA supercoolapp.com. root.supercoolapp.com. (
3          ; Serial
15         ; Refresh
3600       ; Retry
3000000    ; Expire     86400 )    ; TTL
IN NS supercoolapp.com.
IN MX 10 supercoolapp.com.
IN A 10.1.1.55
*.supercoolapp.com. IN A 10.1.1.55

4) Open your Mac OS System preferences, go to Network / Advanced / Dns and add 127.0.0.1 to the top of the DNS servers list.

5) To test your new DNS settings try this:

dig my.supercoolapp.com
ping my.supercoolapp.com

6) On your iPhone go to Settings / your wifi network / DHCP and change the DNS field to your Mac OS system's IP address.

7) That's it! Now open http://my.supercoolapp.com in Safari on your iPhone. You should hit your mobile app running on 10.1.1.55.

Hope this helps!

Google Summer of Code 2011 - My students passed!

2011-08-27T14:48:00.000-07:00

I've been mentoring two students in the Google Summer of Code program this year.

Nirmal's GSoC project was to develop a tool to generate SCA composite diagrams using SVG.

Eranda's project was to develop SCA datastore components using Apache Cassandra, CouchDB and Hadoop.

Both students just passed the final GSoC evaluations! They should be proud of what they've achieved this summer!

I hope we helped them learn a few things, and that they'll continue to be involved with open source in the future.

The Gang of Four would be Proud

2011-07-26T22:10:00.000-07:00

Just came across this on reddit, which gave me a good laugh.

What is an AbstractSingletonProxyFactoryBean? I guess it gets crystal clear after reading this javadoc :)

I already knew that for some time, and I'm not blaming Spring in particular as they're better than most, but SomethingIsDefinitelyWrongInTheJavaDevelopmentCommunity...

I want a NYC app that...

2011-06-27T20:45:00.000-07:00

Great initiative from New York City with NYCBigApps!

New York City is challenging the public to share what apps they want for residents, businesses, and visitors. After 8 days running the challenge there's already more than 300 app ideas! It's amazing to see all the app ideas that people come up with.

Imagine if creating these apps was as easy as blogging about them :)

Well, it shouldn't be too hard actually, with all the nice Web APIs that are available out there now for search, maps, location, photos, shopping etc.

Let's look at the very first app idea submitted to NYCBigApps:
I want a NYC app that shows me all music instrument stores in NYC.

That one is actually pretty easy:

Get your location, say NYC.
Do a Google map search for music stores around that location.
Size the map to 320 pixels to fit nicely on your smartphone screen.

After about 5 mins fiddling with this, here we go! Looks like a simple app, just a map with markers, not very fancy, but does the job.

View Larger Map

Wouldn't it be great if we could make it so easy to create mini apps that the public could finally unleash their creativity, take control and make their own apps in minutes as ideas come up?

With HTML5 becoming easier to use and more mobile friendly, we're getting closer to that dream every day!

Meritocracy in Action: The Apache Membership Process

2011-06-08T08:58:00.000-07:00

The Apache Software Foundation (ASF) will be holding its annual Members' meeting this July. Among the Foundation's business that takes place during this meeting is the election of new ASF Members.

ASF committers who demonstrate merit in the Foundation’s growth, evolution, and progress are nominated for ASF Membership by existing members and elected bi-annually.

The ASF just published the list of new Members elected at the January 2011 Members' meeting on its blog. That was 6 months ago but it's never too late to blog about it :).

I just discovered that blog today as people started to send me emails when they saw my name there...

It is a great honor for me to be in that list as a new -- well... 6 month old :) -- elected ASF Member!

For more info, see the complete list of ASF members, committers and how the ASF works.

Comparing the performance of C++, Java, Scala and Go

2011-06-03T10:12:00.000-07:00

An interesting paper from Google comparing the performance of C++, Java, Scala and Go was presented at the 2011 Scala days.

Conclusion from the paper: "We find that in regards to performance, C++ wins out by a large margin. However, it also required the most extensive tuning efforts, many of which were done at a level of sophistication that would not be available to the average programmer."

And my 2c: If you care about performance, for example if you run in the cloud and memory footprint and CPU directly translate to dollars on your cloud monthly bill, if you like programming and are not afraid of spending energy to design and program efficient algorithms, then C++ is still your best bet.

Sure that requires a bit more thinking and perhaps more sophisticated programmers, but you get what you pay for.

And to all the folks who criticize C++ for being complicated, well... in my opinion it's as complicated as you want it to be. It's really up to you as a programmer to think and keep your code simple and, by the way, there's a lot of complicated Java and Scala code out there too :)

Mod-security now available under Apache License v2.0

2011-05-26T22:39:00.000-07:00

Great news! ModSecurity v2.6.0 is now available and that's the first release of this great Web firewall module for the Apache HTTP server available under the Apache License v2.0.

This move from the viral GPL License v2 to the Apache License should help further adoption and community development of ModSecurity.

The OWASP ModSecurity core rule set v2.2.0, a set of protection rules against most known Web attacks was also released today under the Apache License v2.0 as well.

More details on the ModSecurity blog.

Coding in the Cloud... Linux Shell in a Web Browser

2011-01-25T00:13:00.000-08:00

The last few weeks I've been exploring how to use an iPad, tablet or netbook to write programs in the cloud with just a Web browser connected to my Linux servers.

I've played with the recent Web based source code editors from Google (Code Mirror), Mozilla (SkyWriter), and Eclipse (Orion). They're OK, but a source code editor is just not enough for me...

I also need a browser-based Linux shell window. There's some good ones out there. I've tried Ajaxterm (a little slow and not all keys worked), Anyterm (keyboard issues as well), and finally settled for Shellinabox.

Shellinabox gets pretty close to a real xterm. I've been using it for two days for various tasks including coding (with vi). I like it a lot! and am starting to think that vi in a shell in shellinabox compares favorably to these new fancy Web based editors...

I've patched it to allow a shell to set the browser title using a VT escape sequence in an environment variable (PS1='\\033]0; [\W]\007\\w \$ ' for example), like people usually do with xterms.

The patch is there on my Github account. I find it useful to help distinguish Shellinabox browser windows and tabs and see what directory they're in.

If I find time, I'll post the steps I'm using to configure Shellinabox, build it and connect to it through a secure HTTPD reverse proxy... later this week.

CLang completes support for C++0x variadic templates

2011-01-20T00:29:00.000-08:00

The CLang project has just completed support for C++0x variadic templates (see [1], [2], [3]).

Pretty good news! CLang now compares well with GCC.

The CLang and underlying LLVM projects use liberal BSD, NCSA and MIT licenses compatible with the Apache 2.0 license (see [4]). That's better than GCC, GPL licensed and usable to build an Apache project but not embeddable in the project itself.

I've been thinking about using CLang in the Apache Tuscany C++ project for a while.

I'd like to use its API to convert SCA composites (XML docs describing SCA component assemblies) into C++0x templates and compile them into machine code on the fly, instead of having to interpret all the XML gorp at run-time.

Think of it like an 'SCA assembly JIT compiler'...

Benefits: ultimate speed, CPU and space, cloud resources (and resulting monthly bill) optimizations, and less XML parsing and interpretation code to worry about.

Requirement: support for C++0x variadic templates...

Patience may pay off after all, as it is finally there.

I gotta try that stuff this weekend!

[1] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2006/n2080.pdf
[2] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2242.pdf
[3] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2488.pdf
[4] http://www.apache.org/legal/3party.html

Coding in the Cloud... Feels good like back to the 80's

2011-01-18T21:30:00.000-08:00

I started programming on a VT100 terminal in 1980, with:

a white/black screen / 24 x 80 characters
no disk, data on a server
vi
assembly language + C on Unix
a few programming books
some deep thinking

I loved it!

Then, after a few detours through:

a dozen or so programming languages
fancy OSes on hi-res overheating laptops
'integrated' development environments
'code assist' (enough said)
no thinking (replaced by 'IDE' and 'code assist')

Recently I've gone back to a more productive environment:

24 x 80 characters
vi
a few programming books
Web search
some deep thinking

I can think again!

But still missing that part:

no disk, data on a server, always on, accessible from anywhere

Then something cool happened today:

The Google Project Hosting team just announced the ability to edit programs directly in the Web browser using an online editor powered by CodeMirror. With Mozilla's SkyWriter, and Eclipse's Orion, that's one more way to edit code 'in the cloud' from a connected device.

That works well for me, as most of the programs I write these days run on servers 'in the cloud' anyway (say, big servers on a network instead of my laptop).

I've also been looking at the iPad as a programmer terminal and wondering about the upcoming iPad-2.

So, here's my future programming environment:

White/black screen / 24 x 80 characters
How about an iPad-2, Google CR-48 or similar netbook and a single Web browser window with big fonts?

I haven't been lucky enough to receive a CR-48, so I'm thinking about buying an iPad-2 out and hacking a programmer-friendly keyboard app for it.
No disk, data on a server
Anybody say Cloud?
VI-like
CodeMirror, SkyWriter, a hosted Orion server, or I'll just hack up a vi-like editor on top of one of these projects.
Assembly language + C Well, I have to use Java at work, but on my spare time I like to code some C, C++, Scheme, Lisp, Python or F#. Feels better. I haven't written assembly language for a while, but now that I think about it...
A few good programming books
Receiving the new TAOCP-4a book tomorrow, and will probably spend a few nights reading it.
Web search
Allow for a second browser window.
Some deep thinking
Still thinking...

I already love it!

Hmmm, the new iPad-2's rumored fancy screen should be enough for some 24 x 80 coding, right?

Is IPad-2 the new VT100 terminal? :)

Accelerating Innovation with Apache Extras

2010-12-15T22:32:00.000-08:00

Google and the Apache Software Foundation just launched Apache Extras.

Apache Extras is a home for communities and open source projects related to Apache technologies, which cannot be hosted by the Apache Software Foundation because they are unable or unwilling to use the Apache v2 license or the Apache collaborative consensus based processes (a.k.a. The Apache Way).

This new channel will help build collaborative communities around Apache Open Source related technologies. Apache Extras provides the infrastructure services typically required by open source projects, such as code repositories, bug tracking, and project web sites/wiki. It is hosted by Google Code Project Hosting, so it will be very familiar to developers already using Google Code Project Hosting.

The announce is here, more on the Google code blog, my friend Luciano's blog, the Apache Software Foundation blog, the Apache Extras management home, the FAQ, guidelines and home page.

Ski Season Opening!

2010-11-24T22:28:00.000-08:00

Tahoe got a lot of snow in the last few days and the ski resorts are now open.

I already went to Squaw Valley on Saturday for the opening day. Only one lift was open but it was enough for my daugther and I to have a great time skiing there. The drive back was a little long as we had a big snow storm but it was worth it.

Going back to Squaw Valley again tomorrow morning with my kids and a friend and hoping for a great day as the snow accumulation is already about 9ft at 8200ft!

and the weather forecast looks really good too!

Schedule:
4:30am leaving San Carlos... ouch that's early :)
5:00am pick up a friend in San Francisco
6:30am pick up my daugther in Davis
8:30am breakfast in Squaw Valley
9:00am on the slopes!

JSON vs XML, structured vs semi-structured data

2010-11-17T22:55:00.000-08:00

That debate has been going for years now... When should you use JSON vs XML?

Interesting blog posts on this today from Norm Walsh and Noah Mendelsohn.

Both agree that:
"XML wasn't designed to solve the problem of transmitting structured bundles of atomic values. XML was designed to solve the problem of unstructured data. In a word or two: mixed content."

Noah then adds:
"If you want to send along a list of job applicants and their recent salaries, JSON does fine; if you want to send their resumes, well JSON isn’t quite as helpful. A surprising amount of the world’s important information is in just such semi-structured documents. Think insurance policies, shop manuals, and even Web pages themselves."

+1 to that! I've worked -- and struggled sometimes :) -- with XML for years, and more recently with JSON, and couldn't agree more!

Adding my 2c:
When you work with structured business data, use an object (or struct, or record) model in your favorite programming language, and an object serialization encoding like JSON.

When you work with semi-structured document resources, use a document API or a document centric programming language (like XQuery, XSLT) or even better, a document-node centric programming language (like XLinq), and an XML document encoding.

You'll need both programming approaches. Don't use a single approach for everything, unless you're looking for trouble.

When you need to communicate both structured and semi-structured data, don't bundle them. Just link these resources using URI references.

Simple enough?

Full support for Git at Apache

2010-11-16T00:08:00.000-08:00

The Apache infrastructure team is starting to work on full support for Git at Apache. Woo-hoo!!

Many contributors already use the Apache Git mirrors and git-svn, but this will take Git usage at Apache to the next level.

Discussions are happening on the Apache infrastructure-dev mailing list there and JIRA issues have been opened to track the initial tasks there.

Issue INFRA-317 will be particularly interesting, I think... as it's about describing the basic Git workflows that will be used by Apache committers.

Git is a distributed version control system which makes it very easy for individual contributors to work in isolation on local private repositories for some time, then publish and merge their work with others later.

That's different from the traditional Apache way, where contributors from a project community collaborate more closely around a centralized (Subversion) repository.

So, I think it'll be interesting to see how these distributed and centralized workflows can be reconciled, to allow people to use Git for everything it's good at, while still following the Apache way.

My VI configuration file

2010-11-15T00:53:00.000-08:00

After a few years using Eclipse, I've gone back to VI. I actually use VIM, an extended version of the original VI.

I won't expand on the reasons, but I just find it simpler, faster, and less distracting than a graphical IDE like Eclipse.

I've experimented a bit and found that I'm able to better focus on writing code with VI in a single barebones 24x80 window, instead of spending time and energy clicking and moving around outline views and windows.

I use exuberant-ctags to navigate big codebases, and Google search as a richer replacement for code-assist.

After I used VI to show and write code in my ApacheCon demos, a few folks asked me for my VI configuration supporting syntax highlighting, auto-indenting and code navigation.

Here's my $HOME/.vimrc file. Nothing fancy, but I find it useful:


" vi-improved mode
set nocompatible

" cycle buffers without writing
set hidden

" backup while writing
set writebackup

" file completion
set wildmenu
set wildmode=list:longest

" update window title
set title

" display cursor location
set ruler

" display current command
set showcmd

" short message prompts
set shortmess=atI

" silent
set noerrorbells

" switch to current file's directory
set autochdir

" remember marks, registers, searches, buffer list
set viminfo='20,<50,s10,h,%

" keep a big history
set history=1000

" syntax highligting
syntax on

" auto smart code indent
set autoindent
filetype indent on
set smartindent
set smarttab
set tabstop=4
set softtabstop=4
set expandtab
set shiftwidth=4
set shiftround

" allow backspacing in insert mode
set backspace=indent,eol,start

" incremental search
set incsearch
set nohlsearch

" ignore case
set ignorecase
set smartcase

" restore last cursor position
function! RESTORE_CURSOR()
    if line("'\"") > 0 && line ("'\"") <= line("$")
        exe "normal! g'\""
    endif
endfunction
autocmd BufReadPost * call RESTORE_CURSOR()

" navigate code using exuberant ctags
set tags=tags;$HOME
set tags+=$HOME/.vim/tags/cpp
set tags+=$HOME/.vim/tags/curl
set tags+=$HOME/.vim/tags/httpd
set tags+=$HOME/.vim/tags/libmozjs
set tags+=$HOME/.vim/tags/libxml2

Hope this helps.

ApacheCon NA 2010 - Apache Tuscany track sessions

2010-11-05T20:19:00.000-07:00

On the plane back from the ApacheCon North America 2010 conference in Atlanta.

Most US flights have internet on board now. That's pretty cool!

As promised, here are some of the slides we presented on Thursday in the Apache Tuscany track.

More presentations also on Luciano Resende's blog and Paul Fremantle's blog.

Hope you liked conference! See you on the Apache mailing lists!

ApacheCon NA 2010 - Building Apps with Apache Tuscany

View more presentations from jsdelfino.

ApacheCon NA 2010 - Developing Composite Apps for the Cloud with Apache Tuscany

View more presentations from Jean-Sebastien Delfino.

ApacheCon NA 2010 - High Performance Cloud-enabled SCA Runtimes

View more presentations from jsdelfino.

JavaOne 2010 - Composite Apps for the Cloud with Apache Tuscany

2010-10-19T23:03:00.000-07:00

Time flies! Almost a month has passed since JavaOne... I wanted to post my JavaOne 2010 presentation earlier, but have been busy with work... and a nice vacation in France.

Luciano Resende and I presented our experience with various cloud infrastructures and platforms, the challenges we're seeing, and how SCA composition and the Apache Tuscany and Apache Nuvem projects can help.

Here's the presentation abstract:

Today's cloud environments pose new challenges for application developers: hiding cloud infrastructure from business logic, assembling components on heterogeneous and distributed cloud environments, and optimizing the provisioning of the required cloud resources.
This session will demonstrate how to use Apache Tuscany and the Service Component Architecture (SCA) to develop, build, and run an application composed of several service components in a distributed cloud environment.
We'll illustrate how to encapsulate cloud infrastructure services as SCA components to simplify the construction and assembly of the application and how to move components around and rewire the application to adjust to new business and cloud deployment conditions.

... and the slides:

S314011 - Developing Composite Applications for the Cloud with Apache Tuscany

The session went pretty well. The room was full, the demos worked (see slides 22 and 27 for diagrams of what we demo'ed). There were interesting questions and a good discussion after the presentation.

New book: Tuscany SCA in Action available

2010-08-18T15:48:00.000-07:00

A new book by Simon Laws, Mark Combellack, Raymond Feng, Haleh Mahbod and Simon Nash, Tuscany SCA in Action provides a simple step-by-step guide on how to develop applications leveraging SCA (Service Component Architecture) and Apache Tuscany.

InfoQ has just published an interview of the authors with interesting questions and answers on SCA and an excerpt of the book.

There's a few things I really like about it:

The authors are the founders of the Apache Tuscany project and have been involved in SCA from the beginning, so they know what they're talking about.

The book uses a lot of examples, available in the Tuscany project, easy to download and try as you read.

The examples are actually part of an official Tuscany release, in sync with the Tuscany v1.6 release (and I'm hoping that they'll be ported later to the evolving Tuscany v2.x release stream).

I think this book is a very good introduction to SCA and its implementation in Apache Tuscany. The printed book is not available yet, but an ebook version is already available there.

Apache Libcloud v0.3.1 released

2010-05-13T12:45:00.000-07:00

The Apache Software Foundation and the Apache Libcloud project have announced the release of version 0.3.1 of Apache Libcloud. Apache Libcloud is a pure Python client library for interacting with many of the popular cloud server providers.

Apache Libcloud v0.3.1 is available for download from the project download page as well as PyPi.

I've been playing around with Libcloud to experiment and deploy SCA apps running on the Tuscany native (aka Tuscany-cpp) and Tuscany Java runtimes to EC2, and it's been working pretty well for me. The Libcloud API is very simple and makes it really easy to write Python scripts that automate the deployment, start, stop etc of compute nodes running SCA apps.

If I find some free time this weekend, I'll try to clean up a bit the Libcloud-based deployment scripts I've been playing with, and add them to the Tuscany native sample apps.

Also, I think it should be relatively easy to hook similar scripts with the Tuscany SCA domain manager Web app, which currently only supports local nodes but could be extended to deploy SCA components to remote nodes in the cloud.

The Security Exchange Commission, Python, Excel and Functional Programming

2010-05-04T22:36:00.000-07:00

What does the SEC have to to do with Python, Excel and Functional Programming? Well, the title of this blog is Random Thoughts, so here are a few random thoughts.

Two weeks ago the US SEC Security and Exchange Commission proposed significant revisions to the rules governing the offering process, disclosure and reporting for ABS (Asset Backed Securities).

One of the proposals is to require ABS issuers to file Python computer programs describing the flow of funds (called waterfall) in ABS transactions. I think that's pretty cool!

That's cool for investors

Waterfall provisions in modern securitizations and CDOs (Collateralized Debt Obligations) are complicated. They're described in 'prospectus' documents written in plain english. That's verbose, error prone, and makes it difficult for investors to create credit and cash flow models and conduct their own evaluations of ABS. The last financial crisis revealed that many investors relied on ratings provided by third party credit rating agencies instead of doing their own evaluation homework, and were not fully aware of the risk in the underlying mortgages within the pools of securitized assets.

Providing a Python program modeling the waterfall for each ABS would help investors conduct their own evaluations and make more informed investment decisions. The Python program source code would be filed on EDGAR (Electronic Data-Gathering, Analysis, and Retrieval). Investors could just download the program for an ABS, input their data assumptions and model the waterfall right away.

That's cool for Python and programming

I've dissected a few ABS prospectus documents and I'm starting to think that these Python programs could look just like Excel spreadsheets (which most investors are familiar with), with user defined formulas written as Python functions, in a functional programming style.

I did some intense Excel programming long time ago for a project at France Telecom. I needed to model the characteristics of their applications, middleware and network. I was a C coder at the time and had to switch to Excel programming for 4 months. What a shock! :) My first reaction was to go back to my programmer comfort zone and write Visual Basic instead of Excel formulas. Then I decided to try harder with Excel formulas, and I loved it! After a few weeks I was amazed by the power of such a simple programming model: cells, values and formulas that you can just compose... and how easily I could translate my specific business and network engineering rules to Excel formula 'code'. I guess I'm stating the obvious to the millions of Excel users out there :).

More recently I've become interested in functional programming, and I'm starting to see a lot of similarities between how I approach functional programs and Excel worksheets. If Excel supported recursion, the ability to define reusable formulas and treat them as data, I would dare to say that it's the most successful functional programming language ever! After all, a spreadsheet is just a functional program made of formulas/functions evaluated as their values are needed.

I've also done quite a bit of programming in Python, and find it pretty natural to write Python programs in a functional programming style (mostly inspired from Scheme). Purists will argue that Python is not a real functional programming language like Haskell. I agree, but I've found enough functional constructs in Python to get close to the happy functional feeling I get with Scheme. Python is also less cryptic than Haskell and simpler than F# or Scala.

Plain english, functional programs and Excel programs

One of the things I like about functional programming is how I can translate a plain english description of a real world process or rule into a functional program. Let's take a simple example and write a plain english description of a function (or formula) that calculates the sum of a list of numbers:

"The sum of a list of numbers is equal to the first number in the list plus the sum of the numbers in the rest of the list. If the list is empty the sum is 0."

Let's compare that to a typical imperative programming implementation of "sum" (with assignments, state and side effects):

In Java:

static int sum(int[] numbers) {
    int total = 0;
    int i = 0;
    while (i < numbers.length) {
        total = total + numbers[i];
        i = i + 1;
    }
    return total;
}

In Python:

def sum(numbers):
    total = 0
    i = 0
    while i < len(numbers):
        total = total + numbers[i]
        i = i + 1
    return total

That's quite different from our original english description, right?

It'd be interesting to reflect on what a programmer needs to go through to come up with that code. There seems to be a lot of invention and concepts to grasp here: several variables, a loop, access to values in a list using an index.

How would you you explain the existence of variable "i" and its relationship with the original english description of "sum"? What's a variable by the way? Do we call it variable because it varies over time? Is "numbers" a variable too? But it doesn't vary, right? Is it just a value then? What does it mean to loop over a list? What if I don't know the length of the list upfront?

Here comes incidental complexity... Incidental complexity is different from the inherent complexity of the real world process you're trying to model in a computer program. It is the complexity baggage that comes with a particular way to model that process and the programming language used to do it.

You don't want incidental complexity. It gets in your way as you're just trying to calculate that sum...

Now, let's write "sum" in Scheme:

(define (sum numbers)
    (if (empty? numbers)
        0
        (+ (first numbers) (sum (rest numbers))
    )
)

Then in Python again, using a functional style inspired by the Scheme version:
(using simple Python implementations of "empty", "first" and "rest")

def sum(numbers):
    if empty(numbers):
        return 0
    return first(numbers) + sum(rest(numbers))

Isn't that functional version of "sum" much closer to the original plain english description? To me, that looks like a straight translation of the plain english description to Python, more precise, and executable. No more incidental complexity here!

Wouldn't that be a good way to translate an ABS waterfall prospectus to a computer program? Could an Excel user working at a financial institution write that piece of Python code?

To try to understand that, let's try to get in the mind of an Excel power user and see how he'd model "sum" in Excel if it wasn't already a builtin function.

1  A       B
2  Numbers Sum
3    
4  1       6 = A4 + B5 = first number in the list (A4) + sum of the rest (B5)
5  2       5 = A5 + B6
6  3       3 = A6 + B7 
7

Surprise! That's almost the same thing as the Python or Scheme sum() functions!

"= A4 + B5" is "the first number in the list" (A4) + "the sum of the rest" (B5).

The Excel version introduces a few subtle differences, however:

No recursion, the Excel model is flat and the application of a formula to a series of values is realized by copying/pasting the formula over a series of cells (B4:B6).

I'm not sure what to think about that yet. Recursion is a powerful functional programmer tool, sometimes difficult to master in computer programs, but used extensively in financial formulas.
Formulas have no name and are reused by copy/paste instead of references to their name.

That's interesting. Most people will agree that copying the same formula over many cells makes it more difficult to change it. But naming a function forces the spreadsheet developer to learn that name, like a new word in a language. Once you have 100 functions, that's 100 words, and you've invented a new ad-hoc language (called a Domain Specific Language these days). That new language is often more complex than Python or Excel, and you're forcing people to learn it. Is that better?
Function and function application are mixed. B4 doesn't contain the sum() function, it contains an application of sum() to live data (A4 and B5). In other words B4 contains an instance of sum() loaded with the values from A4 and B5.

Is that bad? No, I think it's actually great as you get instant feedback with the results of your formulas as you shape your code and data. Think of it like a builtin TDD (Test Driven Design), which you wouldn't have to worry about. Think about writing code, testing it at the same time without having to explicitly write tests, and using the test results to guide you as you code. Sounds like a programmer's paradise? That's what you do with Excel, without even thinking about it :)

Off to the next financial crisis

Going back to ABS waterfall Python programs... If the SEC proposal is adopted, I think they'll present a great opportunity to marry the Excel-like worksheet paradigm, Python and functional programming.

Pushing that dream a little further, here would be my wish list for a Python-based ABS waterfall development environment:

Feels like an Excel worksheet, tables, cells, workbooks, the ability to mix presentation, values and formulas.
Takes formulas as simple Python functions.
Allows me to define my own reusable named formulas and just copy/paste their name around.
Like Excel, loads my Python functions with live data and shows me the results all the time.
Tolerates an incomplete worksheet (or program).
Allows several people to collaborate on a worksheet.
Is open-source (as the SEC proposal mentions the open source nature of Python).
Runs in the cloud off a bunch of Web pages -- what doesn't these days :)

Anyone would like to try to build that environment?

Just imagine what millions of Python-enabled-functional-spreadsheet-in-the-cloud power users could do to the next financial crisis :)

Let me know what you think...

Functional Programming, a perfect fit for Web Programming?

2010-04-19T00:15:00.000-07:00

The last few months I've become really interested in Functional Programming. I've learned Haskell, Scheme, and F#. I've also looked at how to do Functional Programming in Imperative Programming languages like Python, Java, C and C++.

I've been doing that as a hobby on evenings and weekends, as I'm only using Java at work. Evenings and weekends have added up and I've actually done quite a bit of real functional programming in Scheme and F# now, as well as some functional 'style' programming in Python and C++ in weekend contributions to the Apache Tuscany C++ SCA project.

I'm only starting to digest that Functional Programming experience.

I've also done Web programming for 15 years and have been lucky to work on projects using so many Web frameworks and languages that I can't remember all of them... Well, I still remember when I almost gave up on programming altogether, out of frustration as I was trying to do something like... AJAX in 1996... and Internet Explorer and Netscape were driving me crazy and crashing all the time :). We didn't have a fancy name like AJAX at the time, just dynamic HTML and Javascript.

I'm not sure I've completely digested Web programming yet, but I've got over a number of good and bad, or let's say 'learning', experiences.

Last week I was bored and I started to think about similarities between Functional Programming and Web Programming. I thought about a few fun analogies:

Start with an HTTP GET http://whatever.com/add&x=1&y=2. That looks to me like the application of a function 'add', with precise input and output. An idempotent function with no side effects. Like with Functional Programming, the result can be produced by executing code (a Python script for example), or just a table lookup (a pre-generated page for example.)

Use a URL like http://whatever.com/add/1/2. That looks less that the location of a script with parameters and more like the location of a resource, or data item. It's starting to blur the line between function application and access to data. Like with Functional Programming, Web functions and data seem interchangeable.

Do an HTTP redirect or use a URL shortening service to shorten http://whatever.com/add/1 into http://bit.ly/axIyzo for example, and you have function currying.

Use a caching proxy server or use client side caching based on last-modified of etags headers, and you're doing function memoization.

The HTTP POST, PUT, DELETE verbs introduce precisely identified side effects. That's what you're looking for when you do Functional Programming. When you can't avoid them, you want to be very explicit and precise about the side effects introduced by your code.

Return an HTML page. That looks like a value representing a computation, which will perform some I/O in your Web browser at the edge of the system. Will functional programmers see the analogy with an I/O Monad?

HTTP is stateless. When you pass state around across multiple HTTP interactions, wrapped in additional hidden fields or parameters on your URLs, are you implementing a State Monad?

Finally, could Functional Reactive Programming and lazy event streams help you implement AJAX callbacks, subscribe to ATOM feeds or Pubsubhubbub updates for example?

Something to think about... What if Functional Programming was a perfect fit for Web programming? What if you could compose Web functions like functional programmers compose functions?

GCC 4.5 released! now supports Lambda Expressions

2010-04-14T15:12:00.000-07:00

GCC 4.5 just got released! with support for lambda expressions (see first class functions also), as part of some experimental support for C++0x (the upcoming ISO C++ standard.)

There are many other changes in GCC 4.5 but this new support for lambda expressions is great news for all the Linux developers out there interested in functional programming!

Experimenting with Databases

2010-04-13T02:38:00.000-07:00

I've been playing around with various Open-Source databases for a little weekend database project.

Last weekend was really rainy. April usually feels like summer here, but this year is an el Nino year so we're getting a lot of rain... I shouldn't complain though, as the rain turns into great powder in Tahoe and we've had a great ski season!!

But I didn't go skiing last weekend... For my database project I had two buckets of data to work with:

Bucket (A)
- low volume of data (35000 entries)
- structured data with a stable structure
- low update and access rate (a few thousands a day)
- need flexible query and offline analysis capability

Bucket (B)
- higher volume (500000+ entries)
- unstructured text
- lots of reads/writes, no deletes
- need super fast key-based lookups for interactive access

I looked at the following SQL and NoSQL databases:
- BerkeleyDB
- Apache Cassandra
- Apache CouchDb
- MongoDB
- MySQL
- PostgreSQL
- TinyCDB
- Tokyo Cabinet

There has been a lot of buzz about SQL vs NoSQL databases recently so I wanted to try both types of databases.

I quickly eliminated the following products as their licenses were not permissive enough for me and not compatible with the Apache license:
- BerkeleyDB (viral as code using it must be made available in source form)
- Tokyo Cabinet (LGPL)
- MySQL (GPL)
- MongoDB (AGPL)

Then I tried the remaining products. I say 'tried' instead of 'tested' as I didn't have much time for a scientific test approach and was mostly trying to get a feel for each database. I just used the simple tools provided out of the box with each product and didn't spend time optimizing their configuration.

I used data from bucket (B), hoping that the products that performed well with (B) would fit the bill for bucket (A) as well: 580000 text entries, keys ranging from 3 to around 200 bytes, same characteristics for values, total 137 Mb

I tried to insert the 580000 entries one by one, then did 100000 random gets.

Here are my first observations:

Apache Cassandra
I used binary release 0.5.1. Cassandra was easy to setup and get started with, although I got a little lost in their Wiki.

580000 inserts with cassandra-cli took 15mn, used 100% CPU, quite a bit of network I/O and 520Mb of disk space.

Average speed:
- insert 1.55ms
- get 1.58ms

Apache CouchDb
I built version 0.11.0 from source and had to work through issues with different levels of libmozjs on Ubuntu. Getting started took 10mns, although I'd have liked a real client instead of just CURL.

580000 inserts with curl took 124 mn, used 100% CPU, a lot of network I/O and... 4.5Gb of disk space!

Average speed:
- insert 12.82ms
- get n/a... couldn't measure gets as the CouchDb view creation just kept hanging.

I also didn't understand how to create CouchDb views right away, and could have used a better 'Getting started with views doc'.

PostgreSQL
Version 9.0alpha4 was easy to build from source, setup and get started. I picked version 9.0 as it's supposed to have new high availability + replication features that I may want to try later...

580000 inserts with psql took 85 mn, used mostly disk I/O (didn't use much CPU at all, which was good) and 243Mb of disk space.

Average speed:
- insert 8.79ms
- get/select (with no index) 173ms

After creating an index, which took 40sec:
- get/select 0.085ms

SQLite
I built version 3.6.23.1 from source and got started in about 5 mns.

20000 inserts with sqlite3 took very long, using mostly disk I/O (I lost patience and killed it after 15mn.)

Average speed:
- insert 45ms
- get n/a... didn't measure as I had given up loading the database

I think the problem was that sqlite3 does an fsync for each write. Their doc describes how to not fsync, but then I can't afford losing my data... and I won't be able to batch inserts either.

TinyCDB
That one is a different animal... I used tinycdb version 0.77. It's really tiny (core code under 1400 lines of C) and easy to build from source, with a tiny doc, but very simple to use.

TinyCDB is a 'constant' database. You can't modify it after you've added entries to it and saved the database file. To update the database, you need to rebuild it from scratch with the new values. It's so fast though, that doing that is not necessarily a problem...

580000 inserts with the cdb command took 2.25 sec and 150Mb of disk space

Average speed:
- insert 0.00379ms
- get 0.00683ms

That's about 1 to 2 orders of magnitude faster than the other databases!

With these initial observations, I'm tempted to experiment further with PostgreSQL for bucket (A) and TinyCDB for bucket (B).

Both packages are proven and stable technologies too. PostgreSQL has been deployed in production for years, and TinyCDB is used in production in well known DNS and mail packages like djbdns, fastforward, mess822, qmail etc.

I'll probably have to put together a simple cascading scheme to handle updates with TinyCDB (a small TinyCDB db containing recent and/or frequent updates, periodically flushed into a bigger one, for example), but it shouldn't be too hard to do.

I'll try to post an update here after another weekend or two playing with this...

Apache Vysper XMPP Server first release

2010-04-05T08:28:00.000-07:00

The Apache MINA project has announced the very first release of their XMPP server product Apache Vysper.

Apache Vysper 0.5 is easy to install, easily extensible and embeddable. It is based, of course, on MINA.

You can find the Vysper release here: http://mina.apache.org/vysper/apache-vysper-05-release.html

Vysper comes with Multi-User Chat and Publish-Subscribe extensions enabled out-of-the-box. It does not yet implement the whole XMPP spec and stack, so be careful when you plan to use it in a production environment.

I think this is great news! as Vysper is the first XMPP server with a friendly Apache open-source license. The other server with a friendly license is Prosody, written in LUA, with an MIT/X11 license. All the other good servers (jabberd, ejabberd etc) are GPL.

I experimented a bit with Vysper in the Apache Tuscany project to test some XMPP Chat components and it worked pretty well for me! I also found it really easy to embed, register Jabber/XMPP ids, start the server etc. Here's an example from a Tuscany test case.

Google Summer of Code 2010 now accepting student applications

2010-03-31T22:44:00.000-07:00

The Google Summer of Code 2010 program is now accepting student applications!

Google Summer of Code is a global program to introduce students, ages 18 and over, to the wonderful world of Open Source development. GSoC offers student developers stipends to write code for various Open Source projects. Students can choose from 150 Free and Open Source software projects, in technical areas as diverse as gaming to humanitarian efforts to operating system design.

More info on the Google open-source blog and the GSoC home page.

Each Open Source project contributes mentors and a list of ideas for the students to work on. I've signed up to volunteer as a mentor for the Tuscany project from the Apache Software Foundation. The GSoC ideas from all Apache projects can be found there, the Apache Tuscany project ideas here.

Here are a few ideas from the Apache projects that I found particularly interesting:

CASSANDRA-876 - session consistency for Apache Cassandra.

COMDEV-26 - A SPDY protocol connector for Apache Tomcat.

CXF-2738 - a Zookeeper based service repository for CXF.

HARMONY-6455 - support for closures in Java / Harmony.

MAHOUT-342 - map/reduce enabled neural network with back propagation.

VYSPER-189 - Websockets / BOSH support for XMPP Ajax clients.

ZOOKEEPER-702 - Smart distributed failure detector for Zookeeper.

ZOOKEEPER-703 - DNS server backed by Zookeeper.

and -- shameless plug :) -- the ideas I've submitted for Apache Tuscany:

TUSCANY-3521 - a Pubsubhubbub binding. Curious about Pubsubhubbub? it's a really promising Web based publish / subscribe protocol, initially from Google and Six Apart.

TUSCANY-3522 - a No-SQL datastore component over Apache Cassandra, CouchDb, Hadoop/HBase, a good opportunity for students to experiment with these new cool No-SQL databases.

TUSCANY-3523 - distributed Haskell components, or "functional programming meets distributed composite applications".

Finally, some cool ideas in these ideas lists from Facebook, Mozilla, Python, Eclipse, Ubuntu, and Haskell.

That's just my selection, there's many more to check out...

So, students... want to gain real world software engineering experience and get paid? check out these project ideas, and apply now!

CERN Large Hadron Collider ready for record particle beam collisions

2010-03-29T21:53:00.000-07:00

"We are opening the door to New Physics, to a new period of discovery in the history of humankind," said Rolf-Dieter Heuer, director-general of CERN, the European Organization for Nuclear Research.

Pretty exciting news from the Swiss / French border! In a few hours, high energy particle beams will start circulating in opposite directions around the CERN Large Hadron Collider (LHC), causing particle collisions that could help unlock secrets of the dark matter, dark energy and what happened in the universe 13.7 billion years ago...

Well, it's probably going to take years to digest all the collision data, but it looks like a promising step...

More info: CERN press release, Reuters, Associated Press

Live Webcast starting 11:30 pm PST... looks like I'm not going to sleep much...

Binding to port 80 without running as root

2010-03-28T14:27:00.000-07:00

I've been looking for a way to allow server programs like Apache httpd, memcached, Couchdb, Qpid etc to bind to privileged IP ports under 1024 like 80 or 443 for example, without running them as the root user. Normally, on Linux you need to run as root to be allowed to bind to these standard privileged ports

I don't like to run anything as root on a server as it makes it vulnerable to security holes in untrusted application code... and stupid mistakes. Has anybody ever completely messed up a server with a stupid mistake like rm -rf of the wrong directory? :)

To mitigate the risks you can run as root in chroot jails, but placing everything you need in a chroot jail and communicating with what's running in the jail quickly gets complicated.

Apache httpd has built-in support for dropping root privileges after binding to a privileged IP port, to run untrusted application code as a normal user (see the User configuration directive.) I don't like that either as the httpd parent process still runs as root. Also, some of the log files are still created under root and I have to become root again to process them (if running with a tight 0007 umask) or relax the file permissions and make them visible to other users on the server.

So, I've been trying to run a tight compartmentalized ship and run most server programs under normal user accounts. That makes it really easy to maintain isolated sandboxes with different configurations of programs, replicate them across machines, control resource consumption with user quotas, run scripts to monitor each sandbox activity, setup emergency procedures to shutdown everything running in a particular sandbox, etc.

... Then I found privbind!

Privbind is a nice little utility that allows a program to bind to privileged IP ports without running as root. With privbind I'm now able to run all my server programs as regular users, and still bind them to privileged IP ports.

Here's how to use it to start Apache httpd: sudo privbind -u foo apachectl -k start
That runs Apache httpd under user foo, but still allows it to bind to privileged port 80.

More info on how privbind works can be found in the privbind Manual.

P.S. An alternative is to bind the server programs to non-standard unprivileged ports (8080 instead of 80 for HTTP for example.) and use an iptables firewall configuration to forward the IP traffic from port 80 to port 8080. I'll talk more about that in another post...

Composability as a way to control Complexity

2009-12-16T12:50:00.000-08:00

I came across this great video of a conversation with Brian Beckman on Complexity in computing, on MSDN's Channel 9.

In that video Brian Beckman compares Inherent or Essential complexity (of the problem you, programmer, are trying to solve) and Accidental complexity (not inherent in the initial problem, but introduced by the many technologies and programming languages you have to use for example.)

He also introduces Functional Programming, how it limits the ways to hook things together and dependencies on the context in which software components executes, and how that makes it much easier to compose complex systems out of small building blocks.

In summary, Composability is the way to control Complexity, and functional programs are easier to compose.

Also, don't miss older talks and conversations with Brian Beckman, perhaps even more interesting:

Don't fear the Monads is an enlightening white board introduction to Functional Programming and Monads. Brian Beckman discusses Monoids (a mathematical concept representing a collections of things and rules to compose them that themselves follow some rules, like associativity for example), Monads (used to carry extra data around values, representing side effects for example, plus composition operators), and how they help you compose large and complex systems out of small function building blocks.

In The Zen of Stateless State, part 1 and part 2, he shows how functions with no side effects can still be useful in the real world, flow and alter state, and still be composed using State Monads (simple functions that take a state and return a state + content pair and an operator that can thread the state through a composition.)

Then he discusses how functions + monads force programmers to be explicit and very precise about what their code does (no undisciplined side effects under the covers of a function) and how that makes programs more reliable and composable.

Bottom line, mathematical precision is the key to software reuse and composition.

If you're not convinced, watch the videos...

Reading list - Elements of Programming

2009-11-30T00:25:00.000-08:00

I just finished a fun book on the mathematical foundations of programming: Elements of Programming by Alexander Stepanov and Paul McJones.

I highly recommend this book if you're interested in a deeper understanding of programming or looking for principles to decompose software into components.

It starts with simple principles, a foundation mix of mathematics and code, and then builds on... It's short (230 pages) but will take a while to read as it's full of exercises, which you really need to do if you want to get the most out of it.

It's not an easy read, but Elements of Programming could change the way you think about programming...

The code from the book and an interview of the authors are available online.

Connecting to WPA2 networks with Ubuntu 9.10

2009-11-23T10:07:00.000-08:00

Upgrading to Ubuntu 9.10 / Karmic broke my Wireless connection. I can connect to open networks but with hidden WPA2 networks the Network Manager Connect button is always disabled...

Looks like others are running into that bug too: https://bugs.launchpad.net/ubuntu/+source/network-manager-applet/+bug/446394. A fix has been committed in the Gnome repository, but there's no official installable package for it yet.

If you're impatient like me and need the fix now, here's how to build it:

Get the network manager applet source code from the Gnome GIT repository:
git clone git://git.gnome.org/network-manager-applet

The following version works for me:
e4cf9c6f1f95ce7d52d7cec435d95a5ddbf935a7 Tue Nov 17 00:58:14 2009 -0800

See the change log:
http://git.gnome.org/cgit/network-manager-applet/log/

and the fix for the disabled Connect button bug:
http://git.gnome.org/cgit/network-manager-applet/commit/?id=4162285fd7c53e8b684e1c1513a2ba6ce865e073

Build it:
./autogen.sh
./configure
make

If the build fails, you may have to install missing dev dependencies:
intltool
libdbus-1-dev
libdbus-glib-1-dev
libgconf2-dev
libgee-dev
libglade2-dev
libgnome-keyring-dev
libgtk2.0-dev
libiw-dev
libnm-glib-dev
libnotify-dev
libpolkit-gobject-1-dev

That list is what worked for me, and I already had many dev packages on my laptop. If you're still missing dependencies just try to guess what's missing from the build messages and install them until the build is happy.

Install:
sudo make install

The updated Network Manager applet works like a charm, I don't need to carry an Ethernet cable around anymore :) Bonus: the applet UI has improved a bit since the Karmic release.

Hope this helps (as a few folks at work have been running into that bug too and were asking me about it on Friday.)

Scratch - Programming and Component Assembly for All

2009-11-19T09:39:00.000-08:00

Scratch makes the cover story of the November issue of the Communications of the ACM. The whole article is available on the MIT site here and the ACM site there.

Here's an overview of Scratch, from their web site:

Scratch is a new programming language that makes it easy to create your own interactive stories, animations, games, music, and art -- and share your creations on the web.

Scratch is designed to help young people (ages 8 and up) develop 21st century learning skills. As they create and share Scratch projects, young people learn important mathematical and computational ideas, while also learning to think creatively, reason systematically, and work collaboratively.

Basically, you assemble a program out of building blocks using a nice little IDE.

Here's what s Scratch program looks like:

Then you can share your program with others, in sort of a social app store. The Scratch site has been there 2 years, and now hosts 601,000 projects, created by 90,450 contributors... That's a lot!

I wonder if something like Scratch would help non-programmers assemble enterprise applications out of components, or sketch business process applications for example...

Simpler than JSON?

2009-11-18T00:23:00.000-08:00

With XML falling out of fashion, JSON is all the rage these days.

The last few years we've been using a simple online store application to demo application composition with SCA. That store app allows you to pick fruits and vegetables out of a catalog and put them in your shopping cart.

There's versions of the store app in different languages, Java, Python, and an HTML+JavaScript client in the Apache Tuscany project, and more languages to come over time. The shopping cart is currently represented in XML in an ATOM feed.

I've been wondering... Should we switch from XML to JSON? Is JSON simpler? Is the JavaScript syntax so great? Any alternatives? Let's try and compare different representations of our shopping cart:

XML:
Here's the XML currently produced from Java bean representations of the cart, fruits and vegetables using some fancy JAXB code:

<ns2:root xmlns:ns2="http://tuscany.apache.org/xmlns/sca/databinding/jaxb/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="cart">
<items xsi:type="fruit">
    <name xmlns="">Apple</name>
    <price xmlns="">2.99</price>
</items>
<items xsi:type="fruit">
    <name xmlns="">Orange</name>
    <price xmlns="">3.55</price>
</items>
<items xsi:type="vegetable">
    <name xmlns="">Broccoli</name>
    <price xmlns="">1.99</price>
</items>
</ns2:root>

Yuk...

Here's without the namespace mess, as it's not needed by the store app to disambiguate what's flowing (as the app knows what it's doing), and would probably over-complicate any non-XML representation as well...

<cart>
<fruit>
    <name>Apple</name>
    <price>2.99</price>
</fruit>
<fruit>
    <name>Orange</name>
    <price>3.55</price>
</fruit>
<vegetable>
    <name>Broccoli</name>
    <price>1.99</price>
</vegetable>
</cart>

That looks a little better.

Here's with attributes instead of elements. It's shorter and should work too, I guess?

<cart>
<fruit name="Apple" price="2.99"/>
<fruit name="Orange" price="3.55"/>
<vegetable name="Broccoli" price="1.99"/>
</cart>

The problem is that over the years I've started to develop an allergy to XML (mostly because of the usual namespace mess), so let's go with the flow, be cool, and try JSON...

JSON:
Here's a possible JSON representation. The cart contains an array of objects, actually two levels of objects to distinguish fruits and vegetables.
{"cart":[
{"fruit":{"name":"Apple","price":2.99}},
{"fruit":{"name":"Orange","price":3.55}},
{"vegetable":{"name":"Broccoli","price":1.99}}]}

Another try, with the funky javaClass attribute expected by the Java JSON code to figure the type of each item in the cart... Not great.

{"cart":[
{javaClass:"sample.Fruit","name":"Apple","price":2.99},
{javaClass:"sample.Fruit","name":"Orange","price":3.55},
{javaClass:"sample.Vegetable","name":"Broccoli","price":1.99}]}

Here's another one following the Jettison or Badgerfish XML / JSON mapping. Multiple items of the same name are grouped in an array field. So in my example fruit is an array, vegetable is not. Now, add another vegetable to your cart, and vegetable morphs into an array... Confusing for lack of a better word :)

{"cart":{ "fruit":[
    {"name":"Apple","price":2.99},
{"name":"Orange","price":3.55}],
"vegetable":{"name":"Broccoli","price",1.99}
}}

Now I'm a little disappointed. JSON doesn't look significantly simpler than XML here, and Javascript concepts like arrays, (soft) objects and confusing mappings of multiple occurences of items are starting to bleed in my cart.

Let's try a little harder and find alternative representations:

YAML:
cart: - type: fruit
    name: Apple
    price: 2.99
- type: fruit
    name: Orange
    price: Pear
- type: fruit
    name: Orange
    price: 3.55
- type: vegetable
    name: Broccoli
    price: 1.99

I still need to add an artificial 'type' attribute to each item to represent different fruit and vegetable elements, but YAML is not bad for the cart example which has only one level of nesting. With more levels of nesting, YAML gets a little akward in my opinion (see section 2.1, lists of lists and mappings of mappings, in the YAML spec).

YAML also lacks the nice programming language feel of JSON, which I can just evaluate in the JavaScript interpreter, allowing me to blur the line between my shopping cart data and the code that uses it.

Scheme:
'(cart
(fruit (name "Apple")(price 2.99))
(fruit (name "Orange")(price 3.55))
(vegetable (name "Broccoli")(price 1.99)))

That Scheme language representation is not bad at all!
- it's really concise;
- the Scheme syntax is simple and easy to parse, () for lists, space as a separator;
- as the above expression is just constructing lists, I don't need to make my shopping cart data fit in Objects, Arrays or Maps;
- it's easy to distinguish symbols like fruit or name and user data like "Apple".
- a list can simply be tagged with a fruit or vegetable symbol to indicate its type;
- like JSON, the above expression is a piece of code easy to evaluate and test in a Scheme interpreter.

Python:
Another variation on the same theme with Python tuples, as Python is more widely known than Scheme or Lisp.
('cart',
('fruit',('name','Apple'),('price',2.99)),
('fruit',('name','Orange'),('price',3.55)),
('vegetable',('name','Broccoli'),('price',1.99)))

A little more verbose than scheme, but not bad.

So, I must admit... I think I much prefer the Scheme representation, or the Python tuple syntax. I realize that Scheme was created in the 70's, but hey, that doesn't mean it's bad :) -- see the Wikipedia entry on Scheme for its history.

I also realize that JSON is natively supported by the JavaScript interpreters used in Web browsers... but it looks like most people are using handcrafted JSON parser implementations (written in JavaScript) instead of the native JavaScript eval() or JSON.parse() so that argument is not very solid... and it shouldn't be hard to write similar parsers for the Scheme or Python syntaxes as well.

Oh, by the way... check-out that JSON parser page. Here's what it says:
JavaScript is a general purpose programming language that was introduced as the page scripting language for Netscape Navigator. It is still widely believed to be a subset of Java, but it is not. It is a Scheme-like language with C-like syntax and soft objects.

Why not just stick with Scheme's minimalistic syntax then? Why would I need JSON's C-like syntax or these fancy soft objects to represent my shopping cart? :)

Let me know what you think...

ApacheCon US 2009 - Apache is 10 years old

2009-11-01T00:13:00.000-07:00

ApacheCon US 2009 returns to the San Francisco bay area, considered to be the original 'home' of the Apache Software Foundation.

The conference starts on Monday in Oakland, and will celebrate the 10th anniversary of the Apache Software Foundation.

Details are available there, including free Hackathon and BarCamp events on Monday and Tuesday. If you're going, don't miss the Apache Tuscany track:

SCA, Java EE, Spring and Web 2.0 Come Together - Service assembly with Apache Tuscany SCA
Wed, Nov 4, 11am [details]

Do you need to integrate applications? Do you wish application assembly could be as easy as playing Lego with software components? We'll show you how easy it is to turn your code into SCA components and assemble them together with Apache Tuscany.

Luciano Resende and I will also present the latest on SCA and Tuscany, including some cool demos!

Distributed OSGi with SCA using Apache Tuscany
Wed, Nov 4, 1:30pm [details]

Do you want to know more about distributed OSGi? Raymond Feng will show you how you can use Distributed OSGi (RFC 119) and SCA to build a distributed application out of OSGi bundles.

Apache Tuscany: Applying OSGi modularity
Fri, Nov 6, 11:15am [details]

Are you wondering how to turn your code into OSGi bundles? Luciano will share how we did it for the Tuscany runtime code. The benefits, the pain points and issues, and how we resolve them.

Luciano, Raymond and I will be at the Hackathon. Come join us there if you're interested in SCA or the Apache Tuscany project.

Recovering JPEG images from a damaged compact flash card

2009-08-26T01:03:00.000-07:00

I've started to blog again...

I've not been lucky with data storage devices recently.

Last week my Thinkpad crashed. The disk got shredded into pieces under the Linux /lost+found directory. I had to crawl through it for two days to manually recover some of the files I cared about.

A few weeks ago the compact flash card in my digital camera got damaged too, giving me a "This card cannot be used" error message. I had two 2 GB of JPEG images on that card: family photos that I didn't want to lose! After reading the troubleshooting section of my Nikon's user manual I was almost ready to buy some expensive recovery software, or ship my card to a company that recovers images, when I thought:

"Wait... I'm a software developer. How about hacking a little program to recover those images myself?" That's what I did, and it was surprisingly easy! so I thought I'd share that recipe here.

If your compact flash card ever gets damaged (and from what I see on the Web that happens a lot), try the following:

Save a snapshot of your card in a file. On Linux do: dd if=/dev/<card-device> of=<card-file> bs=512
Read that file in chunks of 512 bytes, as the card is usually organized in sectors of 512 bytes.
If you see hex 0xff 0xd8 0xff 0xe1 at the beginning of a sector, you've hit the JPEG/Exif header of a JPEG image. If you're wondering what I'm talking about, just Google it or check the Wikipedia JPEG and Exif pages.
If the card is not too fragmented, most of your JPEG images are stored in contiguous sectors. Save that sector and the next ones into a .JPG image file until you hit the next JPEG header, save the next image, and go on until you reach the end of the card. You'll save up to 511 bytes of garbage at the end of each image file but that's OK, as JPEG viewers use the length of JPEG data encoded in the header and will ignore that garbage at the end of the file.

Here's a little C program translation of the above algorithm:

recover-jpg.c

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

/* Returns true if a sector contains a JPEG image header. */
const int isJPEG(const unsigned char *sector) {
  return(sector[0] == 0xff && sector[1] == 0xd8 && sector[2] == 0xff && sector[3] == 0xe1);
}

/* Buffer used to read sectors from the card */
unsigned char buffer[512];

/* Returns the first sector of the first JPEG image on a card file. */
const unsigned char* firstJPEG(FILE *card) {
  if (fread(buffer, 512, 1, card) != 1)
    return NULL;
  if (isJPEG(buffer))
    return buffer;
  return firstJPEG(card);
}

/* Recovers contiguous sectors containing the pieces of a JPEG image. */
const unsigned char* recoverSector(FILE* card, const unsigned char* sector, FILE* image) {
  /* write sector to the image file */
  fwrite(sector, 1, 512, image);

  /* read the next sector */
  if (fread(buffer, 512, 1, card) != 1)
    return NULL;

  /* done with this image, we've hit the next JPEG header */
  if (isJPEG(buffer))
    return buffer;

  /* recover the next sector from this image */
  return recoverSector(card, buffer, image);
}

/* Recovers JPEG images from a card file. Returns the number of recovered images. */
const int recoverJPEG(FILE* card, const unsigned char* firstSector, const int imageCount) {
  /* no sector to recover, we're done */
  if (firstSector == NULL)
    return imageCount;

  /* create JPEG image file */
  char path[16];
  sprintf(path, "DSC_%4.4d.JPG", imageCount);
  printf("Recovering %s\n", path);
  FILE* image = fopen(path, "wb");

  /* recover sectors from that image */
  const unsigned char* nextSector = recoverSector(card, firstSector, image);

  /* close image file */
  fclose(image);

  /* recover the next image */
  return recoverJPEG(card, nextSector, imageCount + 1);
}


int main(const int argc, const char *argv[]) {
  if(argc != 2) {
    printf("Usage: %s card-file\nto recover JPEG images from card-file.\n", argv[0]);
    printf("Card-file can be created like this:\ndd if=/dev/<card-device> of=<card-file> bs=512\n\n");
    return(1);
  }

  /* open card file */
  FILE *card = fopen(argv[1], "rb");
  if (card == NULL) {
    printf("Could not open %s\n", argv[1]);
    return 1;
  }

  /* recover JPEG images from it */
  const int imageCount = recoverJPEG(card, firstJPEG(card), 0);

  printf("Recovered %d images!\n\n", imageCount);

  /* close card file */
  fclose(card);

  return 0;
}

To build that program (on Linux Ubuntu 9.0.4) do: gcc -O2 recover-jpg.c -o recover-jpg

A side comment: With the -O2 option, the Linux GCC C compiler optimizes sibling calls, allowing you to use tail recursion instead of loops without growing the stack. That's what I did in the firstJPEG(), recoverJPEG() and recoverSector() functions, just for fun :)

To run the program: recover-jpg <your-card-file>

It'll save the JPEGs found on the card into files named DSC_0000.JPG, DSC_0001.JPG etc in the current directory.

That program successfully recovered the 700 JPEG images stuck on my compact flash card! I hope it'll work for you too if your card ever gets damaged...

Please drop me a comment here if you find any bugs in this little piece of code. Thanks!

Apache Tuscany SCA 1.1 released

2008-02-06T15:58:00.000-08:00

I've been busy working on the Apache Tuscany project. We've just announced our 1.1 release, here's the announcement:

The Apache Tuscany team is pleased to announce the 1.1 release of the SCA project.

Apache Tuscany provides a runtime environment based on the Service Component Architecture (SCA). SCA is a set of specifications aimed at simplifying SOA application development. The specifications are being standardized by OASIS as part of the Open Composite Services Architecture (Open CSA).

The Apache Tuscany SCA 1.1 release adds a number of features including a JMS binding, improved SCA policy support and a Widget implementation extension for representing client side Javascript Widgets as SCA components.

The release includes implementations of the main SCA specifications including:
- Assembly Model v1.0
- Policy Framework v1.0
- Java Common Annotations and APIs v1.0
- Java Component Implementation v1.0
- Spring Component Implementation v1.0
- Web Services Binding v1.0
- JMS binding v1.0
- EJB Session Bean Binding v1.0
- BPEL Client and Implementation v1.0 (experimental)

It also includes implementations of features not yet defined by the SCA specifications, including:
- JSON-RPC and Direct Web Remoting bindings
- RSS, ATOM, and HTTP bindings
- RMI binding
- Pub/Sub notification binding
- Widget component implementation
- XQuery component implementation
- Scripting component implementation (Groovy, Javascript, Python, Ruby)
- OSGi component implementation
- Support for OSGi bundles
- databindings for JAXB, SDO, JSON, AXIOM, Saxon, XmlBeans, DOM, SAX and StAX

The Tuscany SCA Runtime can be configured as a single node SCA domain or as an SCA domain distributed across multiple nodes. In addition Tuscany SCA supports the following host deployment options:
- running standalone
- running with embedded Jetty or Tomcat
- running with distributed nodes across multiple JVMs
- running as part of a standard Web application

The Webapp samples included in this release have been tried and tested on:
- Tomcat 5.5.20 and Tomcat 6.0.14
- Jetty 6.1.3
- Geronimo 2.0.2 Tomcat6 jee5
- WebSphere 6.1 fix pack 9

For full detail about the release see the RELEASE_NOTES and CHANGES files.

To download the distributions or for more information about the release please go to: http://incubator.apache.org/tuscany/sca-java-releases.html

To find out more about OASIS Open CSA go to: http://www.oasis-opencsa.org.

Apache Tuscany welcomes your help. Any contribution, including code, testing, contributions to the documentation, or bug reporting is always appreciated. For more information on how to get involved in Apache Tuscany visit the website at: http://incubator.apache.org/tuscany.

Thank you for your interest in Apache Tuscany!
The Apache Tuscany Team.

---
Tuscany is an effort undergoing incubation at the Apache Software Foundation (ASF), sponsored by the Apache Web services PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.

Back from ApacheCon US

2007-11-26T19:53:00.000-08:00

Back from ApacheCon US conference in Atlanta. Already two weeks ago... I was on vacation last week.

I attended a few interesting technical sessions:

Hacking Atom with Apache Abdera - Garret Rooney
Abdera looks good. I'm probably going to start looking into porting Apache Tuscany's Atom binding to it soon.
Comparing Java Web Frameworks - Matt Raible
Two sets of slides there and there.

Here are my notes:
- Don't believe the hype.
- Choose the right tool for the job.
- Most of these frameworks are awful anyway.
- Flex is really popular.
A little REST and Relaxation - Roy T. Fielding
The slides are there.

Quote from the presentation: "An architectural style is a coordinated set of architectural constraints that restricts the roles and features of architectural elements, and the allowed relationships among those elements, within any architecture that conforms to that style."

Huh Oh, I already feel relaxed...
WS-* vs. REST: Facts, Myths and Lies - Sanjiva Weerawarana
My notes:
- <xml><is><great></is></xml>
- <or>...</or> <is><it/></is>?
- <do><i><feel><relaxed/></feel></i></do>?
- Don't believe the hype.
- Choose the right tool for the job.
- Building distributed systems is tough anyway.
Back to the Future and One Laptop Per Child - Jim Gettys
My notes:
- This is really really useful.
- This is amazing technology.
- Again, this is really really useful.
- For $200 you can participate in the Give One Get One program.
- If you're a software developer you can volunteer to help. More information there and there.

Something to think about: How many of you are making sure that your own child has access to a computer? and why are you doing it?

This One laptop Per Child program can make a difference...

Airbus A380 landed in San Francisco

2007-10-06T10:14:00.000-07:00

On Thursday morning the new Airbus A-380 landed in San Francisco for the first time!

Here are two photos taken from the bay shore across the street from my office:

It's big!!

About to land... view of the city in the background.

Developing an SCA application with Apache Tuscany and Eclipse

2007-10-03T12:12:00.000-07:00

I'm working on Apache Tuscany. Tuscany implements the Service Component Architecture (SCA) specification.

The last few months I spent a lot of time in Eclipse developing SCA samples and demos for Tuscany. I found myself tweaking my various Eclipse workspaces again and again to do the following:

- Define a Tuscany User Library containing all the Tuscany JARs to make it easier to add them to my project buildpath.

- Associate .composite files with the XML editor and register the SCA and Tuscany XML schemas to get code-assist in the editor and validation of my composite files.

- Write programs to launch Tuscany configured with my various composites... wishing that I could just right-click on the .composite and select a Run As / Tuscany...

Finally, I got tired of repeating these steps again and again and decided to put together a small tool made of two simple Eclipse plugins to package Tuscany in a Library and take care of all that configuration.

I'm thinking that it could be useful to others as well so here's a brief description of how to use it.

The tool works with Eclipse 3.3 Europa. It can be installed on top of the Europa "Eclipse IDE for Java Developers" distro. Here's how to install it:

From the Eclipse Workbench menu bar select Help - Software Updates - Find and Install. The Install - Feature Updates dialog will open.

Select "Search for new features to install", press Next. The "Update sites to visit" dialog will appear.

Press New Remote site.

In the New Update site dialog enter,
Name: Tuscany
URL: http://people.apache.org/~jsdelfino/tuscany/tools/updatesite
Press OK.

Make sure that Tuscany is selected in the list of sites to visit as shown here:

Now press Finish, the Updates - Search results dialog will appear. Select Tuscany as shown below:

Press Next. In the Feature license dialog, accept the license agreement then press Next. Press Next again, the following Installation dialog will appear:

Press Finish to download the Tuscany plugins. The download is about 35Mb as it includes the Tuscany runtime plus all the dependencies for all the SCA implementation types and bindings supported by Tuscany. But I found having everything in one download more convenient as it saves you from having to solve the JAR dependency puzzle and fish for all the required JARs...

After the download completes you should see the following dialog:

You're now ready to install the Tuscany plugins. Press Install All, the plugins will be installed:

Restart your Workbench.

You're now equipped with a tool to help you write, test, and run an SCA application in Eclipse!

To show how to use the tool let's develop a simple HelloWorld application with an SCA Java component providing a Service with two bindings, a Web Service binding and a JSONRPC binding. It should just take about 5 minutes :)

First let's create a Java project. From the Workbench menu bar select New - Java Project. In the New Java Project dialog, enter a name for the project as follows:

Press Next. In the Java Settings dialog, select the Libraries tab.

Now you need to add the Tuscany library to the project. Click Add then in the Add Library dialog select the Tuscany library as follows:

Press Next then Finish. The Tuscany library should be on your project's build path, as shown here:

Press Finish to create the Project.

Now let's create a Java interface and a Java implementation class for our Hello SCA component.

Right click on the src folder, then select New - Interface. In the New Java Interface dialog, enter
Package: hello
Name: Hello

Press Finish to create the Hello interface. The Java editor will open, add the following method to the Hello interface:

String hey(String name);

Press Ctrl+S to save the file.

The editor and the contents of the file are shown here:

Now let's create the HelloImpl Java implementation class. Right click the Hello interface, then select New - Class. In the New Java Class dialog, enter
Name: HelloImpl
Make sure that hello.Hello is listed in the list of interfaces.

Press Finish to create the Hello interface. The Java editor will open, add the following code to the HelloImpl.hey(...) method:

return "Hey " + name;

Press Ctrl+S to save the file.

The editor and the contents of the file are shown here:

Now let's turn the HelloImpl class into an SCA component. To do that we need to create an SCA Composite file describing the component its services and their binding configuration.

From the workbench menu bar, select New - Other, in the New Wizard dialog select the Tuscany - Composite wizard, as shown here:

Press Next, you're about to create your first SCA composite :) In the New SCA Composite dialog, make sure that Hello/src is selected as parent folder then enter,
File name: Hello.composite

Press Finish, Hello.composite will open in the XML editor. You should be able to use code assist in the XML editor to edit your SCA composite, as shown below:

Add the following to your SCA composite:

<component name="Hello">
<implementation.java class="hello.HelloImpl"/>
<service name="Hello">
<binding.ws uri="hello-there"/>
<t:binding.jsonrpc uri="hello-nicer"/>
</service>
</component>

This declares an SCA component implemented by the HelloImpl class, providing our little Hello Service over two bindings, Web Service (SOAP) and JSONRPC.

Press Ctrl+S to save the file.

The editor and the contents of the file are shown here:

That's it for coding! Let's try to run this SCA Composite now...

It's pretty simple. Right click Hello.composite then from the popup-menu select Run As - Tuscany.

The Tuscany Runtime should start the Hello composite and you'll see the following in the Console view:

Your SCA composite is now up and running, providing a Hello Web Service at http://localhost:8085/hello-there and a JSONRPC service at http://localhost:8080/hello-nicer.

You can click these links to check that the services are running OK and get their WSDL and SMD descriptions, as shown below:

The WSDL describing the Web Service:

The SMD describing the JSONRPC service:

That's all for today... If you got to the end of this post, I hope you'll find this little Tuscany tool useful.

How to use Apache Tuscany with WebSphere 6.1

2007-10-02T11:08:00.001-07:00

Recently a number of Apache Tuscany users started to use our Tuscany SCA 1.0 release on top of WebSphere v6.1 and ask questions about it on the tuscany-user list.

To be able to help I had to set up WebSphere v6.1 on my Laptop and configure it to work with Tuscany. It was pretty straightforward so I thought it'd be useful to share the steps here:

First install WebSphere v6.1.0.9. The "9" indicates WebSphere Fixpack 9, which contains a Fix for a WebSphere bug with Servlet filters... needed as Tuscany uses a Servlet filter to dispatch HTTP requests to SCA services.

Start your WebSphere server, then point your Web browser to the WebSphere console at http://localhost:9060/admin.

To activate that WebSphere Fix, navigate to Application Servers - server1 - Web Container - Customer Properties and configure a com.ibm.ws.webcontainer.invokefilterscompatibility property to true as described there.

Here's a screen capture showing what to do:

Download the Tuscany SCA release 1.0 binary distribution from http://incubator.apache.org/tuscany/sca-java-releases.html and untar or unzip it.

The distribution contains a number of ready to use samples. Let's start with samples/calculator-webapp.

This is a sample Calculator implemented with SCA components wired together, implementing the basic add, subtract, multiply, divide operations (in Java), and a JSP invoking the calculator.

Here's a diagram showing the SCA composite and how the SCA components are assembled together:

The sample README describes how to build the WAR file for the Webapp using Apache Ant, but the distribution already contains a WAR file ready to use, let's try to deploy it to WebSphere...

From the WebSphere Console page, navigate to Applications - Enterprise Applications. You should see the following page:

Click on the Install button, in the application installation form click Local file system - Browse and select:
samples/calculator-webapp/target/sample-calculator-webapp.war.

In the Context root field, type sample-calculator-webapp as shown below:

Click Next to proceed with the installation, the following page will appear:

You don't need to change any of these settings, just click Next, the following page will appear:

Click Next, the following page will appear:

Click next... again :) the following page will appear:

Click Finish, installation will proceed and you should see the following page:

Now click Save to save the changes and make the installation effective. This will go back to the list of installed applications, you should now see sample-calculator-webapp, as shown here:

At this point the Calculator webapp has been installed. We just need to make a small change to its classloader configuration. This is necessary to allow Tuscany to find all its dependencies in the webapp itself instead of the WebSphere lib directory (as some of the Tuscany dependency JARs are newer than the ones that come with WebSphere).

To do that click on sample-calculator-webapp war in the Enterprise Applications list, you should see the following page:

Click the Class loading and update detection link, you should see the Class Loader configuration page. Select the "Classes loaded with application class loader first" and "Single class loader for application" options as shown here:

Click OK, then Save to save your changes. You should be back to the Enterprise Applications page.

You're now ready to start the Calculator sample!

Check the check mark beside sample-calculator-webapp war and click the Start button, you should see the following:

The application is running... time to point your Web browser to http://localhost:9080/sample-calculator-webapp/. The Calculator JSP will execute, and you should see the following:

That's it, you've run your first SCA application with Tuscany on WebSphere v6.1!

Note: That last screencap shows localhost:8080 instead of localhost:9080 but that's just because I've tweaked my server to use 8080 instead of the default 9080, I wanted the different servers on my laptop to use the same port number as having to juggle with different ports was driving me crazy :)

Last time I checked all samples from the Tuscany 1.0 release worked on WebSphere v6.1 (6.1.0.9 or later) except for our helloworld-dojo-webapp sample.

If I find more time to blog in the next few days I'll try to describe some of these other samples and also quicker ways to deploy an SCA application using a script instead of clicking... if I find the time :)

2007-09-25T21:07:00.002-07:00

I've been busy the last few weeks, we just released Tuscany SCA 1.0. Here's the announcement...

The Apache Tuscany team is pleased to announce the 1.0 release of the Java SCA project.

Apache Tuscany provides a runtime environment based on the Service Component Architecture (SCA). SCA is a set of specifications aimed at simplifying SOA application development. These specifications are being standardized by OASIS as part of the Open Composite Services
Architecture (Open CSA).

This Apache Tuscany release represents a major milestone as the first 1.0 implementation of the core SCA specifications, including:

Assembly Model v1.0
Policy Framework v1.0
Java Common Annotations and APIs v1.0
Java Component Implementation v1.0
Spring Component Implementation v1.0
Web Services Binding v1.0
EJB Session Bean Binding v1.0
BPEL Client and Implementation v1.0 (experimental)

It also includes implementations of features not yet defined by SCA specifications, including:

JSON-RPC and Direct Web Remoting bindings
RSS, ATOM, and HTTP bindings
RMI binding
Pub/Sub notification binding
XQuery component implementation
Scripting component implementation supporting Groovy, Javascript, Python and Ruby
OSGi component implementation and support for SCA contributions packaged as OSGi bundles
databindings for Service Data Objects (SDO), JAXB, XmlBeans, Axis2's AXIOM, JSON, Saxon, DOM, SAX and StAX

The Tuscany SCA Runtime can be configured as a single node SCA domain or as an SCA domain distributed across multiple nodes.

In addition Tuscany SCA supports the following host-deployment options:

running standalone
running with embedded Jetty or Tomcat
running with distributed nodes across multiple JVMs
running as a module in Geronimo (experimental).
running as a standard Web application

The Webapp samples included in this release have been tried and tested on Tomcat and WebSphere 6.1.

For full detail about the release see the RELEASE_NOTES and CHANGES files.

To download the distributions or for more information about the release please go to: http://incubator.apache.org/tuscany/sca-java-releases.html

To find out more about OASIS Open CSA go to: http://www.oasis-opencsa.org.

Apache Tuscany welcomes your help. Any contribution, including code, testing, contributions to the documentation, or bug reporting is always appreciated. For more information on how to get involved in Apache Tuscany visit the website at: http://incubator.apache.org/tuscany.

Thank you for your interest in Apache Tuscany!
The Apache Tuscany Team.

---

Tuscany is an effort undergoing incubation at the Apache Software Foundation (ASF), sponsored by the Apache Web services PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.

Apache Tuscany Java SCA 0.99 released

2007-08-31T22:38:00.000-07:00

I've not blogged for a while as I've been busy working on Apache Tuscany... We just got our 0.99 release out.

Here's the announcement:

The Apache Tuscany team is pleased to announce the 0.99-incubating release of the Java SCA project.

Apache Tuscany provides a runtime based on the Service Component Architecture. SCA is a set of specifications aimed at simplifying SOA Application Development which are being standardized at OASIS as part of Open Composite Services Architecture (Open CSA).

This release of Apache Tuscany SCA builds on the stability and modularity established with the previous releases and includes more complete implementation of SCA specifications, support for distributed SCA domains, SCA policy, OSGi implementation types, and pub/sub support with notification components. Start up time and memory footprint of the runtime has been reduced and there have been numerous bug fixes.

This is expected to be the last point release before the 1.0 final release.

For full detail about the release see the RELEASE_NOTES and CHANGES files.

To download or for more information about the release go to: http://incubator.apache.org/tuscany/sca-java-releases.html

To find out more about OASIS Open CSA go to: http://www.oasis-opencsa.org.

Apache Tuscany welcomes your help. Any contribution, including code, testing, improving the documentation, or bug reporting is always appreciated. For more information on how to get involved in Apache Tuscany visit the website at: http://incubator.apache.org/tuscany.

Thank you for your interest in Apache Tuscany!
The Apache Tuscany Team.

---
Tuscany is an effort undergoing incubation at the Apache Software Foundation (ASF), sponsored by the Apache Web services PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.

Ubuntu - Linux for human beings

2006-11-22T18:39:00.000-08:00

Yesterday I had to install Ubuntu on my laptop. Ubuntu is a pretty popular Linux distribution, also known as "Linux for human beings" because it's supposed to be easy to use.

I am working on the Apache Tuscany project and somebody had opened a bug report complaining about build issues on Ubuntu so I wanted to test it and see what was going on.

The part of the Tuscany runtime that I'm working on is written in C++, our Linux build uses the usual automake and configure tools and is very portable but I was curious about any portability issues in our code since most of the Linux people on the Tuscany project build on Fedora and Ubuntu is a different Debian based distribution.

After two unsuccessful attempts to use pre-packaged Ubuntu VMWare images I decided to bite the bullet and do a fresh install of the Ubuntu 6.10 Desktop distro CD image (which you can download here) on a spare partition on my laptop.

Well... I must say that I am amazed by the quality of this distribution!

It installed in about 20 minutes, only asked me about my language, keyboard and timezone, detected all the devices on my Thinkpad T42, and came up with a Gnome desktop all loaded with everything you need on a desktop machine... Firefox, Evolution, Open Office, Totem for movies and Rythmbox for music etc.

I have been working on Linux for some time now, and was thinking: Yeah Ok, nice try, but what about getting wireless to work? what about suspend/resume? and don't even think of trying to hibernate... Well... all of this just works! and configuring the wireless access to my home network was even simpler than on my daugther's Mac-OS X, and way more simpler than on my other Windows machine, which keeps losing its wireless settings all the time!

Ubuntu still failed the absolute test for me: Plug an iPod into the USB port and wait to see if it's recognized and a nice music player pops up... I tried and nothing happened, except some cryptic error messages in the system logs... I guess I can live with that :) I've not seen any other Linux distribution pass this test anyway without some serious fiddling with the Linux kernel or udev configuration.

After recovering from my disappointment with the iPod, I finally decided to test our Tuscany build in silence, with no music, and here again I was really impressed... I was able to install all the development tools I needed in a matter of minutes:

the build_essential Ubuntu package
the automake tools
a Subversion client
Eclipse
Apache Axi2C
Apache Ant
the Java 5 JDK
and the Ruby and Python development libraries...

... and with just a few mouse clicks, thanks to the nice Synaptic package manager included in the distro.

Conclusion: Ubuntu is not just a Linux for human beings, it's also good for software developers!

Oh, and Tuscany builds and works great on Ubuntu 6.10 by the way, so I eventually closed the bug report :)