We just released some new features for Speeqe, a simple group chat application for the Web.  This release includes private room messages, external MUC support, early access for interested people to host their own rooms, external Jabber authentication, and image inlining.

Please check it out.  We’re looking for people interested in starting their own Speeqe rooms.  If you’d like your own space, just email us, and we’ll make it happen.

Jabber Bugs

We use negative presence priority to keep the Speeqe client from stealing your normal private messages.  Unfortunately, if you set a negative priority on your resource, both Jabberd 2.x and the Google Talk server stop sending you presence of any kind.  This means that you never get the presence status of people in a chat room, not even intiially.  This is a pretty big bug for us, and if anyone has any suggestions on workarounds while we try and work with Google and smoku to get it resolved we’d love to hear them.

External MUCs

As we demoed in Portland at the 5th XMPP Summit, you can now use Speeqe with external MUC rooms.  Just substitute the address of the room in this URL: http://www.speeqe.com/room/ADDRESS/ .

After you enter a room for the first time, you can log in using any federated Jabber account (including Gmail).  Once you do this once, any room you join will reuse the same information.

UI Love

The UI is a work in progress.  We know some things are currently a bit ugly, and we’re working hard to address them.  In particular this release was more about technical features than UI design.  The next release should have more UI design elements.

Everything in the room is themeable.  In a release or two, we will open up the theme creation tools to everyone.  Themes are just HTML pages, and Speeqe uses CSS selectors to decide where to fill in content.  It can already do quite a lot – see the demo themes for an example.

My father-in-law grows corn and soybeans in rural Minnesota.  I’m currently spending a week on this farm visiting with my in-laws and trying not to work too hard (ha!).  Yesterday I helped empty a grain bin into a semi trailer for sale at the local grain elevator.  For me, this whole task was very like system administration. It was a lot of easy but repetitive labor, and every few minutes I came up with some idea to reduce the amount of work I would have to do.

An Example

The grain bin has a flat bottom with a hole in the middle.  When the bin is full, grain runs through this hole like sand in an hourglass, is transported horizontally by an auger, and finally is dumped into the trailer via another auger.  At a certain point the grain will no longer be pulled into the hole by gravity alone, and so there are secondary holes all the way along the auger.  Even these holes will not allow the bin to fully empty, so there is a rotating auger in the bottom that sweeps around moving the last bits of grain to the holes.   Even with that sweeping auger, it is still necessary to get in there with a broom and shovel and push the grain to the hole in order to fully empty the bin.

Some Automation Solutions

After a few minutes of this, it seemed to me that the ridges in the floor should be radial so that sweeping grain to the middle is easier.  The straight ridges there now made it easy in some directions and really difficult in others.

Next, I thought that an inverted cone design would reduce the manual labor to almost nothing and remove the need for multiple holes.  Combined with the radial ridging and some kind of vibration, I think you could get all but a few stray grains into the hole without ever getting into the bin.

Inventing Is Like Perl

It finally made sense to me in a visceral way why farmers are such prolific inventors.  System administrators try to reduce their tedious labor to zero;  farmers want to do the same.  No one wants to spend all their time typing the same commands or shoveling piles of grain.  The classic tool of sysadmins for this is Perl, but until robotic farming is more widespread, farmers will have to settle for repurposing or inventing physical solutions.

Just like Perl, some of these solutions are elegant, like the auger, and some are complete disasters.

Like thousands of others, I am playing around on Identi.ca today.  Microblogging is a hot topic especially as Twitter’s growth problems have provided fodder for the armchair scalability experts.  We’ve also been thinking about microblogging in the context of Speeqe.  I’m not sure where our experiments will lead, but you can see our initial endeavor as the Tweeqe theme over at the Speeqe demo site.

Our original idea was very simple.  We wondered what would happen if we moved the text entry box to the top of the page and made new content post at the top.  This mimics both Twitter and most blog and blog commenting systems.  It gives group chat a completely different and fresh feel.

Just changing those simple things has gotten us thinking about group chat in new contexts, like microblogging.  I was pretty surprised to see Dave Cridland wish for a similar thing in his blog post about Identi.ca today:

More interesting would be to model identi.ca as a MUC service, wherein each user has control of a MUC room that is populated by the people they subscribe to. That would give a history interface, as well as a “last status” display, almost for free.

Here he has outlined a private group chat room filled with the people that interest you.  With Speeqe, you’d create a room, then use a Pubsub service or an Atom importer to bring in your feeds for example.  The other way works as well.  Why not have your microblog be a group chat room?  Just create a Speeqe room with the Tweeqe theme and start posting.  After all, what is Twitter but a sort of on-the-fly group chat?

These ideas are still young and need nurturing, but we’d love to hear feedback from the community.  What are your new ideas for group chat?

In order to build XMPP applications inside web browsers, the XSF created the Bidirectional-streams Over Synchronous HTTP (BOSH) specification.  This document specifies the interface for a server side connection manager which holds your XMPP connections open and uses Comet to mimic a two way pipe to the browser.  Having used BOSH-based clients as my primary Jabber interface for almost two years, I have come to believe that BOSH leads to more reliable clients.

BOSH Hides An Unreliable Connection

BOSH typically holds your connection for a few minutes while there is no activity.  The great thing about this is that even if the underlying Internet connection is unreliable, your BOSH connection will live on.

I’m currently writing this post in rural Minnesota where my Internet connection goes up and down once every 10 minutes or so.  I have not been able to keep iChat (either AIM or Jabber) connected all day, but my Speeqe client disconected only once.  Of course, the Speeqe client is still in heavy development, so I’m not sure if that was related to the spotty connection.

The practical application of this is that I can sleep my laptop and go to a new location without being disconnected, as long as I open it back up before the server times out my BOSH session.  It will also gracefully handle the case where I switch out one WiFi connection for another.

BOSH Does Not Lose Data

I’m not sure if other BOSH servers support this, but Punjab will cache and resend request responses. If one of the responses fails to reach the client, the client will get the same data back after replaying the request.  This means that even though I switch IPs or repeatedly lose connectivity, I never lose any data.  The worst thing that has ever happened on my BOSH sessions is chats have arrived out of order, but even this occurrence is rare.

This is a lot different from a straight TCP connection.  If I lose connectivity while chatting in iChat, I have no idea if I missed part of the conversation, and the server cannot replay data to me from a known point.  I usually have to ask people to paste the chat history starting from the last message I know I received.

BOSH Streams Are Compressed For Free

Since most web servers and proxies compress outgoing text data with GZip, BOSH streams can get this compression for free in the server-to-client direction.  This means faster data transfers over slow links without having to implement or worry about compression in the client.

Unfortunately in the Web browser case, sending data to the BOSH server compressed is not automatic, and implementing GZip in JavaScript may not be a workable solution. The compression benefit is only one sided due to these constraints.  Amusingly, this creates a bandwidth asymmetry very similar to cable and ADSL connections.

How many TCP based XMPP clients currently support stream compression?  I think not many.  In contrast, all current BOSH clients have at least one sided stream compression out of the box.

BOSH Is Firewall Friendly

Since BOSH operates over HTTP, it works in many places that direct TCP connections do not.  I have stayed in several hotels which blocked long lived TCP connections of any type, and many corporate environments do not let port 5222 through the firewall.  BOSH gets around all of this quite nicely.

The Downside of BOSH

These benefits come with a small price, and part of that price is the added latency of involving the HTTP protocol.  BOSH in the browser is a worst case for this latency due to connection limitations, but even here performance is pretty good.  We’ve had millions of real-time chess games played over BOSH, and the experience is very close to the same as the native Chesspark client.

This latency is further reduced if HTTP pipelining is used, or if your application can use a BOSH hold value larger than 2 which is the limit of most browsers.

The other part of the price is the extra bandwidth consumed by the overhead of HTTP requests.  Tuning the client and Web server to avoid useless HTTP headers can remove a lot of these issues, as can the aforementioned stream compression.  It would make Strophe a lot more efficient if Web browser headers could be controlled from the JavaScript engine.

Native Clients and BOSH

It would be interesting for native clients to support BOSH as a transport method alongside direct TCP connections.  This would reap the benefits of firewall friendliness and a more reliable connection if Internet connectivity is poor.  Since non-browser clients do not suffer the arbitrary restrictions on HTTP connections, most of the latency problems can be eliminated, and it would be easy to make stream compression work in both directions.

We’ve created a Twisted BOSH client interface for testing Punjab and various Chesspark services, and if people are interested, I can write more about this and publish the code.

Get BOSHing

What are you waiting for?  Go read the BOSH specification in XEP 124.  Go write an interesting XMPP Web application with Strophe.  Go check out a BOSH server like Punjab.  Go run BOSH in the cloud.

I’ve been working furiously over the last week to prepare a 1.0 release of Strophe. For the C library Doxygen was an easy choice.  The markup is based on Javadoc, and the tool just works.  For JavaScript, there are few choices, and most of the choices are just terrible.

Documentation Styles

There are two ways to do in-code documentation.

The first way is implicit. The tool parses the code, finding which things are functions and which things are classes, and then grabs comment blocks located near these structures.  This method requires detailed knowledge about the programming language.

The second way is explicit.  The tool requires the documentation author to notate what is what.  This requires little information about the programing language since most languages share syntactic structures like functions, classes, variables, and so on.  The generating tool needs to know only what comments look like.

Some tools support one of these methods, and some support both.  Ideally they would all support explicit documentation, but also help you make the documentation more concise by figuring out what you are documenting automatically.  Keep these distinctions in mind while reading about the tools below.

The Choices

DoxygenDoxygen does not support JavaScript, and it is not clear why, other than no one has gotten around to it.  It must be relatively hard to do because the current workaround involves a non-trivial Perl script to translate JavaScript source into pseudo-C++ code which is then fed to Doxygen.  I tried to find a way to download this script but failed, and I was not willing to cut and paste from the browser or spend time trying to find and check out the subversion repository since the script is not in the distributed source.

It seems weird that it does not support any language it knows the comment syntax for, since it allows many kinds of explicit documentation.  Maybe the authors will fix this omission someday.

JSDoc.  Next up was JSDoc, which also has Javadoc style syntax.  Out of the box on Mac OS X, JSDoc segfaults.  It seems to be related to recursion depth of the regular expressions, and installing a Perl from source worked around the issue.  The problem with JSDoc is that it assumes a lot about how you write the code and allows no explicit documentation.  It cannot detect constructors inside JSON style object definitions and doesn’t seem to deal well with prototypes.  If you program in the style it supports, it probably works great, otherwise you are screwed.  In addition to this, JSDoc produces the worst HTML output I’ve ever seen.  I think it could give early versions of FrontPage some competition in this regard.  Don’t take my word for it though; just look at their example.  It has all the text content inside of <ul> tags!

HeaderDoc.  Apple has written a tool called HeaderDoc which does documentation for several languages, including Objective-C and JavaScript.  I believe this is because it is more in the explicit camp, but I haven’t looked at the code. This tool is actually pretty neat, but the output is not that great.  It includes the entire function in the documentation for example.  It’s simple, and it works, but not much else.

Naturals DocsNatural Docs output looks pretty nice out of the box.  It supports a number of languages including JavaScript.  It has two levels of language support.  First, you can tell it what comments look like and notate your code explicitly.  It also has full implicit support for some languages, although JavaScript is not one of them.  It unfortunately has Javadoc style comment support only when in full implicit mode.  This is not that big of a deal since its native commenting style is simple and works well.

Natural Docs For The Win

In the end, the choice is simple for JavaScript.  HeaderDoc and Natural Docs are the only two that work in the general case.  Natural Docs is the clear winner on quality of output alone, but it also has a lot of other qualities that are great.  It is configured by simple text files, and it is easy to define support for new languages.  This gives it a huge advantage over tools like Doxygen or HeaderDoc which seem a lot harder to work with if you aren’t starting in a supported language.

Natural Docs is not without its issues, but they are mostly minor.  The indexes for grouped documentation don’t seem to be generated correctly, but I can forgive this since most of the other tools had no indexes at all.  It also can’t document part of a class in one place and part in another unless you don’t mind the documentation also showing up in two locations.  I think both of these problems could be easily fixed if they bothered you.  I was able to work around the latter easily with very minor code tweaks.

An Example

You can see the Strophe preview documentation to get an idea of what the output look like.  Here is what the input looks like for a typical function.

/** Constructor: Strophe.Connection
* Create and initialize a Strophe.Connection object.
* Parameters:
* (String) service - The BOSH service URL.
* Returns:
* A new Strophe.Connection object.

I think the output and tool are nice enough to use that I will probably convert the C version of Strophe to Naturals Docs as well, instead of using Doxygen.

Hopefully this inspires some people to document their code better.  I’m sure I’m not the only one that could improve in this area.

Three years ago, we released libstrophe to the world. libstrophe is a cross-platform C library for writing XMPP clients. libstrophe exists because none of the C libraries at the time worked well on Windows platforms. A year or so later, I discovered the potential of XMPP on the Web and created a JavaScript version as a first step to a Web based Chesspark client. I never officially released strophe.js, but it is available under the GPL license for anyone who was brave enough to dig it out of the Chesspark JavaScript code. It’s time to make it official.

Some Features Of Strophe

Strophe has a lot of neat features.  Probably the most compelling is that it has been well tested for 2 years in all major browsers.  It catches many errors and includes optimizations that no other AJAX framework seems to.  It has been used by hundreds of thousands of people via the Chesspark site.

Strophe was developed in a climate where low latency is critical.  People do not enjoy waiting for game moves or clock updates, and so Strophe does its best to deliver excellent performance.

Strophe is very simple.  Not only is the API small and easy to understand, but there are numerous helpers in there to make life easy.  Strophe.Builder helps you build stanzas quickly.  For example:

$iq({type: 'get'})
    .c('query', {xmlns: Strophe.NS.ROSTER})

Will generate this XML stanza:

<iq type='get'>
<query xmlns='jabber:iq:roster'/>

Strophe In Action

If you’d like to see what is possible with Strophe, please check out the Chesspark Web client and the Speeqe demo site.

Take a Sneak Peek

I’m almost finished preparing Strophe 1.0 for release, which includes the original libstrophe C library, the strophe.js JavaScript library, and complete documentation for both.  I hope to make Strophe the best XMPP library around for native or web development.

Until it is released, you can can see a preview of the JavaScript version or read the API documentation.  This version was refactored and made more modular compared to the Chesspark version, and it hasn’t been fully tested yet.  It does however work great in my initial tests with a slightly modified version of the Chesspark Web client.

I’m very interested in any feedback the community has for Strophe.

My wife had a Powerbook G4 which she loved for years.  She refused to upgrade because the Macbook and Macbook Pro designs had too big of a footprint for her.  When the Macbook Air came out, she hesitated, but decided the Air was close enough to what she wanted to warrant an upgrade.  Months later, she really regrets this.

Most people complain about the lack of large harddrive or optical media reader, but on these topics I think Apple’s product decisions make some sense.  Here are some lesser known issues.

The Macbook Air is back heavy.

If you sit it open on a flat surface, most of the weight is in the rear.  This is terrible for a laptop.  My wife finds that everytime she removes her hands the whole thing almost tumbles off her lap.  It would have been better to have the thick part in the front, but I suppose this doesn’t make for as nice of a demo.

Apple apparently does not have any parts in stock.

If your Macbook Air breaks, you are looking at a long wait for a repair.  In my experience, typical turnaround times for Apple on laptop repairs are less than a week, usually around 4-5 days.  The wait time for the Air appears to be somewhere north of 2 weeks.

My wife’s machine’s video card went haywire, and she dropped it off at the local Apple store for repair on June 12.  It has still not come back, although the status indicator finally claims it is now fixed.  A friend who works at Apple told me her co-worker’s Air broke and they experience similar wait times, even though Apple fast-tracks employee repairs.

Four or five days without a machine is liveable for most, but two weeks is a deal breaker.  Mac hardware has a good reputation, but it still breaks.  Every Apple laptop I’ve owned has been repaired at least once during its lifetime, and all the iBooks I had went in multiple times.  The only exception is my wife’s Powerbook G4, which still hums along perfectly.

So if you plan for the Air to be your primary machine, you best keep your old machine around as a backup.

Updated: Corrected the date the Air was dropped off for repair.