One of the biggest benefits to using ejabberd is the ease with which it is possible to extend the server’s functionality. After only a week of production operation, I’ve already written and deployed four new modules at Chesspark. Let’s get our hands dirty, fire up an editor, and learn how to write a new ejabberd module. Unlike most tutorials, the module created here is a real piece of code that we have running in production at Chesspark.
Getting Started
The code presented here is in Erlang, which is what ejabberd is written in. If you are unfamiliar with Erlang, you will probably still be able to follow along just fine. If you have a question about Erlang syntax or an Erlang function, the Erlang Web site has tutorials and reference documentation for your enlightenment.
Anders Conbere has written a series of excellent tutorials on writing ejabberd modules at his blog: building ejabberd, writing a generic module, writing an HTTP module, and writing an XMPP bot module. I will assume that that you have read at least his tutorial on generic modules.
Finally, there is some incomplete documentation about developing ejabberd modules in the ejabberd wiki.
Stopping Storms With A Little mod_sunshine
During our ejabberd migration, we ran into a bug in our desktop client that caused it to send the same presence stanza over and over in an infinite loop. While providing an updated build of the client solved the problem for many of our users, some users refused to upgrade. Not only did this cause a lot of undue stress on our XMPP server, it also affected all of the broken client’s user’s online contacts. These presence storms were silently degrading the service quality for nearly everyone.
We decided that the best solution would be to detect presence storms and disconnect the offending clients. This required writing a fairly simple ejabberd module that I named mod_sunshine. We’ll recreate the same code here.
mod_sunshine’s Humble Beginnings
Every ejabberd module implements the gen_mod behavior. An Erlang behavior is just a set of functions that a module implementing the behavior is required to support. The gen_mod behavior requires two functions: start/2 and stop/1. These functions are called when ejabberd starts and stops the module.
The skeleton of an ejabberd module appears below.
-module(mod_sunshine).
-behavior(gen_mod).
-export([start/2, stop/1]).
start(_Host, _Opts) ->
ok.
stop(_Host) ->
ok.
For those of you wondering, the preceding underscores on the function arguments signal to the Erlang compiler that these arguments are unused.
This code should be placed in a file called mod_sunshine.erl. Assuming you have Erlang and ejabberd somewhere on your system, you can compile mod_sunshine.erl with this command: erlc -I /path/to/src_or_include -pz /path/to/src_or_ebin mod_sunshine.erl.
This module can be added to the server by adding a line like the one below to the server configuration. You’ll also need to place the module’s .beam file in the ejabberd ebin directory alongside the rest of ejabberd’s .beam files.
{mod_sunshine, []}
Hooks, Hooks, and More Hooks
ejabberd modules extend functionality of the server by connecting themselves to hooks. A hook is just a place in the code where ejabberd offers a connection point. There are hooks for when a user sends a packet or receives a packet, for when sessions are created or removed, and for many other things. A list of hooks can be found in the ejabberd events and hooks documentation.
Each hook in ejabberd is associated with a chain of functions to execute. ejabberd modules will add functions to these chains, and when the hook is triggered, the chain of functions is executed. Each function has a priority in the chain, and ejabberd will execute them in order. Additionally, it is possible for a function to terminate the hook processing so that later functions in the chain do not get executed.
ejabberd modules typically add some functions to one or more of these hook chains and then wait around for them to be executed. This is very similar to GUI code where one responds to input events, only instead of mouse clicks and keyboard presses, modules will respond to packets being sent or users coming online.
Just to make sure you fully understand the hook processing and the function chains, let’s go through an example.
Passing Around Offline Messages
When a message comes into an ejabberd server for a user that is currently offline, ejabberd executes the chain of functions for the hook offline_message_hook.
The ejabberd session manager adds a default function to the offline_message_hook chain that bounces the incoming message with a service unavailable error. This hook is added at the lowest priority so that it executes after anything else in the chain.
The mod_offline module, which comes with ejabberd, uses this same hook to add support for storing offline messages in a database. These messages are then sent to a user when they next come online. It does this by adding a function to the chain at a higher priority than the session manager’s. When a message gets sent to an offline user, this function executes first and stores the message in the database. Since it would be silly to return an error now that the message has been stored, mod_offline‘s function signals to the hook processor that no more functions should be run in this chain. The session manager’s default function will never be executed.
At Chesspark we generate a lot of messages that are meaningless if they are not delivered to someone while they are online. We built a module that filters offline messages and discards those that are inappropriate for database storage. This works by adding a function to the offline_message_hook chain at a very high priority. This means that our function is the first function to receive the message sent to an offline user. If the module determines the message is inappropriate for storage, we signal to the hook processor that it should not continue executing chain functions, silently dropping the message. Otherwise, the module let’s the rest of the chain execute normally, which means that mod_offline, which is next in the chain, will receive the message and store it.
Hooking Presence For mod_sunshine
Now that we know a lot about hooks and hook processing, it’s time to put it to some use in mod_sunshine. First, we have to know which hook to use.
Unfortunately, none of the documentation is very clear about which hooks are executed at what times. I have found that experimentation is the only way to figure this out. Use the name of the hooks to make a guess, and then use some logging statements to figure out if that hook gets run when you expect. You can also search for the hook’s name in the ejabberd source code to find out when it is executed and what is expected of the functions in the chain.
For mod_sunshine, the hook we are interested in is set_presence_hook. This hook is processed whenever a connected user sends a presence stanza to the server. It’s common to add a function to the chain in start/2 and remove the same function in stop/1, and that’s what we’ve done below. Additionally, we must export the function we wish to add.
-module(mod_sunshine).
-behavior(gen_mod).
-export([start/2, stop/1, on_presence/4]).
start(Host, _Opts) ->
ejabberd_hooks:add(set_presence_hook, Host, ?MODULE, on_presence, 50),
ok.
stop(Host) ->
ejabberd_hooks:delete(set_presence_hook, Host, ?MODULE, on_presence, 50),
ok.
on_presence(_User, _Server, _Resource, _Packet) ->
none.
A function in the set_presence_hook chain takes four parameters: the user, server, and resource of the person that sent the presence stanza as well as the actual presence stanza. It is required to return none.
A Module That Does Nothing Is Hardly A Module At All
So far we have a module that doesn’t almost nothing. It adds a function to the set_presence_hook chain when it is loaded, removes the function from the chain when it is unloaded, and does absolutely nothing when the chain is executed. It’s time to get to the meat of the module.
First, mod_sunshine needs to know what a presence storm is. We can define this as a user sending the same presence stanza count times in interval seconds. Since we might not know what the optimal values are for count and interval, it is best to leave those as options to be defined in the ejabberd configuration file.
Next, the module needs to store its state somewhere so it can check if a user is sending a presence storm. Since ejabberd passes no state to on_presence/4, we must keep track of this ourselves. We’ll do this with mnesia, one of Erlang’s built in databases.
Finally, the module needs to disconnect any user who causes a presence storm. Unfortunately, ejabberd does not seem to have an API or message for this out of the box, so we will have to add this ourselves to the internal ejabberd c2s module.
Intermission And Logging
It’s time for a small break before we dive into the real code, so let’s digress for a second and talk about logging.
It will probably happen in your Erlang programming career that something goes wrong and you can’t figure out what that something is. A time honored tradition exists for exactly these situations – printing some information to a log file.
Should you need to do this, ejabberd has several macros for writing to its log file at different logging levels. They are defined in ejabberd.hrl which you will have to include in the code to use them. I usually prefer to use ?INFO_MSG. These functions take a string followed by a list of arguments. Be sure to remember that you must pass an empty list if there are no arguments.
Here’s mod_sunshine.erl from before with modifications to announce its startup and shutdown.
-module(mod_sunshine).
-behavior(gen_mod).
-include("ejabberd.hrl").
-export([start/2, stop/1, on_presence/4]).
start(Host, _Opts) ->
?INFO_MSG("mod_sunshine starting", []),
ejabberd_hooks:add(set_presence_hook, Host, ?MODULE, on_presence, 50),
ok.
stop(Host) ->
?INFO_MSG("mod_sunshine stopping", []),
ejabberd_hooks:delete(set_presence_hook, Host, ?MODULE, on_presence, 50),
ok.
on_presence(_User, _Server, _Resource, _Packet) ->
none.
Ok, break time is over; let’s get back to the real stuff.
Dealing With Options
To pass options to mod_sunshine we have to change the ejabberd configuration file. The line we used above should be modified to the one below.
{mod_sunshine, [{count, 10}, {interval, 60}]}
This will tell mod_sunshine that a presence storm is anyone sending 10 copies of the same presence stanza within 60 seconds. Now we just need to get these options into the code.
Alert readers will notice that while start/2 is passed the options in the Opts variable, the on_presence/4 function does not receive these. How are we to get the options?
Thankfully, gen_mod has an API function to fetch the module options from the configuration – gen_mod:get_module_opt(Host, Module, Opt, Default). This function needs the virtual host the options are defined for. This just happens to be the contents of the Server variable in on_presence/4. gen_mod:get_module_opt/4 also lets you define the default option values.
Adding options to mod_sunshine is now easy.
-module(mod_sunshine).
-behavior(gen_mod).
-include("ejabberd.hrl").
-export([start/2, stop/1, on_presence/4]).
start(Host, _Opts) ->
?INFO_MSG("mod_sunshine starting", []),
ejabberd_hooks:add(set_presence_hook, Host, ?MODULE, on_presence, 50),
ok.
stop(Host) ->
?INFO_MSG("mod_sunshine stopping", []),
ejabberd_hooks:delete(set_presence_hook, Host, ?MODULE, on_presence, 50),
ok.
on_presence(_User, Server, _Resource, _Packet) ->
%% get options
StormCount = gen_mod:get_module_opt(Server, ?MODULE, count, 10),
TimeInterval = gen_mod:get_module_opt(Server, ?MODULE, interval, 60),
none.
Persisting State With Mnesia
Mnesia is easiest to use when paired with Erlang records. For the Erlang beginners, Erlang records are like C structs. They are normal Erlang tuples with named fields. We’ll need to create a record for mod_sunshine to store the user, packet, the time the packet was originally sent, and the number of times the packet has been sent. Here’s what that looks like.
-record(sunshine, {usr, packet, start, count}).
The name of the record is sunshine. Note that the first of the four fields is not a typo of ‘user’. It is an acronym for ‘user, server, resource’ and will store the full JID of the user sending presence.
Creating Mnesia Tables
Now that we have a record, we must create an Mnesia table to use them. Each row of the Mnesia table will be in this format, and the key is the first item of the record, usr. mnesia:create_table/2 does the job for us. Mnesia can create various kinds of tables – disk tables, memory tables, and tables that allow duplicates. By default it creates an in-memory, unsorted table that does not allow duplicate keys, and this happens to be exactly what we want. The Mnesia man page has information on other table types if you are interested.
We’ll create and clear this table in start/2.
start(Host, _Opts) ->
?INFO_MSG("mod_sunshine starting", []),
mnesia:create_table(sunshine,
[{attributes, record_info(fields, sunshine)}]),
mnesia:clear_table(sunshine),
ejabberd_hooks:add(set_presence_hook, Host, ?MODULE, on_presence, 50),
ok.
The first argument to mnesia:create_table/2 is the name of the table, which must be the same as the record name. This is followed by a list of options, of which we only use attributes. record_info(fields, sunshine) is a built in function of Erlang that returns a list of the fields in the record sunshine. Mnesia won’t destroy an existing table if you call create_table/2, so to make sure there isn’t any old data in there we use mnesia:clear_table/1.
Reading And Writing To The Database
Mnesia is a transactional database. To read and write to it, one normally uses mnesia:transaction/1 which takes a function to execute inside the transaction. This can be slow, so often mnesia:dirty_read/2 is used to skip transactions for reads, and we will use that here.
In order to keep the user, server, and resource key consistent, we must stringprep the username and server. Among other things, this ensures that mixed case usernames get lower cased. This is easy in ejabberd since it provides a library of functions for JID manipulation called jlib. To use it, we include jlib.hrl in our module.
Once we have a consistent key, we look it up in the sunshine table. There are three possibilities: no record is found, a record is found for the current packet, or a record is found for a different packet.
The skeleton for this is below.
on_presence(User, Server, Resource, Packet) ->
%% get options
StormCount = gen_mod:get_module_opt(Server, ?MODULE, count, 10),
TimeInterval = gen_mod:get_module_opt(Server, ?MODULE, interval, 60),
LUser = jlib:nodeprep(User),
LServer = jlib:nodeprep(Server),
case catch mnesia:dirty_read(sunshine, {LUser, LServer, Resource}) of
[] ->
%% no record for this key
ok;
[#sunshine{usr={LUser, LServer, Resource},
packet=Packet, start=_TimeStart, count=_Count}] ->
%% record for this key and packet exists
ok;
[#sunshine{usr={LUser, LServer, Resource},
packet=_OtherPacket, count=_OtherCount}] ->
%% a record for this key was found, but for another packet
ok
end,
none.
Those new to Erlang are probably a little weirded out right now. Erlang uses pattern matching extensively. First we read from the database, and attempt to match the pattern of the result against each section of the case statement. If a variable in the pattern already contains a value, it must match the value in the result. If a variable in the pattern does not have a value, it gets the value of the result in that spot.
If we get an empty list, there is no record matching the key. Note that we get a list as a result because some Mnesia table types support duplicate keys. In our table, the result will always be an empty list or a list with one item.
The next two patterns match a row. #sunshine{...} is a record reference for the sunshine record. In the first of the two patterns, all the variables have values except for _TimeStart and _Count. This means the result must be a record that matches the record for this user and this packet. The second pattern matches a record for this user, but with any packet, as _OtherPacket is without a value.
Pattern matching is nice and powerful. Not only did we single out exactly the results we needed without any if statements, we also already put the interesting fields into their own variables!
Now we just need to code the actions for each of these three cases.
In the first case, we create a new entry for that user and packet.
In the second case, we need to determine we’re within TimeInterval seconds of StartTime. If not, we need to reset the count and start time for this user and packet. Otherwise, if the count is above StormCount, the user is sending a presence storm and needs to be disconnected, and if the count is below StormCount we just increment the count.
In the final case, we’ve received a different presence packet, so we need to overwrite the user’s record in Mnesia with a new one for this packet.
Remember that Mnesia writes happen inside transactions as you look at the code below. The writes are put into anonymous functions which are then passed to mnesia:transaction/1 for execution inside a transaction.
on_presence(User, Server, Resource, Packet) ->
%% get options
StormCount = gen_mod:get_module_opt(Server, ?MODULE, count, 10),
TimeInterval = gen_mod:get_module_opt(Server, ?MODULE, interval, 60),
LUser = jlib:nodeprep(User),
LServer = jlib:nodeprep(Server),
{MegaSecs, Secs, _MicroSecs} = now(),
TimeStamp = MegaSecs * 1000000 + Secs,
case catch mnesia:dirty_read(sunshine, {LUser, LServer, Resource}) of
[] ->
%% no record for this key, so make a new one
F = fun() ->
mnesia:write(#sunshine{usr={LUser, LServer, Resource},
packet=Packet,
start=TimeStamp,
count=1})
end,
mnesia:transaction(F);
[#sunshine{usr={LUser, LServer, Resource},
packet=Packet, start=TimeStart, count=Count}] ->
%% record for this key and packet exists, check if we're
%% within TimeInterval seconds, and whether the StormCount is
%% high enough. or else just increment the count.
if
TimeStamp - TimeStart > TimeInterval ->
F = fun() ->
mnesia:write(#sunshine{usr={LUser,
LServer,
Resource},
packet=Packet,
start=TimeStamp,
count=1})
end,
mnesia:transaction(F);
Count =:= StormCount ->
%% TODO: disconnect user
F = fun() ->
mnesia:delete({sunshine, {LUser, LServer,
Resource}})
end,
mnesia:transaction(F);
true ->
F = fun() ->
mnesia:write(#sunshine{usr={LUser, LServer,
Resource},
packet=Packet,
start=TimeStamp,
count=Count + 1})
end,
mnesia:transaction(F)
end;
[#sunshine{usr={LUser, LServer, Resource},
packet=_OtherPacket, count=_OtherCount}] ->
%% a record for this key was found, but for another packet,
%% so we replace it with a new record.
F = fun() ->
mnesia:write(#sunshine{usr={LUser, LServer, Resource},
packet=Packet,
start=TimeStamp,
count=1})
end,
mnesia:transaction(F)
end,
none.
Disconnecting Users
The only thing left to do is disconnect the user when a presence storm is detected. I wish this was as easy as ejabberd_sm:disconnect(User, Server, Resource), but it seems that the ejabberd developers have not yet added something along these lines. To solve this, we will use Erlang’s message passing to notify the user’s c2s process that it should disconnect the user.
After some exploring, I discovered you can get the c2s process identifier for a given user by calling ejabberd_sm:get_session_pid/3 which takes the user, server, and resource. Once we know the process identifer, we can send it a message with Erlang’s ! operator.
First let’s finish out on_presence/4 by replacing the placeholder comment with the disconnect message to the c2s process for the user, shown below.
%% disconnect the user SID = ejabberd_sm:get_session_pid(LUser, LServer, Resource), SID ! disconnect,
Our module is now finished, except that the c2s process will not understand our message. We’ll have to provide a new message handler for the disconnect message in ejabberd_c2s.erl. Add the following clause to handle_info/3 in ejabberd_c2s.erl.
handle_info(disconnect, _StateName, StateData) ->
send_text(StateData, ?STREAM_TRAILER),
{stop, normal, StateData};
You can insert this clause before the first handle_info/3 clause. Just search ejabberd_c2s.erl for handle_info, and when you find the function definition, just paste this code immediately before it, but after the function’s comment.
Now you must recompile ejabberd and restart the server. If you know what you are doing, you can load this patch into the server while it is running. We did this at Chesspark to avoid taking the server down. This is another beautiful feature of ejabberd.
mod_sunshine In Action
We’re all finished, and mod_sunshine can be deployed to stop the evil presence storms. To test that it works, just send the same presence stanza a bunch of times, really fast. You will find yourself quickly disconnected. You will probably need to write a quick test client for this as you might not be able to trigger duplicate presence with your normal XMPP client. I highly recommend Strophe.js for this task.
I hope that this tutorial has been helpful to you, and that you use this knowledge only for good. Go have fun implementing your wild and crazy ideas for server modules! If you have any suggestions and questions, please let me know in the comments.
The full source of mod_sunshine.erl is available here.
Filed under: chesspark, code, jabber, xmpp | 2 Comments
Tags: code, ejabberd, erlang, jabber, module, presence, server, tutorial, xmpp
Yesterday I blogged about choosing an XMPP server. Once you’ve made this decision, it is time to get it set up and running great. If this is your first XMPP server, this is often just some software installation and a little bit of configuration. However, for those who already have an existing set up, or who have special data or run time requirements, this can involve a substantial amount of work. It took us about a week to move from jabberd2 to ejabberd at Chesspark, and this is how we did it. While I talk mostly about ejabberd and jabberd2, many of the issues are similar to those any deployment or redeployment would face.
What We Knew At The Start
There were three main things that we need we needed to work on in order to transition to ejabberd. These were authentication, data migration, and porting a custom server module.
Authentication. Many sites already have a user database they they want to reuse for the XMPP service. This almost never looks like the bare bones schema that the XMPP servers are configured to use by default. The first task then becomes to configure the XMPP server to authenticate from this custom user data.
Jabberd2 is a lot more configurable for database authentication than ejabberd. For starters, it allows you to define the SQL query used to authenticate users right in the configuration file. With ejabberd the choices are to write your own authentication module, modify the database authentication module to provide your own queries, or use external authentication via a script.
External scripts are pretty easy in ejabberd, but we opted for modifying the queries used for authentication in the ejabberd source. For anyone trying this themselves, the queries are all in src/odbc/odbc_queries.erl.
The Chesspark service is open to the entire federated XMPP network, and we store member information for everyone in the same place. This means that we must store the bare JID as opposed to just the node portion. Some users are local to our server, like jack@chesspark.com, and some are not, such as metajack@gmail.com. Only the local users will be used for XMPP authentication, but almost every XMPP server is written to expect usernames without domains, and ejabberd is no exception.
Here is what ejabberd’s default authentication query looks like:
get_password(LServer, Username) ->
ejabberd_odbc:sql_query(
LServer,
["select password from user "
"where username='", Username, "';"]).
We changed it to:
get_password(LServer, Username) ->
ejabberd_odbc:sql_query(
LServer,
["select password from member "
"where lower(username)='", Username, "@", LServer, "';"]).
This wasn’t very difficult at all. We don’t need in-band registration, so we didn’t change all of the user creation and deletion queries, but those would be very similar to this one.
Data Migration. Each XMPP server has its own format for storing data, and if you are switching from one server to another, you will probably want to take your data with you. At Chesspark, we already had 400,000 users worth of rosters, private XML storage, and privacy lists, and we didn’t want our users to lose this information. Our task was to figure out how to migrate all the jabberd2 data to a form ejabberd could use.
Most XMPP server schemas are quite simple and quite similar. We achieved 95% of the data migration by using SQL’s ALTER TABLE command to add, remove, and rename columns. The only trouble we ran into was figuring out which columns in jabberd2 mapped to which columns in ejabberd and dealing with virtual hosts.
Figuring out the mapping was pretty straightforward. We made some users in a test database under the default ejabberd configuration and did some common operations. Then we inspected the database tables and determined what the columns were for. For example, we saved some private XML as a user, then looked at how ejabberd’s private_storage table stored and formatted the resulting data. In almost every case, it was extremely simple to migrate the data. The notable exception is privacy lists, which were stored in two tables in ejabberd instead of just one like jabberd2. This was easily fixed up with a small Python script.
Dealing with virtual hosts was a bit harder. We run multiple servers at Chesspark to handle our branded chess sites like WuChess.com. We wanted to move from multiple jabberd2 servers to a single ejabberd installation with virtual hosting. Unfortunately, since ejabberd stores user names without domains, this enforces unique user names across all virtual hosts, a requirement we weren’t able to meet with our existing user data. We decided to patch ejabberd to use the domain along with the user name to make this easier.
These changes were very similar to the changes for authentication above and we modified nearly every query in odbc_queries.erl. Unfortunately, not every function takes the parameter LServer which is the stringprepped domain, and this made the patch more difficult. We had to add the LServer variable where it was missing, and then modify the ejabberd internals to pass this along in places where it didn’t already do so. This was pretty easy, except for one or two places where ejabberd passed data in a giant list and we had to fix specific entries in the lists.
Unfortunately, not all the SQL queries are in odbc_queries.erl as the code would have you believe. We also had to patch the queries in other modules of the code where they had not yet been moved to odbc_queries.erl. Hopefully the ejabberd team will finish moving these to one place in the future.
After all this patching, it required a few days of testing and tuning to make sure everything was running smoothly. Supporting virtual hosts by adding the domain to usernames is definitely something I feel should be in upstream ejabberd, and I will send my patch upstream to see if they will add this.
The Custom Module. Chesspark is built from a set of components that implement and understand chess rules, player matching rules, and various other things. These components must receive presence notifications to know when users are on and offline. It can happen that a presence notification is slightly delayed and stanzas are sent to an offline client. These don’t cause any harm, but they do take up disk space. With a lot of users in the database, this disk space can start to be substantial, so we wrote a custom server module in jabberd2 to ignore traffic to offline storage if it came from certain components.
Porting this module turned out to be very simple. It took me a few hours to write and test the module. It would have taken a bit longer had this been my first ejabberd module instead of my second. Most of the work involved figuring out various bits of the gen_mod API in ejabberd. For those of you interested in writing or understand ejabberd modules, I highly recommend Anders Conbere’s blog where he has provided several tutorials.
Putting Ejabberd Into Production
After all this work and testing, it was time to put ejabberd into production. We gave notice to the users, took the site offline, and began the data migration process. This process took longer than our trial runs for some reason, totaling a little bit over an hour.
Once the site came back up, we immediately noticed some problems that we had not caught in our testing. First, database traffic was extremely high and load was about 4 times higher than we expected. Next, we discovered that one of our client applications had an infinite loop bug triggered by receiving an echo of your own presence stanzas. Finally, memory use on the ejabberd server was extreme, and over the next day or so it ran out of memory and crashed 3 times.
Solving The Problems
The database problems were the easiest to diagnose thanks to our database profiling tools. It was easy to spot a query that was missing an index, and this immediately caused load to drop. Query traffic was still much to high, and we found two causes for this.
The first cause was our buggy client application which was in an infinite loop sending presence. This caused a presence storm on the server, causing thousands of useless packets to be sent both to the server and to the user’s contacts. This affected database queries because of the second problem.
The other problem is that mod_privacy_odbc in ejabberd is just terrible. It does two database queries, one for rosters and one for roster groups, for every single packet that is delivered to a local user. It does this for the silly purpose of letting stanzas through from your roster contacts that would otherwise be blocked by your privacy settings. Why this exists, I have no idea, but coupled with the presence storms, it caused a massive amount of database traffic.
A combination of solutions was used to solve these problems. First we pushed out an upgraded client that did not have the presence storm bug. Next, we disabled mod_privacy_odbc temporarily to ease the database traffic. After these were implemented, the server was in good shape, but our users who had contacts still running buggy clients were not.
To solve the problem of people who refused to upgrade, we wrote a new ejabberd module called mod_sunshine, which stopped the presence storms. mod_sunshine keeps track of duplicate presence sent from users and terminates their session if they send more than X duplicate presence stanzas in Y seconds. Configured mod_sunshine to kill clients that sent more than 10 stanzas in 60 seconds removed nearly all of the presence storm traffic, and we dealt with the confused users who could not connect with apologetic emails and some upgrade hand holding.
After some new indexes, a new client, and mod_sunshine, we were able to stabilize the database usage and turn back on mod_privacy_odbc.
The last problem was memory use, which was much too high. We are not sure yet of the true reason for ejabberd’s hungry memory needs, but we were able to get past the immediate problem of server crashes by upgrading from 2GB to 8GB of RAM on our XMPP server. Also, moving from the latest stable release, ejabberd 2.0.1, to the trunk of ejabberd development removed a lot of the memory use. Memory use is still quite high, about 1.4GB for ~250 simultaneous users, and we’ll continue to investigate. There does seem to be one reported issue that TLS connections take up substantially more memory than non-TLS connections. Thankfully, unlike jabberd2, we have not detected any major memory leaks, although we are still keeping a close eye on this.
The End Result – Happy Users
After a week of patching and preparation and a few frantic days of chasing down glitches, ejabberd is running quite well. Our users are extolling the lower latency of the new server, which was our ultimate aim with the upgrade. It seems we have achieved some success for now, but as always, we continue to measure and analyze performance and dream of new improvements we can make.
If you have a story about an XMPP server upgrade gone right or wrong, anecdotes about your favorite or least favorite XMPP server, or just comments or question, please share them with us below.
Filed under: chesspark, code, jabber, xmpp | 1 Comment
Tags: chesspark, code, ejabberd, jabber, jabberd, server, upgrade, wuchess, xmpp
Choosing An XMPP Server
Choosing an XMPP server is a big decision. Should you go with the popular one or the one written in the most popular language? Perhaps you don’t plan to become a systems administrator and you need one which is easy to set up and maintain. Unfortunately for people making this important choice, there is not much guidance published beyond features comparisons. What follows is an account of our decision making process on XMPP servers – how we came to pick jabberd2 originally, and how we switched to ejabberd.
A Brief History Of XMPP Servers
XMPP began as Jabber and had only one server, jabberd. As popularity of the protocol grew, more servers appeared, and now there are half a dozen major contenders, both commercial and open source.
The main players these days seem to be:
- jabberd – The original server. It started in C, but now appears to be a mix of C and C++ code. The main users for years was jabber.org itself. The code may have changed substantially in the last couple of years, but I remember it being rather crufty. Matthias Wimmer has been maintainer at least since 2004, and he continues to maintain it today, although it does go through some long periods of inactivity.
- jabberd 2.x – A rewrite of jabberd, originally by Rob Norris. The code is pure C, modular, and fairly easy to understand. Chesspark picked this to start with in 2006. Another notable user of jabberd 2.x is Meebo. Unfortunately it was abandoned by its original maintainers some time ago. Tomas Sterna stepped up to the plate and took over the project, and it is now actively maintained again.
- ejabberd – An XMPP server written in Erlang which claims to be quite scalable. Erlang is the language created decades ago by Ericcson to power their telephone switches. It has many features that make it well suited for XMPP servers. ejabberd has been around and active since early 2005, and is supported officially by Process One. It also has a growing developer community. Jabber.org switched from jabberd to ejabberd some time ago, and continues to run ejabberd today.
- Openfire – A Java server written by JIVE Software. This was formerly a commercial product called Wildfire. I have no personal experience with it, but it appears to have an active community.
- Tigase -Another Java server that started in late 2004. It is actively maintained. I have no personal experience with Tigase other than meeting Artur at the recent XMPP Summit, but the Seesmic folks speak very highly of the project.
This list is incomplete. Notable omissions include Google Talk (not publically available), Jabber XCP (the commercial offering from Jabber.com), and djabberd (Danga’s jabber server written in Perl).
Chesspark’s Initial Decision
Chesspark chose jabberd2 as its XMPP server about 3 years ago. I recall being impressed with the clean and modular code base as well as its ability to change the SQL queries right in the configuration file. It also supported PostgreSQL which was the RDBMS we preferred. I don’t think that Tigase or ejabberd were considered; they were likely too young at the time. Jabberd was the only other real choice, but we were not impressed with the code.
One major factor which influenced our decision was code readability and maintainability. We wanted an XMPP server that we could patch ourselves if needed, and we didn’t want to be stuck in case the project was abandoned down the road. This turned out to be a wise decision – the jabberd 2.x project was unmaintained for a long period while we used it. Over the last few years we’ve made patches and assisted others with patches as best we can.
Jabberd 2.x Disappointment
Over time, Chesspark’s user base got larger as the site became more popular, and jabberd 2.x’s warts began to show. Here are the main ones in the approximate order we discovered them.
- Database transactions abused. Jabberd 2.x does no queries out of the box that require the use of database transactions. By default it is configured to do every query in a transaction, even if it is a simple SELECT. This is a common problem with many libraries we’ve used at Chesspark. Thankfully, jabberd 2.x can be configured to turn this off. Normally this would not affect anything, but the small amount of overhead caused can add up fast when jabberd 2.x does lots of queries, which it certainly does.
- Memory leaks. There are several memory leaks that persist to this day. Even with a small userbase like Chesspark has, this forces us to restart the server about once a week. As memory usage climbs, the server latency gets higher. Our attempts to find this leak have been unsuccessful to date.
- Non-blocking design inconsistent. The server uses a non-blocking design common to scalable daemons; this is great. Unfortunately, all database calls use the blocking database API and a single database connection. This means that even with light load, packet latencies can be quite high if the database isn’t ridiculously fast.
For us, the show stopper was latency. Games depend on near real-time performance, and latency destroys the user experience. We generated test load which logged in a few hundred users, did roster operations, logged out, and repeated. We then measured latency of chess moves on the same server. We were shocked when we saw the numbers – over 3 seconds of lag between an IQ query and its response.
All of these things could theoretically be fixed, and I hope that they are fixed eventually. Diversity in server choices is a feature of XMPP – the more the merrier.
Finding A New Server
Once we decided to abandon jabberd2, we needed to find a new server.
Feature wise, all the current servers support the stack we need – authentication with TLS and SASL, ability to use PostgreSQL as a backend, private XML storage, external components, and privacy lists. Years ago, some of these features were hard to come by, but today they are common. ejabberd, Openfire, and Tigase have pubsub and BOSH support as well, neither of which was available in any open source server when we started.
We knew right off the bat that we didn’t want to be writing C. While we have a lot of C experience, we like to reserve C for the few times it is actually needed and spend the rest of our time in more productive and higher level languages. This removed jabberd from the list.
From here the language choice is Erlang or Java. Erlang is a dynamically typed, functional language – quite a radical departure from the norm for people most C and Java hackers. We work a lot in Python, so Erlang was the closer fit. Many people make the decision to work in Java, and from there they will need to pick between Tigase and Openfire.
One thing to note is that some people seem scared away by the Erlang language. Don’t be one of these people. Erlang is well documented and pretty easy to learn. We knew nothing about Erlang a few months ago. That did not slow us down too much when we needed to write ejabberd modules or make changes in ejabberd internals. Even without knowing Erlang, we were able to write extensions to ejabberd much faster than for jabberd2.
The last part of our decision was to test server latency with ejabberd. We ran the same test that we ran on jabberd2, and ejabberd didn’t flinch. The measured latency at idle was twice as fast in some cases with ejabberd, and there was very little change even as we pounded the database to levels that would have made jabberd2 cry.
Life With Ejabberd
ejabberd is not perfect; no server is. Here’s a list of our current gripes:
- Memory hog – Erlang uses a lot of memory for basic string handling since a string is represented as a list of integers. There also seems to be a bug in TLS that causes it to use quite a bit more memory than non-TLS connections. These add up to quite a bit of memory usage. For Chesspark, we use over a gig of RAM for a few hundred connections. Jabber.org uses about 2.7GB of RAM for its 10k+ connections. I’m not sure what the discrepancy is between these numbers; we are still looking. I expect the TLS memory issue to be solved soon, and the Process One folks told me that they were going to switch the string handling to use Erlang binaries which are more memory efficient.
- Lots of database queries – As with jabberd2, ejabberd does an enormous amount of database queries. With mod_privacy enabled, two roster related queries are done for every packet sent. ejabberd also uses the database inappropriately, with idioms like SELECT, DELETE, INSERT which can lead to race conditions. Thankfully, this does not seem to be a big problem with a correctly tuned database, as ejabberd doesn’t block on the queries.
- Lack of comments in the code – The code is often quite clear, but comments would be helpful. There is some basic, but very helpful, developer documentation, but the code contains virtually no comments. Luckily, this is not as bad as it seems,because many of the idioms in the server are really Erlang and OTP idioms, so reading up a bit on Erlang and OTP answers a lot of questions. I’m also not sure other servers are better. Jabberd2 probably had more comments, but it also had less documentation in general; I’ve found working with ejabberd to be easier.
There are many excellent things about ejabberd that make up for these and other shortcomings, and have made us very happy we made the switch.
- Hot code loading – After we write a new ejabberd module, we can deploy it in production without pausing or restarting the server. We can also redeploy it later if we find a bug. We have even redeployed core server pieces this way with success. This can even be done to some degree right in the Web interface.
- Live console – It is possible to open an Erlang shell inside the running ejabberd node. This makes it really easy to poke around to see which processes are running, how much memory they are using, and which internal database tables are getting full. Java, C, Python, and Ruby have nothing quite like this, although Twisted Python’s manhole is similar.
- Very low CPU usage – You’d think a C based XMPP server would be a winner in this area, but that is not always true. ejabberd uses very little CPU usage, except when things go wrong. Chesspark’s XMPP server is sitting around a load of 0.1 to 0.2; Jabber.org sits at 0.1. This is exactly what you want. An XMPP server should be I/O bound, not CPU bound.
So far we’re pretty happy. How did you pick your XMPP server?
Filed under: chesspark, code, jabber, xmpp | 13 Comments
Tags: c, chesspark, code, ejabberd, erlang, jabber, jabberd, jabberd2, java, openfire, server, tigase, xmpp
It’s taking a bit longer than planned to whip Strophe into releasable shape, but work is progressing well. Since the last update, I have made a new and better Trac instance for the project, made a public SVN repository for the code, added SASL ANONYMOUS support to both libstrophe and Strophe.js, fixed DIGEST-MD5 authentication with ejabberd, did compatibility testing for both libraries, and wrote up a basic Strophe.js example. Still to come are TLS fixes for libstrophe, porting the libstrophe documentation to Natural Docs, working around a BOSH bug with Tigase, and writing up some examples.
On a related note, Matthew Wild is working on an XMPP library for Lua built on top of libstrophe.
Also, remember to vote for my XMPP talk and panel at SXSW 2009.
UPDATED 8/26: It turns out Tigase and Openfire are smarter than the other XMPP servers with DIGEST-MD5 authentication, and that it wasn’t a bug at all. My apologies (kudos) to the Tigase and Openfire teams. The issue is now fixed in trunk.
Filed under: code, jabber, xmpp | 12 Comments
Tags: c, code, jabber, javascript, library, libstrophe, lua, software, strophe, strophejs, testing, xmpp
Learn About XMPP At SXSW 2009
I submitted two XMPP themed proposals for SXSW 2009. The interactive panel picker is now open, so please go vote for my entries if you have any interest in XMPP at SXSW. As far as I can tell, I’m the only one speaking on the topic.
The first submission is a panel called “The XMPP Powered Web“.
XMPP is being used to build the next generation of Web applications. The open messaging standard has crossed from the desktop to the Web and is powering applications for microblogging, gaming, social networks, communication, and data portability. Learn how others are using XMPP and how you can use it, too.
I am planning to invite some well known developers using XMPP for interesting Web projects to join me. Google, Flickr, Seesmic, Chesspark, and others are already building on XMPP technology into the Web, and it is time for the rest of the world to get into the game as well. Vote here.
The second proposal is a solo talk about XMPP basics called “X Is For XMPP: An Open Messaging Primer“.
XMPP is an open, extensible messaging standard designed for instant messaging and presence. The protocol is based on XML and is simple and easy to understand. XMPP can be used to power next generation Web apps. Learn how XMPP works and how you can use it in your own applications.
I’ll cover the basics of the protocol and dive into various ways XMPP can be used. I’ll cover the big XEPs like Pubsub, BOSH, and MUC, as well as topics like using components and writing bots. Vote here.
If any of you are planning to be at SXSW 2009, let me know. I’d love to meet more people building cool things with XMPP.
Filed under: code, jabber, xmpp | 3 Comments
Tags: code, jabber, presentation, sxsw, sxsw2009, xmpp
Thoughts On Scalable XMPP Bots
It used to be that chat bots were toy projects written by young programmers to impress their friends in their favorite IRC channel. These days, some bots are much more ambitious projects. For example, both identi.ca and Twitter are using XMPP bots to distribute messages to users in real-time. At Chesspark, we have chess playing bots that must be able to play hundreds of games at a time. It’s no longer good enough just to have a bot; now the bots have to scale.
Client Bots
The first step along the bot programmer’s journey is to write the bot as an XMPP client. Many client libraries already exist, and the client protocol is well understood. It doesn’t take long to have a working echo bot in any number of languages.
There is a huge problem with client bots – rosters. If part of the client bot’s functionality depends on subscribing to the presence of its users, the roster size becomes the limit to scalability. There are two reasons for this.
First, querying large rosters from persistent storage is an expensive task for the XMPP server. Many servers are written with the assumption that all rosters are fairly small. Once rosters become tens of thousands of entries long, this starts to affect performance elsewhere in the system.
Second, a large roster means an enormous stanza in the connection startup. As I’ve written before, large stanzas are bad. Pretty much all processing will stop for the bot while it receives and deals with this stanza. If that stanza takes more than a few seconds to receive, users will notice this delay when the bot starts up.
Both of these issues affect startup only. It’s not a big deal if your bot has 100% uptime, but this is unachievable. Even worse, if you run into a crash bug, the bot can start to destroy performance of itself and of the server it is connected to. In May of last year, this is what happened to Twitter as its bot passed the 40,000 roster item mark.
What can the bot developer do? Rewrite the bot as a component.
Component Bots
A component is a trusted piece of an XMPP server that can send and receive arbitrary stanzas. Components speak a different protocol to communicate with the server. Because the name of a component is a domain (example: arbiter.chesspark.com), a component can pretend to be many users. Any stanza addressed to user@component.server.com will be delivered to component.server.com no matter what the value is of ‘user’.
Because the XMPP server has no idea what services a component will perform, it does not require a component to maintain a roster for any users it pretends to be. This means that the large roster problem does not exist for components.
If the component still needs to keep track of user presence, it is easy to subscribe and handle subscription requests to and from users. In most cases, the bot won’t even need to keep track of this information. Even if it needs to know the presence of all users, it can just put these values in memory as they change and send presence probes to get them on restart.
S2S Bots
It may turn out that your bot needs super-scalability; in that case, the next step is to have the bot speak the server to server (s2s) protocol directly, and pretend that it is an entire XMPP server. There are several reasons why you might want to do this.
The XMPP component protocol does not currently provide for load balancing or clustering. Some servers may allow this using extensions or by writing server specific components. In order to horizontally scale a component, one must handle this scaling by implementing some inter-component protocol and then using a custom load balancer that speaks the componenent protocol to the server. Since s2s connection endpoints are managed via DNS SRV records, an s2s bot can use DNS as a load balancer and only have to handle communication between the partitions of the bot. While this doesn’t get you scaling by itself, it at least reduces the problem considerably.
In some cases, the XMPP server may be too much overhead if your service is essentially just an XMPP bot. Why send traffic through a routing process inside a server if it is already known that it is for use externally? Even small amounts of overhead can add up quickly at scale. In terms of managing infrastructure alone, taking out an XMPP server from the equation saves a lot of work.
Further Thoughts
We’ve built a lot of bots at Chesspark. To date we have used client bots and component bots to achieve our goals, but in pursuit of high availability, we will need to address the single points of failure. Speaking s2s or developing custom load balancing for XMPP components is on our agenda for the future.
Christopher and I have a bunch of example code that we’ve been preparing showing how to implement XMPP bots in Twisted Python. Look for some tutorials in the near future.
Filed under: chesspark, code, jabber, xmpp | 7 Comments
Tags: bots, chesspark, code, jabber, scalability, xmpp
New Features At Speeqe
We just released some new features for Speeqe, a simple group chat application for the Web. This release includes private room messages, external MUC support, early access for interested people to host their own rooms, external Jabber authentication, and image inlining.
Please check it out. We’re looking for people interested in starting their own Speeqe rooms. If you’d like your own space, just email us, and we’ll make it happen.
Jabber Bugs
We use negative presence priority to keep the Speeqe client from stealing your normal private messages. Unfortunately, if you set a negative priority on your resource, both Jabberd 2.x and the Google Talk server stop sending you presence of any kind. This means that you never get the presence status of people in a chat room, not even intiially. This is a pretty big bug for us, and if anyone has any suggestions on workarounds while we try and work with Google and smoku to get it resolved we’d love to hear them.
External MUCs
As we demoed in Portland at the 5th XMPP Summit, you can now use Speeqe with external MUC rooms. Just substitute the address of the room in this URL: http://www.speeqe.com/room/ADDRESS/ .
After you enter a room for the first time, you can log in using any federated Jabber account (including Gmail). Once you do this once, any room you join will reuse the same information.
UI Love
The UI is a work in progress. We know some things are currently a bit ugly, and we’re working hard to address them. In particular this release was more about technical features than UI design. The next release should have more UI design elements.
Everything in the room is themeable. In a release or two, we will open up the theme creation tools to everyone. Themes are just HTML pages, and Speeqe uses CSS selectors to decide where to fill in content. It can already do quite a lot – see the demo themes for an example.
Filed under: 1 | 3 Comments
Tags: group chat, jabber, muc, speeqe, xmpp
My father-in-law grows corn and soybeans in rural Minnesota. I’m currently spending a week on this farm visiting with my in-laws and trying not to work too hard (ha!). Yesterday I helped empty a grain bin into a semi trailer for sale at the local grain elevator. For me, this whole task was very like system administration. It was a lot of easy but repetitive labor, and every few minutes I came up with some idea to reduce the amount of work I would have to do.
An Example
The grain bin has a flat bottom with a hole in the middle. When the bin is full, grain runs through this hole like sand in an hourglass, is transported horizontally by an auger, and finally is dumped into the trailer via another auger. At a certain point the grain will no longer be pulled into the hole by gravity alone, and so there are secondary holes all the way along the auger. Even these holes will not allow the bin to fully empty, so there is a rotating auger in the bottom that sweeps around moving the last bits of grain to the holes. Even with that sweeping auger, it is still necessary to get in there with a broom and shovel and push the grain to the hole in order to fully empty the bin.
Some Automation Solutions
After a few minutes of this, it seemed to me that the ridges in the floor should be radial so that sweeping grain to the middle is easier. The straight ridges there now made it easy in some directions and really difficult in others.
Next, I thought that an inverted cone design would reduce the manual labor to almost nothing and remove the need for multiple holes. Combined with the radial ridging and some kind of vibration, I think you could get all but a few stray grains into the hole without ever getting into the bin.
Inventing Is Like Perl
It finally made sense to me in a visceral way why farmers are such prolific inventors. System administrators try to reduce their tedious labor to zero; farmers want to do the same. No one wants to spend all their time typing the same commands or shoveling piles of grain. The classic tool of sysadmins for this is Perl, but until robotic farming is more widespread, farmers will have to settle for repurposing or inventing physical solutions.
Just like Perl, some of these solutions are elegant, like the auger, and some are complete disasters.
Filed under: code | Leave a Comment
Tags: farming, inventions, perl, programming, system administration
Microblogging With Group Chat
Like thousands of others, I am playing around on Identi.ca today. Microblogging is a hot topic especially as Twitter’s growth problems have provided fodder for the armchair scalability experts. We’ve also been thinking about microblogging in the context of Speeqe. I’m not sure where our experiments will lead, but you can see our initial endeavor as the Tweeqe theme over at the Speeqe demo site.
Our original idea was very simple. We wondered what would happen if we moved the text entry box to the top of the page and made new content post at the top. This mimics both Twitter and most blog and blog commenting systems. It gives group chat a completely different and fresh feel.
Just changing those simple things has gotten us thinking about group chat in new contexts, like microblogging. I was pretty surprised to see Dave Cridland wish for a similar thing in his blog post about Identi.ca today:
More interesting would be to model identi.ca as a MUC service, wherein each user has control of a MUC room that is populated by the people they subscribe to. That would give a history interface, as well as a “last status” display, almost for free.
Here he has outlined a private group chat room filled with the people that interest you. With Speeqe, you’d create a room, then use a Pubsub service or an Atom importer to bring in your feeds for example. The other way works as well. Why not have your microblog be a group chat room? Just create a Speeqe room with the Tweeqe theme and start posting. After all, what is Twitter but a sort of on-the-fly group chat?
These ideas are still young and need nurturing, but we’d love to hear feedback from the community. What are your new ideas for group chat?
Filed under: jabber, startups, xmpp | 1 Comment
Tags: group chat, identi.ca, jabber, microblogging, speeqe, startups, xmpp
metajack 
You must be logged in to post a comment.