Want $500? Help Us Fix Jabberd2!

22May08

We make heavy use of jabberd2 2.1.x at Chesspark. smoku has been making great progress taking this somewhat orphaned code base, maintaining it, and adding new features. However there is a memory leak in the session manager component that neither he nor we have been able to find.

Please see the following MRTG graphs of memory usage to witness the problem:

xmpp.memory-month.png
xmpp.memory-year.png

The jumps in the graphs are server restarts.

The server is doing nothing abnormal. It is handling some private xml storage, privacy lists, rosters and avatars in the normal ways and is backed by a Postgresql database. The number of users online at any one time is in the hundreds. Group chat traffic is fairly high with our Chesspark application since all games are also MUC rooms.

I am offering a bounty of $500 and a Chesspark t-shirt to the first person who fixes this leak. I will also award a partial $100 bounty (and shirt) if you can write a reproduction script that will help others find and fix it.

The intention is to improve the jabberd2 code base, so your patch should be suitable for acceptance into the official tree.

Advertisements


7 Responses to “Want $500? Help Us Fix Jabberd2!”

  1. 1 rob

    I haven’t looked at the code since 2006 so I don’t know what’s changed since then, but during my watch there was a major memory issue in that the session manager would never release user data that it had previously loaded. There was a really good reason for this at the time that I’ve forgotten now. Your graphs look rather familiar. Does that issue still exist?

    –Rob (ex jabberd2 lead developer)

  2. 2 john

    Hey Rob,

    there’s this short writeup on the memory leak issue, on the jabberd2 wiki:

    http://jabberd2.xiaoka.com/wiki/KnownBugs#xhashpoolleakbymisuse

    Does this seem familiar? It doesn’t seem to be on purpose, contrarily to what you’re saying. Would you have any insight on the matter?

  3. Tomasz, the lead jabberd2 developer, created a ticket for your bounty at the jabberd2 website. I’ve attached a patch to the ticket that might help the memory leak if you’re using the MySQL storage backend. If it’s not too inconvenient, I encourage you to try it out and leave a comment on the ticket with your findings: http://jabberd2.xiaoka.com/ticket/219

    Also, in the past I’ve investigated the usage of many of the hash tables in jabberd2 and most of them appear to free the data they’re responsible for in a timely fashion.

    valgrind’s massif tool is a great way to track down the primary users of memory in a program, but it’s sometimes hard to use in a production environment, and it’s hard to simulate real world traffic in a test environment.

    Tracking down memory leaks in programs that use memory pools is hard 😦

  4. 4 metajack

    @Mark: so the key leaking issue that john mentioned isn’t really a problem? Unfortunately we do not use mysql, but postgresql. Although perhaps the postgresql backend has a similar leak? I apologize for not specifying that in the bounty above. I’m happy to pay you a partial bounty when smoku commits your patch.

  5. It’s very possible that the the key leaking issue that John mentioned is still a problem… I wasn’t extremely thorough, and I’m sure the code has changed a bit since I looked at it late last year. But I would venture to say that it’s not likely to be major cause of the memory growth you’re seeing.

  6. @metajack: I’ve uploaded another patch to the jabberd ticket at http://jabberd2.xiaoka.com/ticket/219 This one is not mysql-specific. I’d love to hear if it helps the problems you’re seeing with memory usage.

  7. Hi again. metajack, I think you’ve switched to using ejabberd? But I thought I’d leave a follow-up post in case someone else stumbles across this. The two patches on ticket 219 have been committed, along with another change to get rid of another cache which may have contributed to the memory growth you saw.



%d bloggers like this: