Roundcube Next – If we would start over again

Roundcube indeed became a huge success story with tens of thousands of installations worldwide. Something I never expected back in 2005 when I started the project as a fresh alternative to the well established but already aged free webmail packages like SquirrelMail or Horde IMP. And now, some 9 years later, we find ourselves in a similar position as the ones we previously wanted to replace. Although we managed to adapt the Roundcube codebase to the ongoing technological innovations, the core architecture is still ruled by the concepts which seemed to be right back when we started. And we’re talking about building a web app for IE 5 and Netscape 6 when browsers weren’t as capable and performant as they are today and when the term AJAX has not yet been known nor did we have nifty libraries such a jQuery or Backbone.js at hand.

It more often happens that, when discussing the implementation of new features to Roundcube, we find ourselves saying “Oh man, that’s going to be an expensive endeavor to squeeze this into our current architecture! If we could just…”. This doesn’t mean that the entire codebase is crap, not at all! But sometimes you just silently wish to give the core a fresh touch which respects the increased requirements and expectations. And that’s the challenge of every software product that has been around for a while and is still intensively developed.

When looking around, I see inspiring new webmail projects slowly emerging which don’t carry the legacy of a software product designed almost a decade ago. I’m truly happy about this development and I appreciate the efforts of honest coders to create the next generation of free webmail software. On the other hand it also makes me a bit jealous to see others starting from scratch and building fast and responsive webmail clients like Mailpile or RainLoop which make Roundcube look like the old dinosaur. Although they’re not yet as feature rich as Roundcube, the core concepts are very convincing and perfectly fit the technological environment we find ourselves in today.

So what if we could start over and build Roundcube from scratch?

Here are some ideas how I could imagine to build a brand new webmail app with todays tools and a 9 years experience in developing web(mail) applications:

  • Do more stuff client side: the entire rendering of the UI should be done in Javascript and no more PHP composing HTML pages loaded in iframes.
  • The server should only become a thin wrapper for talking to backend services like IMAP, LDAP, etc.
  • Maybe even use a common API for client-server communication like the one suggested by Inbox.
  • Design a proper data model which is used by both the server and the client.
  • Separate the data model from the view and use Backbone.js for rendering.
  • Widget-based UI composition using simple HTML structures with small template snippets.
  • Keep mobile, touch and hi-res devices in mind when building the UI.
  • Do skinning solely through CSS and maybe allow single template snippets to be overridden.
  • More abstraction for storage and caching layers to allow alternative backends like MongoDB or Redis.
  • Separate user auth from IMAP. This would allow other sources or accounts to be pulled into one session.
  • Use more 3rd party libraries like require.js, moment.js, jQuery or PHPMailer, Monolog or Doctrine ORM.
  • Contribute to the 3rd party modules rather than re-inventing the wheel.

While this may now sound like a buzzword bingo from a web developers conference (and the list is certainly not complete), I indeed believe in these very useful and well developed modules that are out there at our service. This is what free software development is all about: share, use and contribute.

But finally, not every part of your current Roundcube codebase is badly outdated and should be replaced. I’d definitely keep our current IMAP, LDAP and HTML sanitizing libraries as well as the plugin system which turned out to be a stable and important component and a major contributor to the Roundcube’s success.

And what keeps us from re-building Roundcube from the ground up? Primarily time and the fear of jeopardizing the Roundcube microcosmos with a somewhat incompatible new version that would require every single plugin to be re-written.

But give use funding for 6 month of intense work and let’s see what happens…

The Kolab story

Today I’d like to share a success story of a picture perfect project collaboration as it only happens in the open source world without any commercial, political or geographical borders. It all started back in 2009 after a short interview about Roundcube was published on a blog. Short time after we got an email from Georg Greve, founder of the FSFE and member of the Kolab Groupware project. At that time, Kolab already made its name as a free competitor to Microsoft Exchange and Outlook and they were just about to found a new company to push Kolab to the next level. One thing Kolab definitely needed was a better web client to access all the groupware data from anywhere. And this is where Roundcube seemed to fit in perfectly. Although Roundcube was “just” an email client, the Kolab guys saw great potential in our codebase and the vital community around it. And now, more than three years after, we can all witness the great success of this decision.

Lots of lessons learned

After some first meetings and discussions about a possible collaboration between Kolab and Roundcube, we agreed on a general interest from both sides. For the newly founded company Kolab Systems who (as opposed to the Roundcube project) directly hits customers with real needs and urgent bugs to be fixed, it was important to get direct and reliable access to Roundcube developers in order to be agile enough to establish professional services around the open source software stack that is Kolab.

This is also where we started to learn how to run an open source project in a more professional and sustainable way. They taught us the importance of having release branches with long-term support, a clear road map and well documented changesets. Kolab also helped us to better understand the whole licensing topic and with them getting us some legal consultancy from the FTF we finally made the transition to GPLv3 with the added exceptions for skins and plugins. Escpecially the latter one opened the doors for Roundcube to become more accepted in commercial environments which previously had their concerns about the strict terms of the GPL.

Great boost of development

But let’s also have a look at the product side of the story. All users of Roundcube have been on the receiving end of technical and other benefits starting from the 0.7 release. To give you some background about myself:

When Georg and I were talking for the first time, my day-job left me very little time to work on Roundcube. I hadn’t contributed much code in a while, and was even considering to step back from the project altogether because I fell short of my own expectations. Of course it’s impossible to say what would have happened in an alternate universe, but fact is that Kolab Systems enabled me to get back to work on Roundcube. And because Kolab is fully open source, everything has become available to all who use Roundcube.

Here are just some of the features and plugins we added to Roundcube as part of the Kolab integration work:

  • ACL plugin
  • Calendar plugin
  • Tasks plugin
  • Advanced contact search
  • Savable contact search queries
  • Personal spell check dictionaries
  • ODF document viewer plugin
  • Inline PDF viewer plugin

And these are only the “visible” additions to our software which have primarily been initiated and forced by Kolab. Lots of improvements on the stability and quality of the underlying codebase, namely the IMAP and LDAP libraries, are referable to reports coming from the Kolab community.

The work on the various plugins used to bring full-stack groupware functionality to Roundcube also resulted in several upstream patches to other open source projects such a jQuery UI and the jQuery fullcalendar. And that’s what makes free software development truly awesome!

Quality guaranteed

Another very positive aspect of the collaboration with a mature OSS project such as Kolab with a competitive company in the background is the quality assurance and testing we get from them. While we get a lot of bug reports from our own community that help stabilizing our code, there’s no real QA process set up for Roundcube, mainly due lack of resources. Because Kolab Systems takes responsibility (and money) for the installations of the Kolab Groupware, they have a strong interest in quality assurance. And with Roundcube now being an important component of their suite, we can just ride on the wave of QA management and processing. The only price we pay is to fix bugs within a reasonable time. And again, the results of this are accessible for everybody who regularly updates their Roundcube installation.

On an operative and decision-making level, Kolab Systems and the Kolab project itself strive to be “good citizens” in the open source world the same way as we do. Without having to ask for it, it was made clear that Roundcube would remain its own fully independent and emerging project run by the community. Whenever there is a question of whether a certain feature is in Roundcube’s general interest, whether it should go into Roundcube core, or into the Kolab specific modules, or perhaps even approach differently altogether, that decision is always ours to make with no interference from Kolab.

So we kept Roundcube as the lean, focussed web mailer that I believe it should be, and moved all other functions into their separate modules. From the perspective of Roundcube, this has led to improvements on all aspects that matter: adoption, code, quality, community and allowing both Alec and myself to spend lots of time that benefits all users of Roundcube.

Summarizing all these facts and stories, this “K”ollaboration finally IS nothing but a success story. Two great open source projects came together to become even greater.

Migrating our subversion repository to github

We recently moved our svn repository to git hosted at github. Now I’d like to share my experience doing that migration and explain the steps we took to do so. I pretty much followed the process described by John Goulah in his blogpost. So here are the concrete steps I took:

1. Download the entire subversion repository to the local machine for performance reasons.
2. Create an authors mapping file which looks something like this:

#svnuser = gituser <email>
thomasb = thomascube <thomas@...>

3. To start, initialize the git repository with reference to the old svn repos:

$ git svn init file://<path-to-local-svn-repos> roundcube-git

4. cd into the roundcube-git folder and edit the .git/config file to configure what to migrate. I just added the following blocks:

	authorsfile = authors.txt
[svn-remote "svn"]
	url = file://<path-to-local-svn-repos>
	fetch = trunk/roundcubemail:refs/remotes/svn/trunk
	branches = branches/{release-0.6,release-0.7,release-0.8}:refs/remotes/svn/*
	tags = tags/roundcubemail/*:refs/remotes/svn/tags/*

Don’t forget to place the authors.txt file into the destination folder!

5. Now we can start importing the subversion data by executing

$ git svn fetch

Depending on the size of your repository this will take a while to run. Grab yourself a coffee or two.

6. There are a few more tasks to run in order to finish this off. Download the scripts from the git-svn-abandon repository and put them somewhere in your $PATH. Then run

$ git-svn-abandon-fix-refs

7. Now we also want to create a table mapping svn revisions to git commits. This can be used later on to update references to revisions in Trac comments for example. The scripts used to do that can be downloaded from the TRAC-SVN-to-GIT-migration project.

$ > rev-lookuptable.txt

8. After creating the mapping table, we can now remove the comments referring to the old svn revisions from every git commit messages. This is done with

$ git-svn-abandon-cleanup

Done. The git repository is now complete. For Roundcube, we’re using a public github repository so what’s left is to push all the data to that:

$ git remote add origin
$ git push --all
$ git push --tags

The next challenge is to update our Trac platform to place nice with the github repository.

9. Showing a git repository as source in Trac is pretty easy. Clone the repository somewhere on the machine where trac runs and install the GitPlugin for Trac. Just follow the instructions on the referred wiki page. In addition to that, the github-trac plugins helps to keep the Trac clone updated when new changes are pushed to the remote github repository.

10. Finally, we wanted all references to SVN revisions in tickets and comments to point to the according commits in our new github repository. And we therefore created the mapping table in step 7. But it turned out that the revision mapping table created by wasn’t correct. None of the listed git commits finally existed in the git repository. Maybe the git-svn-abandon-cleanup changed them. So I was forced to build a new one in order to update all references. I achieved that by comparing the logs from both the SVN and GIT repository with this quickly hacked PHP script.

svn log file://<path-to-local-svn-repos> > svn.log
git log > git.log
php git-svn-create-rev-map.php svn.log git.log > rev-lookuptable.txt

Now we could start the (slightly modified) convertTracTickets.php script from the TRAC-SVN-to-GIT-migration tools and change all references in our Trac tickets. But don’t forget to backup your database before that step.

Say hello to the new face of Roundcube

We’re very happy to give you a first preview of the all new skin for Roundcube webmail. It was created by FLINT, the very talented graphic designer who already designed our website. He worked hundreds of hours for free just to make Roundcube the best-looking webmail software on the planet. And the results are overwhelming: the design is very stylish and modern while remaining clear and functional and it makes our current default skin look like Squirrelmail ;-)

The all new mail view of Roundcube

See all the drafts at our Flickr page.

The fake contents of the screens are in German and they include some features which are not yet implemented but it’s nice to have an idea of the future.

“Roundcube = Sensational” – December 2010

So nice… we really appreciate messages like that.

From: N. I.
Date: Tue, Dec 14, 2010 at 16:09
Subject: Roundcube = Sensational

I'm a long-time user of many OSS packages and am aware of the limited time you guys have. I just wanted to say that I've installed the stable version of Roundcube onto my Ubuntu Server and it is, quite frankly, absolutely fantastic. I've played aroud with Horde, Squirrelmail etc over the las couple of years and have found absolutely nothing which comes close to the fantastic look and feel and ease of installation and configuration.
I'm looking forward to using Roundcube and hope to contrib code/docs etc in the future.
Once again, well done on a truly excellent job!
yours (via my new Roundcube installation)
N. I.

“Thanks” – August 2010

What would you do?

From: <unreadable>
Date: Fri, Aug 6, 2010 at 10:49
Subject: Thanks

[root@local roundcubemail]# patch -p0 < roundcubemail-0.1.1_chpwd_forward.patch
patching file index.php
Hunk #1 FAILED at 389.
1 out of 1 hunk FAILED -- saving rejects to file index.php.rej
patching file program/js/app.js
Hunk #1 FAILED at 283.
Hunk #2 FAILED at 294.
Hunk #3 FAILED at 988.
3 out of 3 hunks FAILED -- saving rejects to file program/js/app.js.rej
patching file program/localization/en_US/
Hunk #1 succeeded at 336 with fuzz 2 (offset 93 lines).
patching file program/localization/en_US/
Hunk #1 FAILED at 79.
1 out of 1 hunk FAILED -- saving rejects to file program/localization/en_US/
patching file program/steps/settings/
Hunk #1 FAILED at 262.
1 out of 1 hunk FAILED -- saving rejects to file program/steps/settings/
patching file program/steps/settings/
patching file program/localization/zh_CN/
Hunk #1 succeeded at 264 with fuzz 2 (offset 87 lines).
patching file program/localization/zh_CN/
Hunk #1 succeeded at 96 with fuzz 2 (offset 17 lines).
patching file program/steps/settings/
[root@local roundcubemail]# echo $?
[root@local roundcubemail]#

Why ? ?

Thanks !

My environment: OpenLDAP + roundcubemail-0.3.1

“I would like…” – January 2010

How a webmail solution could protect you from mentally ill people.

From: K.W.
Date: Mon, Jan 25, 2010 at 14:27
Subject: A feature is missing! I would like…

…to block one speciffic sender of e-mails. AND tell that person I have blocked his/her ALL mails to my account.

This is mentaly ill person that keeps on sending me lots of information I've not requested. Actually it is a case for the police, but I am an entrepeneur, have 3 kids, a house, broken cars and a wif to catch up with. O…I forgot the cat. I just want to skip the sick person and not bather any more with the problem.

THAT feature doesn't exist in the solutions you offer. How do we solv this?

The company I use as e-mail provider is SUPPRESSED

Sincerely,  yours…
Mr. K. W.


Get every new post delivered to your Inbox.

Join 29 other followers