Using Android with the open source Exchange server Zarafa under Linux

Introduction

Over the last four days I migrated my Android contacts and calendar to my own Linux server. Read how to setup the open source exchange server Zarafa under Debian and sync it with you Android smartphone.

In detail, this how-to explains how to setup Zarafa 6.40 (Zarafa Collaboration Platform - Community Version) on Debian Lenny (5.0), working with Exim. You can also use every other Linux distribution, or mail MTA, since there are also binary packages for RHEL, Ubuntu, OpenSuse or SLES, and the source .tgz provided. The Zarafa howtos on the Website also integrate Postfix as mail server.

Install Overview

Installing Zarafa, Setting up the Zarafa Gateway (IMAP/POP Server), iCal/CalDav service, sabre-zarafa (a CardDAV Plugin), Zarafa server (Background Service), MySQL (zarafa stores Mail/Calendar in db), z-push (Active Sync Emulation - for pushing onto Android)

1. Download Zarafa

download ZCP here

2. Install it:

Note: I ran the install.sh, not the packages itself apt-get. the script will fetch and install dependencies with apt-get (like mysql, libxslt1.1, libpoppler3, .. (see the attached install.out for more). When the question about SQL password popped up, i fired up new terminal, and did the third step of this how to (Setup a MySQL DB).

# tar xvfz zcp-6.40.4-24200-debian-5.0-x86_64-opensource.tgz
# ./install.sh

install output

3. Setup a MySQL DB

# mysql -u root -p
mysql> create database zarafa;
mysql> grant usage on *.* to zarafa@localhost identified by '<<mysecretpass>>';
mysql> grant all privileges on zarafa.* to zarafa@localhost;
mysql> exit;

mysql output

4. Create SSL Certifications

Generate a private key (.key):

# openssl genrsa -out server.key 2048
<<enter password>>

This will generate a RSA key of 2048 bits. You can also add --des3 or 1024 as a 3des key with 1024 bits.
Now we can use this private key to generate a certificate signing request (CSR). This CSR is then used, usually, for sending to an Certificate Authority, such as Thawte or Versisign, who verify your identity.
We neglect that option and self-sign your CSR.

Generate a CSR: (.csr)

# openssl req -new -key server.key -out server.csr

Generate a Self-Signed Certifacte (.crt) which is valid for 365 days.

# openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

You most likely already have a .crt and .key file if you use any SSL connection with apache. You just need to generate the .csr.

Single File Certifacte (.pem)

For our Zarafa setup we need to combine the certificate (.crt) and the private key (.key) in a single file. Do so by typing:

# cat server.key > server_combined.pem
# cat server.crt >> server_combined.pem

This gives us the required Server certificate. This text is a note in the server.cfg:

# Required Server certificate, contains the certificate and the private key parts
server_ssl_key_file     = /etc/zarafa/ssl/crt-combined.pem

Make sure you first processed the .key, afterwards the .crt. The order is important

5. Configure Zarafa

# cd /etc/zarafa/

Here you setup your Zarafa install. You've got multiple config files. Mostly you have to enable ssl in every config and point it to the proper certs.

Exemplary, I'll show the difference for the ical.cfg. For all other changes see the attached diffs (.patch):

 # whether unencrypted (non-ssl) connections can be made to the ical server
-ical_enable = yes
+ical_enable = no
 
 # port on which the ical server listens on for unencrypted connections
 ical_port = 8080
 
 # whether ssl connections can be made to the ical server
-icals_enable = no
+icals_enable = yes
 
 # port on which the ical server listens on for ssl connections
-icals_port = 8443
+icals_port = 8085
 
 # default connection to the Zarafa server
 # Please refer to the administrator manual or man page why HTTP is used rather than the UNIX socket.
@@ -55,9 +55,11 @@
 
 # File with RSA key for SSL
 # ssl_private_key_file = /etc/zarafa/ical/privkey.pem (server.key from above) or use the combined
+ssl_private_key_file = /etc/zarafa/ical/crt-combined.pem  (server_combined.pem from above)
 
 # File with certificate for SSL
 # ssl_certificate_file = /etc/zarafa/ical/cert.pem (server.crt from above) or use the combined
+ssl_certificate_file = /etc/zarafa/ical/crt-combined.pem  (server_combined.pem from above)

server.cfg patch. Does everything in the background
gateway.cfg patch. For IMAP/POP.
ical.cfg patch. iCal/CalDav service.

6. Add Users

# zarafa-admin -c zab -p <<secret>> -e "johannes@somehost.de" -f "Johannes Meyer" -a 1

The parameters are

-c: login name
-p: password
-e: email
-f: full name
-a: admin (0/1) (no/yes)
-P: if you want to get ask for password (not shown in bash history)

7. See If Its Working

You can already connect to the service via the webaccess. It's available using this URL:

Your iCal and CalDav can be reached with the following:

or with https for ssl connections. Look inside the ical.cfg for the proper ports.

As a note, I'm running this on a vserver (at Host Europe) and when I've set up my Thunderbird to use SSL, it didn't work, without any error. The problem was that the default port 8443 was already used by my hosts Plesk. So, the connection was always redirected to the from service. To solve this, I simply switched to ical port to 8085.

The Zarafa installer will modify your apache config with the following addition:

        Alias /webaccess /usr/share/zarafa-webaccess
        <Directory /usr/share/zarafa-webaccess>
          DirectoryIndex index.php
          Options -Indexes +FollowSymLinks
         AllowOverride Options

            Order allow,deny
          Allow from all
        </Directory>

8a. Alternative: Setup Exim

You just need to add a router and a transporter. As you know, order of routers is important in Exim. To let Exim know which marcos to use, you'll need to update your update-exim4.conf.conf in /etc/exim4/

I've changed my

dc_localdelivery='maildir_home'

to

dc_localdelivery='zarafa_pipe'

and run a

# update-exim4.conf

Now, every mail is going into your new Zarafa database.

exim_zarafa patch

8b. Alternative - Procmail

Use either this, or the one above.

The advantage of this version is you don't have to mess with your exim configuration, and it will work with any non-database based IMAP/POP3 server. For example dovecot or courier.

My old .procmailrc filtered the messages into folders:

#now put my mail lists into mailboxes
:0:
* ^X-BeenThere: members@lists.somehostxxx.ccc.de
$MAILDIR/.ccc-mailing/

Instead of putting everything into folders, you can additionally lead all mails to the zarafa-dagent. It handles your messages, and if you want, does sorting for you too:

#now put my mail lists into mailboxes
:0:
* ^X-BeenThere: members@lists.somehostxxx.ccc.de
  | zarafa-dagent --folder 'ccc-mailing' user

Because I'm not a big fan of databases, php and all complex stuff, I've set up my procmail to send any new mails to the zarafa-dagent and still filter everything to my old Maildir structure.
You cannot do multiple actions in one procmail condition directly, but you can duplicate mails, then use the second one as a new action.

#now put my mail lists into mailboxes
:0
* ^X-BeenThere: members@lists.somehostxxx.ccc.de
{
  :0c :
  $MAILDIR/.ccc-mailing/

  :0 w
  | zarafa-dagent --folder 'ccc-mailing' xen
}

9. Convert Your Old Mail

Here is a python script that imports your (dovecot delivered) mails (in Maildir format) into your new database.

To use it, type:

# python dovecot-to-zarafa.py <path-to-maildir> <zarafausername>

dovecot-to-zarafa.py

Note: It's not that fast. ~ 2GB of mail in about an hour but at least it's working.

10. Setup z-push (Active Sync/push for Mobile Devices)

# wget http://prdownload.berlios.de/z-push/z-push-1.4.3.tar.gz
# tar xvfz z-push-1.4.3.tar.gz
# mv z-push /var/www

For z-push you need to adjust your httpd config.
Add an Alias to you apache host-site config (i.e. /etc/apache2/sites-available/default and /etc/apache2/sites-available/default-ssl (for ssl)):

        # z-push
        Alias /Microsoft-Server-ActiveSync /var/www/z-push/index.php

Don't forget to do this for https. If you don't, the new active sync account on your android might not connect and end with an expressionless error.
On your android phone, you can now create a new Exchange account under Settings->Accounts & Sync Settings->Add Account->Microsoft Exchange. That will push new mail to your device and sync calendars and contacts.

If you want to try if your z-push is indeed working, you can lead your browser to http://<ip>/Microsoft-Server-ActiveSync. There should be a pop up with a login mask. After the login you'll see the z-push page and the message: "GET not supported".

11. Setup sabre-zarafa (A CardDAV backend for SabreDAV)

Download the plugin here and follow the install instruction. Nothing fancy there, except that you need to copy the php-mapi files to the sabre-zarafa/lib/mapi directory (you'll find those under /usr/share/php/mapi/ in Debian).
To use CardDAV with Thunderbird, download the Sogo-Connector. Now under the Address Book Window go to "File->New->New remote Address Book" and the correct URL is https://server/sabre-zarafa/addressbooks/zarafa-username/Contacts or in german https://server/sabre-zarafa/addressbooks/zarafa-username/Kontakte. So, the url is in fact language dependent.

2012-12-20:

Sabre-zarafa version 0.15 worked properly with my Zarafa 7.0(.5) installation. After I've upgraded to the 7.1(.2) branch, I kept getting errors. It seemed some values in the contacts (misc properties in the propertiesToVObject function) weren't handled correctly. The error message was something similar to this (with changing offsets):

FATAL server  - PHP error 8 in sabre-zarafa/vcard/VCardProducer.php:288 : Undefined offset: -1241513954

I found this thread in the zarafa community forums and tried to apply the changes. For me, the patches didn't work (on Linux), even if after trying to remove the carrige return as stated there.

Now, I've uploaded a 0.15 version with the patchset 2 from the thread above. You can download it here or download my single patchfile here (contains patches 0000-0015). This version works for me with the current Zarafa release (version 7.1.2) using Thunderbird 17 with Sogo Connector.

2013-01-03:

The Sogo Connector seems to not work properly anymore. It only synchronizes initially, even if i manually trigger it from the drop-down menu, nothing happens to the local synchronized addressbook. It does create new contacts on the server for me, but does not fetch any new ones to the local side. When I remove the account, and reapply it, everything is there.
I've tested eM Client for the first time (Windows Client with native CalDav/CardDav support), everything works like a charm here with the patched sabre-zarafa (15pl2 - Update 2012-12-20 on Zarafa 7.1). Bidirectional contact- and addressbook-sync. So I think the patch does work, just not the Sogo Connector.

2013-04-24:

The author of the patches took over development of the sabre-zarafa and merged it with a more recent sabreDav. Here is the link to the github project. The newest version (v 0.18) finally works flawless for me.

Also note the install instructions on the github site. You need to fetch a recent version of sabreDAV (1.8.x) now manually, since it doesn't come pre-bundled anymore. Also you need apache log4php and the php-mapi files as written above.

12. Additional Information

You reach your new services with:

Webaccess:
Ical/caldav:
CardDAV:
  • https://server/sabre-zarafa/addressbooks/zarafa-username/Contacts/
IMAP/POP3:
  • Just as usual

For german users, you can change the language to german by editing the ZARAFA_LANG-option if your systems language is not set to German per default.
In /etc/zarafa/userscripts/ properly adjust the *_common.sh scripts to

ZARAFA_LANG="de_DE.UTF-8"

New users/companies/groups are now in german.

Additional Information - Upgrade to Version 7.

Simply follow the instructions from the Zarafa Upgrade Guide. Make sure that you properly upgrade your database after the installation of the new packages and before going online again. Use the zarafa-upgrade script found at /usr/share/doc/zarafa/zarafa7-upgrade. After that take a look at the Zarafa Alerts Page if you find any SQL errors in your server.log.

I've also encountered several lines of out of range values in the server.log.

SQL Failed: Out of range value for column 'val_ulong'

To fix these, update your z-push to an newer version.

If you cannot login with your user using your mail client after the upgrade anymore, it may be that your user has not the sufficient rights to do so. With Version 7 comes a more granulated rights-management-system. In you gateway.log you'll find errors as the following:

[ ] Starting worker process for IMAP request
[ ] IMAP not enabled for user 'user'
[ ] Client disconnected

Help is found here. In short, you'll have to add the following lines to your server.cfg

# In this list you can disable certain features for users. Normally all features are enabled for all users, making it
# possible through the user plugin to disable specific features for specific users.
# This list is space separated, and currently may contain the following features: imap, pop3
disabled_features =

As an alternative, you can also set user-wide rights. Use these commands

zarafa-admin -u john --enable-feature imap
zarafa-admin -u john --disable-feature pop3

to enable imap and disable pop3 for john. To check the john's current rights, use

zarafa-admin --details john

Comments

Stuartt (not verified)
Mon, 04/07/2011 - 03:31

When setting up the SSL cert, I think you put a typo here:

# cat server.key > server_combined.pem
# cat server.crt >> my_server_combined.pem

Should either be server_combined.pem or my_server_combined.pem both times.

zab
Wed, 06/07/2011 - 20:21

Jipp, thanks alot. Always the same situation. When i sit there and think: "uhmm, hey, you could write a howto for it!" And then, if you actually do it, the mess begins!
Even a day, or just a few hours later, i cannot write it down from memory without repeating most of the steps, and those mistakes are the result of trying to save time..

Have you tried the steps, they worked otherwise?

brmiller (not verified)
Sat, 30/07/2011 - 07:01

I followed this guide to the letter two months ago. Everything worked. I had some Android kinks to work out (read: learn how to not be a n00b), and so I hadn't used my Zarafa server. I just brought it back up, and re-edited the configs on my server because I decided I wanted to use a real cert for SSL. Everything still works. Thanks for writing such a concise guide. I am going to enjoy my own personal cloud now. :)

Georg (not verified)
Wed, 14/03/2012 - 10:55

Hi there, this is a nice short guide how to get things accomplished. Now, I still have the problem, that while syncing calendar entries in both ways work flawlessly, this works only for contacts, edited newly in Zarafa. Any idea how I can sync existing contacts of Android's built-in contact app? BTW, calendar entires created in Android's built-in calendar app get synced only, once the "view and sync" property is deactivated for the google account. So I wonder if there are any special settings required for my contact app as well? Any comments are appreciated.

Sat, 05/01/2013 - 14:01

I dont know about everything about Android, since im using a Windows Phone. But it should be, that each Account has its own Adressbook, not accesible by the other Accounts. If you have added a Adressbook for the Zarafa Server (and his Account on the Phone), this Contact is only snced with Zarafa Server. If there is another Acount, maybe for Google, than this Contacts are only synced with Google and so on. To sync the Google Contacts to Zarafa, you should copy this Contact to the Zarafa Adressbook or you could export this Contacts from Google to a CSV or VCF and imprt this List to Zarafa.

Sat, 05/01/2013 - 12:28

It works good, if i sync one contact with Zarafa. Im using for testing the virtual appliance in VMWare. However, if i add now another Contact in Webaccess or WebApp, this contact does not sync to Thunderbird. But when i add a new Contact in THunderbird, this Contact syncs to Zarafa perfectly, also with a Contact Picture.

Additionaly, if i try to Access the CardDAV URL with a Browser, i get a Fatal Error: Maximum execution time of 30 seconds exceeded in /usr/share/sabre-zarafa/lib/Sabre/VObject/Property.php on line 166. Could you please help?

BTW: im in the near of munich to.

Mr T (not verified)
Wed, 20/02/2013 - 10:54

Crosspost to here: https://forums.zarafa.com/showthread.php?6904-SabreDAV-backend-for-Zaraf...

I copied your patched files over to my /var/ww/html/sabre-zarafa directory.
I saw the note about copying MAPI files. So I copied:

cp -R /usr/share/php/mapi /var/www/html/sabre-zarafa/lib

That didn't fix it. So I did another search for mapi and this time, I copied:

cp -R /usr/share/zarafa-webapp/mapi/ /var/www/html/sabre-zarafa/lib/

That ALSO didn't fix it.

I tried changing my CACHE file version in config.inc.php from 2 to 3 to 4. I tried changing my VCARD version from 3 to 4 to 2. I've mixed and matched. Nothing changes the fact that I continue to get this error:

2013-02-20 10:49:15 FATAL server - PHP error 8 in /var/www/html/sabre-zarafa-0.15/vcard/VCardProducer.php:288 : Undefined offset: -1241513954

I have copied an address from my Mac addressbook into the Zarafa account on the left side. The card DOES get created on the server. I can see it through webapp. But the card I created in the webapp NEVER gets copied back to the Mac.

Any help would be appreciated.

+++++++++++++++++++++
UPDATED:
I have tried installing three different "versions" of this program. After upgrading to Zarafa 7.1.3 and copying the MAPI files from that update, I am not getting the offset error in the two PATCHED versions, but I'm still getting that error in the original version. But there are still issues in the patched versions.

I found that if I delete my account in Mac Calendar and recreate it, the first time it connects it will download any cards created on the Zarafa server through webapp. But it does it only ONE time.

Cards created after that initial session never get downloaded. I can close/open the program, refresh the calendar, set it to auto-refresh every minute -- it doesn't matter. I left the client up for hours and confirmed just now that none of the cards I created AFTER the first download were available. Also, any changes I made in the webapp to a card were not downloaded or noticed.

For the record, I have tried the standard sabre-zarafa, the sabre-zarafa-pl2 and the sabre-zarafa-patches-from-fora releases.