wiki:howto/ZarafaMigrateConnectux

Version 36 (modified by Edwin Eefting, 13 years ago) (diff)

--

Zarafa migratie van Connectux naar SYN-3

Algemeen

  • Update uw connectux server naar 6.20, oudere versies worden niet ondersteund door deze procedure. (op aanvraag wel mogelijk)
  • Zorg dat de users zijn aangemaakt op de SYN-3 server. (Als de installatie te groot is om dit handigmatig te doen, neem dan contact op met ons. Wij kunnen hiervoor scripts maken.)

  • Zet de mail ontvangst op beide servers tijdens de conversie uit. Anders kan nieuwe mail verloren gaan.
  • Deze procedure verwijderd alle bestaande Zarafa gegevens op de SYN-3 server.

Database kopieren van Connectux naar SYN-3

  • Zorg dat u mimimaal SYN-3 versie 4.4.1 hebt.
  • Zorg dat de Zarafa database op de SYN-3 server leeg is:
    [Syn-3] root@syn.kado.local ~# svcstop /service/zarafa-server/                                                                                                                               
    Stopping /service/zarafa-server/ ....OK
    [Syn-3] root@syn.kado.local ~#  mysqladmin -f -p`cat /etc/my.passwd` drop zarafa && mysqladmin -p`cat /etc/my.passwd` create zarafa && rm -r /home/system/zarafa/*                
    Database "zarafa" dropped
    [Syn-3] root@syn.kado.local ~# 
    
    
  • Achterhaal de mysql user en pass van de database op de Connectux server:
    linux:/md0 # grep mysql /etc/araneus/zarafa/server.cfg
    mysql_user              = root
    mysql_password          = 12345
    mysql_database          = zarafa
    
  • Om tijd te besparen, zorgen we dat de database rechtstreeks geïmporteerd word op de SYN-3 server, zonder tussenkomt van tijdelijke files. Voer uit op de Connectux server:
linux:/md0 # mysqldump --quick -uroot -p'12345' zarafa | ssh root@10.0.0.6 'mysql -p`cat /etc/my.passwd` zarafa'
root@10.0.0.6's password:
  • Afhankelijk van de database grote, kan dit soms een paar uur duren. Met dstat en top kunt u op de syn3 server zien wat er gaande is.

Mocht de conversie onderbroken worden met de melding 'ERROR 1153 (08S01) at line 24818: Got a packet bigger than 'max_allowed_packet' bytes', dan dient u deze instelling hoger te maken in beide mysql config files

User ID's en database conversie

  • Kopieer de passwd file van de oude server naar /root
  • Gebruik dit script voor de conversie:
    #!/bin/bash
    # Connectux2Zarafa Conversion script (C)2010 DatuX - Released under GPL 
    
    svcstop /service/zarafa-server
    
    
    SQLPW=`cat /etc/my.passwd`
    
    echo "SYN-3: Reading user ID converions:"
    echo "old user id -> username -> new user id"
    #for OLDUID in `echo 'select externid from users where isgroup=0 and externid>0' |mysql -p$SQLPW zarafa`; do
    for OLDUID in `echo 'select externid from users where object_type=1 and externid>0' |mysql -p$SQLPW zarafa`; do
        [ "$OLDUID" == "externid" ] && continue;
    
        USER=`cat ~/passwd |egrep ".+:.+:$OLDUID:"|cut -f1 -d:`
        NEWUID=`getent passwd |egrep "$USER:.+:.+:"|cut -f3 -d:`
        echo "$OLDUID -> $USER -> $NEWUID"
        if [ "$USER" == "" ];  then
            echo "Old user $OLDUID not found in old passwd file!"
            [ "$FORCE" ] ||  exit 1
            continue;
        fi
        if [ "$NEWUID" == "" ];  then
            echo "User $USER does not exist on this system yet!"
            [ "$FORCE" ] ||  exit 1
            continue;
        fi
        QUERY="$QUERY""update users set externid='$NEWUID' where (externid='$OLDUID');"
    done
    
    if [ "$QUERY" == "" ];then
        echo "Something went wrong.."
        [ "$FORCE" ] ||  exit 1
    fi
    
    
    echo "SYN-3: Updating database:"
    echo "query: $QUERY"
    echo -e "$QUERY" | mysql -p$SQLPW zarafa || exit 1
    
    echo "SYN-3: Starting zarafa in conversion mode:"
    rm /var/log/zarafa/server.log
    zarafa-server --ignore-attachment-storage-conflict
    sleep 10
    tail -f /var/log/zarafa/server.log &
    
    while ! grep 'Startup succeeded' /var/log/zarafa/server.log; do
        echo "SYN-3: Waiting for conversion to finish..."
        sleep 60
    done
    
    killall zarafa-server tail
    
    echo "SYN-3: Converting attachments from database to files"
    perl /usr/doc/zarafa/db-convert-attachments-to-files root $SQLPW zarafa /home/system/zarafa delete || exit 1
    chown -R zarafa:zarafa /home/system/zarafa || exit 1
    
    
    svcstart /service/zarafa-server || exit 1
    echo "SYN-3: Conversion done, Zarafa should be available now."
    
    
  • Start met FORCE=1 ./convert als u wilt doorgaan zonder user-controle. (voor als er users ontbreken)\

Problemen tijdens conversie

Hier een voorbeeld van een standaard probleem:

[Syn-3] root@linux.ours.local ~# FORCE=1 ./convert
Stopping /service/zarafa-server ...OK
SYN-3: Reading user ID converions:
old user id -> username -> new user id
10001 -> osman -> 1002
10002 -> herman ->
User herman does not exist on this system yet!
10003 -> sunplan ->
User sunplan does not exist on this system yet!
SYN-3: Updating database:
query: update users set externid='1002' where (externid='10001');\n
SYN-3: Starting zarafa in conversion mode:
SYN-3: Waiting for conversion to finish...
Thu Jun  9 22:21:40 2011: Starting zarafa-server version 6,40,8,27223, pid 24485
Thu Jun  9 22:21:40 2011: Connection to database 'zarafa' succeeded
Thu Jun  9 22:21:40 2011: WARNING: zarafa-licensed not running, commercial features will not be available until it's started.
Thu Jun  9 22:21:40 2011: Start: Adding externid to 'object' table
Thu Jun  9 22:21:40 2011: Done: Adding externid to 'object' table
Thu Jun  9 22:21:40 2011: Start: Creating Single Instance Attachment table
SYN-3: Waiting for conversion to finish...
SYN-3: Waiting for conversion to finish...
SYN-3: Waiting for conversion to finish...
SYN-3: Waiting for conversion to finish...
SYN-3: Waiting for conversion to finish...
SYN-3: Waiting for conversion to finish...
SYN-3: Waiting for conversion to finish...
SYN-3: Waiting for conversion to finish...
Thu Jun  9 22:29:59 2011: Done: Creating Single Instance Attachment table
Thu Jun  9 22:29:59 2011: Start: Locking multiserver capability
Thu Jun  9 22:29:59 2011: Done: Locking multiserver capability
Thu Jun  9 22:29:59 2011: Start: Creating Addressbook Changes table
Thu Jun  9 22:29:59 2011: Done: Creating Addressbook Changes table
Thu Jun  9 22:29:59 2011: Start: Updating 'singleinstances' table to correct tag value
Thu Jun  9 22:29:59 2011: Done: Updating 'singleinstances' table to correct tag value
Thu Jun  9 22:29:59 2011: Start: Create table: synced messages
Thu Jun  9 22:29:59 2011: Done: Create table: synced messages
Thu Jun  9 22:29:59 2011: Start: Force Addressbook Resync
Thu Jun  9 22:29:59 2011: Skipped: Force Addressbook Resync
Thu Jun  9 22:29:59 2011: Start: Rename objecttype columns to objectclass
Thu Jun  9 22:29:59 2011: Done: Rename objecttype columns to objectclass
Thu Jun  9 22:29:59 2011: Start: Convert objecttype columns to objectclass values
Thu Jun  9 22:29:59 2011: Done: Convert objecttype columns to objectclass values
Thu Jun  9 22:29:59 2011: Start: Add object MV property table
Thu Jun  9 22:29:59 2011: Done: Add object MV property table
Thu Jun  9 22:29:59 2011: Start: Link objects in DB plugin through companyid
Thu Jun  9 22:29:59 2011: Done: Link objects in DB plugin through companyid
Thu Jun  9 22:29:59 2011: Start: Update outgoingqueue key
Thu Jun  9 22:29:59 2011: 0x00000009ea8568: SQL Failed: Can't DROP 'PRIMARY'; check that column/key exists, Query: "ALTER TABLE outgoingqueue DROP PRIMARY KEY, ADD PRIMARY KEY (`hierarchy_id`,`flags`,`store_id`)"
Thu Jun  9 22:29:59 2011: Failed: Rollback database
Thu Jun  9 22:30:00 2011: Can't update the database: Unable to upgrade zarafa from version 6.20.18389 to 6.40.8.27223
Thu Jun  9 22:30:00 2011: Server shutdown complete.
SYN-3: Waiting for conversion to finish...

Blijkbaar verwacht zarafa een index die niet bestaat, handmatige de query aanpassen lost het probleem op. (soms moet dit met meerdere indexen)

[Syn-3] root@linux.ours.local ~# mysql -p`cat /etc/my.passwd` zarafa
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 66
Server version: 5.0.91 Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> ALTER TABLE outgoingqueue DROP PRIMARY KEY, ADD PRIMARY KEY (`hierarchy_id`,`flags`,`store_id`);
ERROR 1091 (42000): Can't DROP 'PRIMARY'; check that column/key exists
mysql> ALTER TABLE outgoingqueue ADD PRIMARY KEY (`hierarchy_id`,`flags`,`store_id`);
Query OK, 0 rows affected (0.09 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> ALTER TABLE outgoingqueue DROP PRIMARY KEY, ADD PRIMARY KEY (`hierarchy_id`,`flags`,`store_id`);
Query OK, 0 rows affected (0.05 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> 

We starten het script nogmaals, in FORCE mode omdat de user conversie al geweest is:

[Syn-3] root@linux.ours.local ~# FORCE=1 ./convert 
Stopping /service/zarafa-server ...OK
SYN-3: Reading user ID converions:
old user id -> username -> new user id
ERROR 1054 (42S22) at line 1: Unknown column 'object_type' in 'where clause'
Something went wrong with user conversion..
SYN-3: Updating database:
query: 
SYN-3: Starting zarafa in conversion mode:
SYN-3: Waiting for conversion to finish...
Thu Jun  9 22:36:18 2011: Done: Update acl key
Thu Jun  9 22:36:18 2011: Start: Update externid in object table
Thu Jun  9 22:36:18 2011: Done: Update externid in object table
Thu Jun  9 22:36:18 2011: Start: Update keys in 'changes' table
Thu Jun  9 22:36:18 2011: Done: Update keys in 'changes' table
Thu Jun  9 22:36:18 2011: Start: Update mvproperties key
Thu Jun  9 22:36:18 2011: 0x0000000a356568: SQL Failed: Incorrect table definition; there can be only one auto column and it must be defined as a key, Query: "ALTER TABLE mvproperties DROP PRIMARY KEY, ADD PRIMARY KEY (`hierarchyid`,`tag`,`type`,`orderid`), DROP KEY `hi`"
Thu Jun  9 22:36:18 2011: Failed: Rollback database
Thu Jun  9 22:36:18 2011: Can't update the database: Unable to upgrade zarafa from version 6.40.27223 to 6.40.8.27223
Thu Jun  9 22:36:18 2011: Server shutdown complete.

Nu gaat het mis op een andere table, we doen dezelfde sql magic:

[Syn-3] root@linux.ours.local ~# mysql -p`cat /etc/my.passwd` zarafa
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 77
Server version: 5.0.91 Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> ALTER TABLE mvproperties DROP PRIMARY KEY, ADD PRIMARY KEY (`hierarchyid`,`tag`,`type`,`orderid`), DROP KEY `hi`;
ERROR 1075 (42000): Incorrect table definition; there can be only one auto column and it must be defined as a key
....
mysql> ALTER TABLE mvproperties  DROP KEY `hi`;
Query OK, 103 rows affected (0.07 sec)
Records: 103  Duplicates: 0  Warnings: 0

Na lang prutsen blijkt het nodig zijn deze key te droppen, hierna start zarafa correct en gaat de conversie verder:

[Syn-3] root@linux.lazytours.local ~# FORCE=1 ./convert 
Stopping /service/zarafa-server ...OK
SYN-3: Reading user ID converions:
old user id -> username -> new user id
ERROR 1054 (42S22) at line 1: Unknown column 'object_type' in 'where clause'
Something went wrong with user conversion..
SYN-3: Updating database:
query: 
SYN-3: Starting zarafa in conversion mode:
Thu Jun  9 22:43:11 2011: Start: Update mvproperties key
Thu Jun  9 22:43:12 2011: Startup succeeded on pid 21456
Thu Jun  9 22:43:12 2011: Done: Update mvproperties key
Thu Jun  9 22:43:12 2011: Start: Update DB plugin group to security groups
Thu Jun  9 22:43:12 2011: Done: Update DB plugin group to security groups
Thu Jun  9 22:43:12 2011: Start: Update DB/Unix plugin sendas settings
Thu Jun  9 22:43:12 2011: Done: Update DB/Unix plugin sendas settings
Thu Jun  9 22:43:12 2011: Ignoring attachment storing conflict as requested. Attachments are now stored with option 'files'
Thu Jun  9 22:43:12 2011: Listening for pipe connections on /var/run/zarafa
Thu Jun  9 22:43:12 2011: Listening for TCP connections on port 236
Thu Jun  9 22:43:12 2011: Startup succeeded on pid 21456
./convert: line 52: 21472 Terminated              tail -f /var/log/zarafa/server.log
SYN-3: Converting attachments from database to files
The size of all attachments in the database is: 5719539712 Bytes (5.33 GB)
Available space is: 466469113856 Bytes (434.43 GB)
Finding all attachments...
Processing 32922 attachments

Bovenstaande problemen heb ik nu 2x achter elkaar gehad, dus het lijk een standaard issue met 6.20->6.40.9 conversies. (de 6.20 database stamt wel af van een nog veel oudere zarafa versie die meerdere malen geupgrade is)

Gepost op zarafa forum: http://forums.zarafa.com/viewtopic.php?f=10&t=6783