The time has come and a new major release of our Cassandra DB is out there, with fancy new features and important patches. The challenge in a running cluster with a number of clients is always the downtime. That's the reason we always want to spent the minimum time possible with our nodes unavailable and with the correct keyspace replication it is achievable.
Since Datastax documentation is mostly outdated and there is no single document with a complete guide here is the procedure I recently followed to upgrade DSE from version 4.6.2 to 4.6.5.
First, make sure you have an up-to-date daily backup. I prefer OpsCenter incremental backup method but there are other options as well like sstable2json. Then the second step is to upgrade the nodes one-by-one maintaining cluster availability while updating version. Finally, we have to make sure to upgrade all the version nodes. Different cassandra versions in the same cluster can lead to weird incompatibilities.
Run node drain which flushes all memtables from the node to SSTables on disk. Cassandra stops listening for connections from the client and other nodes. You typically use this command before upgrading a node to a new version of Cassandra.
nodetool drain -h hostname -u username -pw password
Stop the running services, in my case:
- dse service
sudo service dse stop sudo service datastax-agent stop
Keep cassandra, or any other valuable configuration in a safe directory. Remember "Precaution is better than cure"!
cp /etc/dse/cassandra/cassandra.yaml ./cass-upgrade-conf/ cp /etc/dse/cassandra/cassandra-env.sh ./cass-upgrade-conf/
Update the package manager repositories and upgrade.
sudo apt-get update sudo apt-get install dse-full
This can take a while... PS: If you are running Cassandra from source you should download the latest tar.gz instead of using the package manager.
If you want to restore any configuration files that you previously backup-up now it's the appropriate time.
Start the services.
sudo service dse restart sudo service datastax-agent start dse -v
Always check the logs for any error messages.
tail -f /var/log/cassandra/system.log
If you are changing major version
When you are upgrading from a major version (for example, from 1.2 to 2.0) or a major point release (for example, from Cassandra 2.0 to 2.1), upgrade the sstables on each node.
Be sure that OpsCenter is compatible with your version of DataStax Enterprise or Cassandra.
Download and extract the new tarball.
1 curl -L http://downloads.datastax.com/community/opscenter.tar.gz | tar xz
Copy the following files and directories from the old tarball installation directory to the new one.
1 conf/clusters/* 2 conf/event-plugins/* 3 conf/install_id 4 conf/log4j.properties 5 conf/opscenterd.conf 6 ./passwd.db 7 conf/ssl.conf 8 ssl/*
Stop the opscenterd instance (if it is running) and start it from the new tarball installation directory.
Extra useful configuration
Time synchronisation between nodes
1 date 2 sudo apt-get install ntp ntpdate 3 sudo service ntp stop 4 sudo ntpd -gq 5 sudo service ntp start
Recommended Production Settings
1 'echo' 0 > /proc/sys/vm/zone_reclaim_mode 2 'ulimit' -a 3 /etc/security/limits.d/cassandra.conf 4 cassandra - memlock unlimited 5 cassandra - nofile 100000 6 cassandra - nproc 32768 7 cassandra - as unlimited 8 /etc/sysctl.conf --> vm.max_map_count = 131072 --> sudo sysctl -p 9 cat /proc/<cassandra pid>/limits 10 sudo swapoff --all