Cassandra cluster Rolling upgrade

The time has come and a new major release of our Cassandra DB is out there, with fancy new features and important patches. The challenge in a running cluster with a number of clients is always the downtime. That's the reason we always want to spent the minimum time possible with our nodes unavailable and with the correct keyspace replication it is achievable.

Since Datastax documentation is mostly outdated and there is no single document with a complete guide here is the procedure I recently followed to upgrade DSE from version 4.6.2 to 4.6.5.

First, make sure you have an up-to-date daily backup. I prefer OpsCenter incremental backup method but there are other options as well like sstable2json. Then the second step is to upgrade the nodes one-by-one maintaining cluster availability while updating version. Finally, we have to make sure to upgrade all the version nodes. Different cassandra versions in the same cluster can lead to weird incompatibilities.

Step 1

Run node drain which flushes all memtables from the node to SSTables on disk. Cassandra stops listening for connections from the client and other nodes. You typically use this command before upgrading a node to a new version of Cassandra.

nodetool drain -h hostname -u username -pw password

Step 2

Stop the running services, in my case:

dse service
datastax-agent

sudo service dse stop
sudo service datastax-agent stop

Step 3

Keep cassandra, or any other valuable configuration in a safe directory. Remember "Precaution is better than cure"!

cp /etc/dse/cassandra/cassandra.yaml ./cass-upgrade-conf/
cp /etc/dse/cassandra/cassandra-env.sh ./cass-upgrade-conf/

Step 4

Update the package manager repositories and upgrade.

sudo apt-get update
sudo apt-get install dse-full

This can take a while... PS: If you are running Cassandra from source you should download the latest tar.gz instead of using the package manager.

Step 5

If you want to restore any configuration files that you previously backup-up now it's the appropriate time.

Step 6

Start the services.

sudo service dse restart
sudo service datastax-agent start
dse -v

Always check the logs for any error messages.

tail -f /var/log/cassandra/system.log

If you are changing major version

When you are upgrading from a major version (for example, from 1.2 to 2.0) or a major point release (for example, from Cassandra 2.0 to 2.1), upgrade the sstables on each node.

nodetool upgradesstables

Upgrading OpsCenter

Be sure that OpsCenter is compatible with your version of DataStax Enterprise or Cassandra.

Download and extract the new tarball.

1 curl -L http://downloads.datastax.com/community/opscenter.tar.gz | tar xz

Copy the following files and directories from the old tarball installation directory to the new one.

1 conf/clusters/*
2 conf/event-plugins/*
3 conf/install_id
4 conf/log4j.properties
5 conf/opscenterd.conf
6 ./passwd.db
7 conf/ssl.conf
8 ssl/*

Stop the opscenterd instance (if it is running) and start it from the new tarball installation directory.

Extra useful configuration

Time synchronisation between nodes

1 date
2 sudo apt-get install ntp ntpdate
3 sudo service ntp stop
4 sudo ntpd -gq
5 sudo service ntp start

Recommended Production Settings

 1 'echo' 0 > /proc/sys/vm/zone_reclaim_mode
 2 'ulimit' -a
 3 /etc/security/limits.d/cassandra.conf
 4 cassandra - memlock unlimited
 5 cassandra - nofile 100000
 6 cassandra - nproc 32768
 7 cassandra - as unlimited
 8 /etc/sysctl.conf --> vm.max_map_count = 131072 --> sudo sysctl -p
 9 cat /proc/<cassandra pid>/limits
10 sudo swapoff --all