etcd3 leader election using Python

For a recent Sandtable hackathon, we built a prototype of a highly-available cluster (master-worker) framework. We decided to use etcd3 to store master state, for leader election, and implement a presence protocol. Etcd, developed by CoreOS, is a distributed, reliable key-value store written in Golang that implements the Raft consensus protocol.

For leader election, we found an example using Python for etcd v2 but not v3. Since the API has changed substantially, including, for example, leases and mini-transactions, we wanted to share an example using the new API. We use etcd3 for our Python client.

The approach is as follows. Each candidate leader attempts to set a common key using a transaction that checks if the key already exists. If not, the candidate leader sets the key with a lease and assumes the leader role. The leader can then do its work, meanwhile refreshing the lease. Should the lease lapse, because the leader fails or revokes the lease, a new election is triggered and one of the followers will become leader. And so on.

Leave a comment

Please prove that you are human: