A simulations-based performance evaluation of total order protocols

Abstract

Process replication on fail-independent computers is a fundamental requirement for incorporating fault tolerance and thereby guaranteeing service availability in the event of computer failures in distributed systems. A given service request is executed on all process replicas so that even if some replicas crash, the request execution at surviving ones will ensure that the client of that request receives the response. For replication to be effective, however, all replicas must process all requests in the same order so that they all undergo identical state transitions and produce identical responses for a given request, of which any one can be given to the client. To ensure this identical ordering, replication employs a total order protocol that is distributed and crash-tolerant to obtain an ordering sequence on the requests directed at the replicated system. Total order protocols can be typically classified as leader-free and leader-based. LCR and Raft order protocols are well-known leaderless and leader-based categories respectively. The LCR protocol arranges replicas on a logical ring with inter-replica communication supported by unidirectional flow of messages. Due to this structure, it has been proven to offer the highest achievable throughput. Raft is a widely used, leader-based, total order protocol that is simpler and easier to understand than other leader-based protocols and provides a foundation for system implementation. In Raft, one process is designated as the leader and others as followers and the latter obey the ordering decided by the leader; this leads to messages flowing to/from the leader as if the processes are arranged in a star topology with the leader at the centre of topology. Having analysed LCR and Raft protocols [2, 3], we observe that LCR is designed with some potentially performance-limiting assumptions that may not lead to smaller latencies when several processes concurrently forward requests for ordering; these assumptions are: (i) use of a vector clock with one integer for every replica, and (ii), employing a fixed idea of "last" process to order concurrent messages. In addition, in the Raft protocol, all requests are directed at the leader and hence the load at the leader becomes a performance bottleneck as the client request arrival rate at the leader process increases. This bottleneck is exacerbated by the crash tolerance requirement that the leader requires a quorum of acknowledgements from followers before confirming the order it initially placed on a single client request. This thesis seeks to eliminate these performance-limiting design assumptions and propose demonstrably efficient alternative designs. To this end, it makes four contributions. First, we design and evaluate daisy chain total order protocol (DCTOP), a new ring-based leaderless total order protocol using Lamport's logical clocks for sequencing messages, and a novel idea of "last" process that is the closest to the sending process in the opposite direction to message flow. This results in a unique last process rather than a globally fixed one for each process. Secondly, we evaluated DCTOP using a greedy sending approach. This approach ensures processes transmit all their messages first before receiving messages from other processes. We extended this greedy sending approach to include a fairness control approach. fairness is defined as every process Pi has an equal chance of having its sent messages eventually delivered by all processes within the system. To ensure fairness, we modified the fairness control algorithm of the Fixed Sequencer and Ring (FSR) order protocol and applied it to our design. Thus, leading us to implement a fairness-controlled DCTOP. Thirdly, we proposed two variations of Raft, which are all capable of reaching an agreement in fewer communication steps than Raft. We modified the traditional Raft system framework into Chain Raft (RaftCh) and Balanced Fork Raft (RaftBf) variations using a novel idea that restricts a leader to commit a client message sequence number when it receives a single acknowledgement from the last follower from either the chain or fork. The outcome is that the last follower on the chain or fork sending a single acknowledgement for sequence number commitment at the leader process is unique, unlike the quorum of acknowledgements required for the same task in the conventional Raft. The RaftBf offers excellent performance but is only appropriate for cluster environments with five processes or more (ideally, odd numbers of processes). The alternative, RaftCh, on the other hand, is created without this limitation. Finally, a detailed performance evaluation of DCTOP and LCR in a non-fairness-controlled (greedy sending) and a fairness controlled cluster environment is presented. Additionally, Raft and Raft-variant protocols are evaluated, including the evaluation of Raft-variants and DCTOP. Our experiments show that our new approaches offer significant improvements over the existing state-of-the-art methods.