Kafka Optimization Theorem

One of the foundational theorems in distributed systems is the CAP theorem. It simply explains why a distributed data store can provide only two (CP or AP, since P is a given) of the consistency (C), availability(A), and partition tolerance(P) guarantees. While this dilemma explains the main tradeoffs of distributed systems in the case of network partitions, it is not useful in the day-to-day optimizing of systems for specific performance goals in the happy path scenarios. In addition to consistency and availability, developers have to optimize applications for latency and throughput too. In this post, we will uncover how the main distributed systems forces relate to each other and which are the main primitives influencing these forces in the context of Apache Kafka. We will visualize and formulate these findings in the form of a theorem that can serve as a guide while optimizing Apache Kafka for specific goals.

Kafka Primitives

One of the less-known distributed systems theorems is PACELC which extends on CAP by explaining what else (E) happens when a distributed system is running normally in the absence of partitioning. In that case, one can optimize for latency (L) or consistency (C) (hence the full acronym PAC-E-LC). PACELC explains how some systems such as Amazon’s Dynamo favor availability and lower latency over consistency, and how ACID-compliant systems favor consistency over availability or lower latency.


PACELC theorem for distributed data stores

While PACELC explains the main tradeoffs and distributed data stores, it doesn’t apply as-is to Kafka. Kafka is an event streaming platform used primarily for moving data from one place to another. A better way to look at a Kafka-based system is as a collection of producers, consumers, and topics forming distinct data flows. The consequence of that is, the client application configuration can influence every goal significantly and there are first class optimization goals in streaming applications such as end-to-end latency and throughput. Whether you are aware of it or not, every time you configure a Kafka cluster, create a topic, or configure a producer or a consumer, you are making a choice between latency vs throughput and availability vs durability. To explain why this is the case, we have to look into the main dimensions of Kafka topics and the role client applications play in an event flow.


 Apache Kafka optimization goals


The topic partitions (on the X-axis in our diagram) represent the unit of parallelism in Kafka. On the broker, and the client-side, reads and writes to different partitions can be done fully in parallel. While there are many factors that can limit throughput, generally a higher number of partitions for a topic will enable higher throughput, and a smaller number of partitions will lead to lower throughput.

At the same time, a very large number of partitions will lead to the creation of more metadata that needs to be passed and processed across all brokers and clients. This can impact end to end latency negatively unless more resources are added to the brokers. This is a purposely simplified view of partitions to demonstrate the main tradeoff that the number of partitions leads to in our context.


The replication factor (on the Y-axis in our diagram) determines the number of copies (including the leader) each topic partition in a cluster must have. By ensuring all replicas of a partition exist on different brokers, replicas define the data durability. A higher number of replicas ensure the data is copied to more brokers and ensure better durability of data in the case of broker failure. 

On the other hand, a lower number of replicas reduces the data durability but in certain circumstances it can increase the availability for the producers or consumers by tolerating more broker failures. That said, the availability for a consumer is determined by the availability of in-sync replicas, whereas for a producer it is determined by the minimum number of in-sync replicas. With a low number of replicas, the availability of overall data flow will be dependent on which brokers fail (the one with the partitions leader of interest or not) and if other brokers are in sync or not. For our purpose we assume less replicas would lead to higher application availability and lower data durability.

Partitions and replicas are not the only primitives influencing the tradeoff between throughput vs latency and durability vs availability, but they represent the broker side of the picture. Other participants in shaping the main Kafka optimization tradeoffs are the client applications.

Producers and Consumers

Topics are used by the producers that send messages and consumers that read these messages. Producers and consumers also state their preference between throughput vs latency and durability vs availability through various configuration options. It is the combination of topic and client application configurations (and other cluster-level configurations such as leader election) that defines what your application is optimized for.

We can look at a flow of events consisting of a producer, a consumer, and a topic. Optimizing such an event flow for average latency, on the client-side would require tuning the producer and consumer to exchange smaller batches of messages. The same flow can be tuned for average throughput by configuring the producer and consumer for larger batches of messages. In the same way, the number of topic partitions influences throughput vs latency, a producer and consumer message batch sizes influence the same.

Producers and consumers involved in a message flow state preference for durability or availability too. A producer that favors durability over availability can demand a higher number of acknowledgements by specifying acks=all. A lower number of acknowledgments (lower than minISR which means 0 or 1) could lead to higher availability from the producer point of view by tolerating a higher number of broker failures, but reduces data durability in the case of catastrophic events with the brokers.

The consumer configurations influencing this dimension (Y-axis) are not as straightforward as the producer configurations but dependent on the consumer application logic. A consumer can favor higher consistency by committing message consumption offsets more often or even individually. Or the consumer can favor availability by increasing various timeouts and tolerating broker failures for longer.

Kafka Optimization Theorem

We defined the main actors involved in an event flow as a producer, a consumer, and a topic. We also defined the opposing goals we have to optimize for: throughput vs latency and durability vs availability. Given that, the Kafka Optimization Theorem states that any Kafka data flow makes tradeoffs between throughput vs latency and durability vs availability.