{"dataset":{"id":"00000000-0000-0000-0000-000000000041","slug":"kafka-best-practices","name":"Kafka best practices","description":"Source-linked guidance on Apache Kafka fundamentals: partitions and replication, consumer groups, delivery semantics, retention, and producer configuration. Scoped to Apache Kafka (4.3 documentation); not a general messaging corpus.","visibility":"public","version":1,"documentCount":5,"publisher":"Koskamo editorial","license":"Original summaries; linked sources retain their own terms.","reviewedAt":"2026-10-04T00:00:00.000Z","readOnly":true,"createdAt":"2026-10-04T00:00:00.000Z","updatedAt":"2026-10-04T00:00:00.000Z"},"total":5,"documents":[{"title":"Producer configuration for durability and throughput","sourceUrl":"https://kafka.apache.org/43/configuration/producer-configs/","text":"The acks config controls the durability of sent records: acks=0 means the producer does not wait for any acknowledgment, the record is immediately added to the socket buffer and considered sent, no guarantee can be made that the server received it, and retries do not take effect; acks=1 means the leader writes the record to its local log but responds without awaiting full follower acknowledgement, so the record is lost if the leader fails before followers replicate it; acks=all — equivalent to acks=-1 and the default — means the leader waits for the full set of in-sync replicas, the strongest available guarantee. Idempotence (enable.idempotence, default true) ensures exactly one copy of each message is written, and requires acks=all, retries greater than 0, and max.in.flight.requests.per.connection at most 5; explicitly enabling it with conflicting settings throws a ConfigException. With idempotence off and more than one in-flight request, retries can reorder messages within a partition. For throughput, batch.size sets the upper bound in bytes of a per-partition batch (zero disables batching), and linger.ms, which defaults to 5 since Kafka 4.0, waits for more records before sending. Scope: Apache Kafka. Reviewed October 4, 2026.","id":"kafka-best-practices:Producer configuration for durability and throughput","version":1,"createdAt":"2026-10-04T00:00:00.000Z","updatedAt":"2026-10-04T00:00:00.000Z"},{"title":"Retention with cleanup.policy, retention.ms, and retention.bytes","sourceUrl":"https://kafka.apache.org/43/configuration/topic-configs/","text":"The cleanup.policy config designates the retention policy applied to log segments: the default \"delete\" policy discards old segments when their retention time or size limit is reached, while \"compact\" enables log compaction, retaining the latest value for each key; both can be combined as a comma-separated list such as \"delete,compact\", and an empty list means infinite retention with no cleanup policies applied. Under the delete policy, retention.ms sets the maximum time a log is retained before old segments are discarded — it represents an SLA on how soon consumers must read their data, defaults to 604800000 (7 days), and -1 applies no time limit. retention.bytes caps the maximum size a partition can grow to before old segments are discarded; by default there is no size limit, only a time limit, and because the limit is enforced at the partition level it must be multiplied by the number of partitions to compute topic retention in bytes. Both operate independently of segment.ms and segment.bytes. With remote storage enabled, the local retention limits log.local.retention.ms and log.local.retention.bytes still apply to local segments. Scope: Apache Kafka. Reviewed October 4, 2026.","id":"kafka-best-practices:Retention with cleanup.policy, retention.ms, and retention.bytes","version":1,"createdAt":"2026-10-04T00:00:00.000Z","updatedAt":"2026-10-04T00:00:00.000Z"},{"title":"Message delivery semantics","sourceUrl":"https://kafka.apache.org/43/design/design/#message-delivery-semantics","text":"Kafka's delivery guarantees break down into the durability of publishing and the guarantees when consuming. The three models are at most once (messages may be lost but are never redelivered), at least once (messages are never lost but may be redelivered), and exactly once (each message is processed once and only once). A published message is considered committed only when all in-sync replicas have applied it to their log, and a committed message is not lost as long as one broker replicating that partition remains alive. A producer hitting a network error cannot tell whether the message was committed before or after the failure; before 0.11.0.0 the only option was to resend, yielding at-least-once semantics. Since 0.11.0.0 the producer supports an idempotent delivery option — the broker assigns each producer an ID and deduplicates using a per-message sequence number — and transactions that write to multiple topic partitions atomically. Producers trading durability for latency can send fully asynchronously or wait only for the leader. On the consumer side, saving the position before processing gives at-most-once, while processing before saving the position gives at-least-once. Scope: Apache Kafka. Reviewed October 4, 2026.","id":"kafka-best-practices:Message delivery semantics","version":1,"createdAt":"2026-10-04T00:00:00.000Z","updatedAt":"2026-10-04T00:00:00.000Z"},{"title":"Consumer groups and offsets","sourceUrl":"https://kafka.apache.org/43/design/design/#the-consumer","text":"The Kafka consumer works by issuing fetch requests to the brokers leading the partitions it wants to consume, specifying its offset in the log with each request and receiving back a chunk of log beginning at that position. A topic is divided into totally ordered partitions, each consumed by exactly one consumer within each subscribing consumer group at any given time, so the position of a consumer in each partition is just a single integer — the offset of the next message to consume — which can be periodically checkpointed. This makes the equivalent of message acknowledgements very cheap compared with broker-side per-message state. Because consumers control their position, they can deliberately rewind to an old offset and re-consume data, which is essential when a consumer bug is discovered after messages were consumed. For applications built on the group rebalance protocol, static membership lets group members provide persistent entity ids so restarts and code deploys do not trigger rebalances; using it requires upgrading both the broker cluster and client apps to 2.3 or beyond. Scope: Apache Kafka. Reviewed October 4, 2026.","id":"kafka-best-practices:Consumer groups and offsets","version":1,"createdAt":"2026-10-04T00:00:00.000Z","updatedAt":"2026-10-04T00:00:00.000Z"},{"title":"Partitions and replication","sourceUrl":"https://kafka.apache.org/43/design/design/#replication","text":"The producer sends data directly to the broker that is the leader for a partition; any Kafka node can answer a metadata request about which servers are alive and where each partition's leader currently is. The client controls which partition a message goes to: at random for load balancing, or by a key hashed to a partition — for example, a user id key sends all data for that user to the same partition — and the partition function can be overridden. At its heart a partition is a replicated log: the leader chooses the ordering of values and followers copy it. The guarantee a replication algorithm must provide is that once a client is told a message is committed, any newly elected leader must also have it. Rather than majority vote, Kafka dynamically maintains a set of in-sync replicas (ISR) caught up to the leader; a write is committed only when all in-sync replicas have received it, only ISR members are eligible for election as leader, and with f+1 replicas a topic tolerates f failures without losing committed messages. Scope: Apache Kafka. Reviewed October 4, 2026.","id":"kafka-best-practices:Partitions and replication","version":1,"createdAt":"2026-10-04T00:00:00.000Z","updatedAt":"2026-10-04T00:00:00.000Z"}]}