What Is Apache Kafka and When Do You Need It?

Apache Kafka is a distributed event-streaming platform that stores ordered records in partitioned topics so multiple applications can consume and replay them independently. Use it when durable event history, high fan-out, parallel processing, or stream transformations justify a cluster. For simple background jobs, a conventional queue is cheaper to operate.

Kafka is often introduced as a fast message broker, but that description misses the design choice that changes how applications use it: consumption does not normally remove an event. The retained log and each consumer’s position are separate concerns.

Kafka in one practical picture

An application publishes an event such as an order accepted, payment authorised, file uploaded, or sensor reading captured. Kafka appends that event to a topic partition. Other applications read it at their own pace and remember how far they have progressed.

The log is shared; progress is private. A billing service, fraud check, analytics job, and audit archive can read the same topic through separate consumer groups. One group can pause or replay without moving another group’s position.

Kafka combines several jobs:

• accepting events from producer applications;
• storing them durably in partitioned topic logs;
• replicating partition data across brokers for fault tolerance;
• assigning partitions to consumers in a group;
• tracking each group’s progress with offsets;
• retaining data long enough for delayed consumption or replay;
• supporting stream processing and data integration around those logs.

That is more than a queue, but it is also more to operate.

The core Kafka terms

These pieces are easy to name. Their boundaries are what prevent expensive surprises.

A topic is retained history, not a work list

A Kafka topic is a named stream. Producers append events, and consumers fetch them. Reading does not by itself delete those events; topic retention decides how long they remain available.

Retention enables independent timing. A new consumer group can begin later, an existing group can catch up after downtime, and a repaired application can replay earlier data while other consumers continue normally.

Retention is not a backup. Accidental deletion, incorrect retention, compromised credentials, operator error, or a cluster-wide failure can still remove or corrupt what you need. Important streams require a recovery and archival decision beyond leaving events in Kafka.

Topics also need ownership. Without it, organisations accumulate unused streams, duplicated events, unclear schemas, excessive retention, and consumers nobody wants to break. Find the idle line item applies to event infrastructure too.

Partitions set both scale and ordering boundaries

A topic is divided into partitions distributed across brokers. Each partition is an ordered log. Producers append each event to one partition, commonly using the event key to keep related events together.

Kafka guarantees order within a partition, not across the whole topic. If all events for one account must be observed in order, a stable account key can route them to the same partition. Events for different keys may proceed independently.

The key is therefore an application decision, not a tuning detail. A poor key can concentrate traffic in one partition, scatter events that need ordering, or make later repartitioning awkward.

Partition count affects:

• maximum parallelism for a traditional consumer group;
• file, metadata, replication, and recovery overhead;
• how events distribute across brokers;
• the impact of hot keys;
• future changes to key-to-partition mapping.

Adding partitions later does not retroactively redistribute old events and can change where a key maps under common partitioning strategies. Choose with measured workload and ordering requirements, not a fashionable default.

Consumer groups and offsets separate readers

Consumers using the same group identity cooperate. Kafka assigns each partition to one consumer in that group at a time, so members divide the work. When membership changes, the group may rebalance and move assignments.

Different groups are independent. A search indexer and a reporting service can each read the complete topic because they use different group identities.

An offset is a reading position. A consumer fetches from a partition position and commits progress for its group. If processing fails before or after that commit, the application can see a duplicate or skip work depending on its order of operations.

Offsets make replay possible. A group can seek to an earlier offset and process retained events again. That is useful for rebuilding derived state, fixing a consumer bug, or populating a new destination.

Replay also repeats side effects unless the application controls them. Sending an email, charging a card, or calling an external API twice is not repaired by moving an offset. Consumers need idempotency keys, deduplication, reconciliation, or destination-specific transactions.

Producers, brokers, and replication keep the stream moving

Producers choose a topic and usually a key, serialise the event, and send it to a broker. Configuration determines acknowledgement, retry, batching, compression, and idempotence behaviour.

Brokers store partition logs and serve producers and consumers. Partition replicas allow another broker to hold a copy. A partition leader establishes the order, while followers replicate that log.

Replication is not the same as processing correctness. It protects against certain broker failures; it does not validate event contents, fix a bad schema, prevent an authorised deletion, or make a consumer’s external side effect atomic.

The cluster also needs configured authentication, encryption, and authorisation. Kafka supports these controls, but unsecured operation remains possible. Treat client identity, topic access, administration, monitoring, and inter-broker traffic as explicit security work.

Kafka and traditional queues overlap

The comparison is not Kafka good, queues bad. Both can distribute work, buffer producers from consumers, retry failures, and support competing workers. The useful difference is what each system makes central.

Kafka consumer groups can behave like competing consumers, and newer group models cover more queue-like workloads. A queue can also offer fan-out and retention. Compare the exact product semantics instead of drawing a hard category line.

Choose the smallest system that preserves the required contract. If one service submits work for one worker pool and completed jobs need no replay, Kafka’s retained distributed log may add cost without adding reader value.

Delivery semantics need fine print

At-most-once means processing may be lost but should not be repeated. At-least-once means retries preserve work but duplicates can occur. Exactly-once means a defined processing result appears once within a defined boundary.

Kafka’s documented default is at-least-once. Idempotent production prevents retry duplicates in the Kafka log under its supported conditions. Transactions can atomically write records to Kafka partitions and update consumed offsets.

Kafka can provide exactly-once read-process-write behaviour when input, output, and offsets participate in the Kafka transaction, as Kafka Streams does. Consumers must use the appropriate committed-data isolation when that guarantee matters.

That promise does not automatically include arbitrary external effects. A database update, payment request, email, or HTTP call cannot be rolled back merely because a Kafka transaction aborts.

For an external destination, coordinate the destination write with progress where possible, use an idempotency key or transactional outbox pattern, and reconcile failures. State the guarantee as a testable boundary instead of writing exactly-once on an architecture slide.

When Kafka earns its operating cost

Kafka becomes useful when several of these conditions are true:

• multiple independent applications need the same event history;
• consumers must replay data or rebuild derived state;
• sustained event flow needs partitioned parallel processing;
• producers and consumers must be decoupled across long backlogs;
• stream joins, aggregations, or transformations are core application work;
• connectors need to move durable changes among several systems;
• per-key ordering matters while unrelated keys can progress independently;
• the team can own schemas, retention, security, lag, upgrades, and recovery.

The business case should name the streams, subscribers, retention purpose, recovery goal, and avoided complexity. High volume alone is not enough; a simpler log, database, or managed queue may still fit better.

When Kafka is excessive

Kafka is probably excessive when the requirement is merely run this job later. A conventional queue often gives a simpler worker model, acknowledgement, retry, delay, and dead-letter path.

It may also be excessive when:

• there is one producer and one consumer with no replay requirement;
• a database transaction and outbox can safely feed a managed queue;
• event volume is modest and delay tolerance is generous;
• global ordering is required but the design cannot express a stable partition key;
• the team has no owner for brokers, storage, upgrades, security, schemas, and lag;
• every consumer immediately writes to an external system with poorly defined idempotency;
• retention is being used as a substitute for backup or data governance;
• the architecture exists mainly to imitate a larger company.

Do not sell mystery hosting to your own engineering team. If nobody can explain the partitions, offsets, recovery path, and monthly operating cost in plain language, the platform is not ready to become critical.

Count the operational bill

Kafka’s cost is not only broker compute. Include durable storage for retained partitions and replicas, network for production, consumption and replication, monitoring, security, upgrades, schema tooling, connectors, and recovery capacity.

Watch consumer lag, under-replicated data, request errors, broker storage, partition distribution, controller health, rebalance behaviour, and client failures. Secure monitoring access; an exposed management interface can become a control path into the cluster.

Retention multiplies storage obligations. A longer window may enable replay, but it also increases broker storage, recovery time, replication traffic, and possibly cross-region transfer. The invoice tells the truth when every retained copy and connector is included.

Managed Kafka can move broker operation to a provider, but application semantics remain yours. Keys, schemas, consumer idempotency, retention, offsets, replay safety, and external side effects do not become correct automatically.

Common questions