Loading page…
Key messages onto partitions, split partitions across a group's consumers, commit the next offset to read, rebalance on join and leave, and replay from the last commit after a crash.
A stream keeps every message in an append-only log, split into partitions. A message's key picks its partition, so one key's messages stay in order on one partition (Day 1, section 3.3). A consumer group reads the whole topic: each partition goes to exactly one consumer of the group, and each group has its own read positions, so billing and analytics both see every message without getting in each other's way (section 2.3).
Build a simplified, in-memory model of Kafka-style consumer groups: Topic(num_partitions).
| Method | Does |
|---|---|
append(key, value) |
writes to partition partition_for(key, num_partitions) (given) and returns (partition, offset). Offsets start at 0 and count up per partition |
end_offset(partition) |
the offset the next appended message on that partition will get |
join(group, consumer_id) / leave(group, consumer_id) |
a consumer joins or leaves (or crashes): the group rebalances. Joining twice or leaving when not a member does nothing |
assignment(group, consumer_id) |
the consumer's partitions, ascending ([] for an idle or unknown consumer) |
poll(group, consumer_id, max_records=100) |
up to max_records records from the consumer's partitions, each {"partition", "offset", "key", "value"}, taking each partition in ascending order and its records in offset order. Advances the consumer's in-memory position; commits nothing |
commit(group, consumer_id, offsets=None) |
stores the group's committed offsets: the consumer's current positions for all its partitions, or the {partition: offset} you pass |
committed(group, partition) |
the group's committed offset for the partition, 0 if it has none |
The rules, Kafka's semantics in miniature:
num_partitions // consumers consecutive partitions, and the first num_partitions % consumers consumers one extra (Kafka's range assignor does this per topic). 6 partitions over A, B, C is [0, 1], [2, 3], [4, 5]; consumers beyond the partition count get nothing and sit idle.NotAssignedError (given). This is what stops a consumer that lost a partition in a rebalance from committing over the new owner's progress.In JavaScript and TypeScript: new Topic(numPartitions) with append, endOffset, join, leave, assignment, poll(group, consumerId, maxRecords = 100), commit(group, consumerId, offsets?) and committed. append returns [partition, offset], and offsets is a plain object such as { 0: 5, 1: 2 } (its keys arrive as strings: convert them with Number).
Real Kafka spreads partitions over brokers with replication and leader election; consumers find their group through a group coordinator, heartbeat, time out and rebalance through a join/sync protocol with generation numbers; committed offsets live in the internal __consumer_offsets topic; logs have retention and compaction; the default partitioner hashes keys with murmur2 (this lab uses the given FNV-1a); there are other assignors (round-robin, sticky, cooperative sticky) and an auto.offset.reset setting for groups with no commit (this lab always starts at 0). Here everything is one process, one topic, and every call is atomic.
Write your code, then run 3 example tests and 10 hidden tests. Each run is limited to 8 seconds: an infinite loop gets stopped and the runtime restarts, your code stays.