> Jetstream clusters don't scale to many servers (they recommend max 3, I think)
You can have clusters with many servers in them, 3 is actually the minimum required if you want fault-tolerance, that's how you scale JetStream horizontally: you let the stream (which can be replicated 1, 3 or 5 times) spread themselves over the servers in the cluster.
JetStream consumers create state on the server (or servers if they are replicated) and are either durable with a well known name or 'auto-cleanup after idle time' (ephemeral), and indeed allow you to ack/nack each message individually rather than just having an offset.
However, that is with the exception of 'ordered consumers', which are really the closest equivalent to Kafka consumers in that the state is kept in the client library rather than on the servers. They deliver the message in order to the client code, they take care of re-deliveries and recovering from things like getting disconnected from a server, and only ever deliver messages in order, no need to explicitly ack the messages, and if you want to persist your offset (which is the sequence number of the last message the ordered consumer delivered) just like the consumer group clients in Kafka persist their offset in a stream, you would persist your offset in a NATS KV bucket.
And indeed you can now even go further and use batched direct gets to get very good read speed from the stream and no extra server state in the server besides an entry in the offset KV, performance of the batched direct gets is very high and can match the ordered consumer's speed. Besides no incurring no server state, another advantage of stateless consuming is that all the servers replicating the stream will be used to process direct get requests not just the currently elected leader (don't forget to enable direct gets for the stream, it's not on by default). So you can scale the read throughput horizontally by increasing the number of replicas.
The mechanics of replication: streams and stateful consumers can be replicated using 1, 3 or 5 servers. Servers connect directly together to form a cluster and jetstream assets (streams/consumers) are spread out over the servers in the cluster. Clusters can be connected together for form super-clusters. Super-cluster means that access to JetStream assets is transparent: streams/consumers located in one cluster can be accessed from any other cluster. You can have streams that mirror or source from other streams, those mirrors could be located in other clusters to offer faster local access. You can easily move on the fly JS assets from one cluster to another. Leaf nodes are independent servers (which can be clustered) that connect to a cluster like a client would. Being independent means they have their own security for their own clients to connect to them, they can have their own JS domain and you can source to/from streams between the leaf node's domain and the hub (super-cluster). Leaf nodes can be daisy chained.
Sorry, what I meant that each stream (which forms a Raft group) doesn't scale to more. I thought it was 3, but thanks for the correction.
Everything else you wrote confirms what I wrote, no? As for batch direct gets, that's great, but I'm not sure why you didn't go all the way and offered a Kafka-type consumer API that is strictly ordered and persists the offset natively. I've indeed written an application that uses ordered consumers and persists the offset, but it is cumbersome.
Every time I've used Jetstream, what I've actually wanted was the Kafka model: Fetch a batch, process the batch, commit the batch. Having to ack individual messages and worry about AckWait timeouts is contrary to that model. It's a great programming model for core NATS, but for streams I think you guys made a design mistake there. A stream shouldn't act like pub/sub. I also suspect (but can't prove) that this leads to worse performance and higher load on the cluster, because every message has to go through the ack/nack roundtrips.
I'd also like to point out that Jetstream's maximum message size of 1MB is a showstopper. Yes, you can write big messages somewhere else and reference them. But that's more work and more complexity. With Kafka/Redpanda, huge messages just work, and are not particularly a technical liability.
> Sorry, what I meant that each stream (which forms a Raft group) doesn't scale to more. I thought it was 3, but thanks for the correction.
Streams can have more than 3 replicas. Technically they can have any number of replicas but you only get extra HA when it's an odd number (e.g. 6 replicas doesn't offer more HA than 5, but 7 does). Typically the way people scale to more than one stream when a single stream becomes a bottleneck is by using subject transformations to insert a partition number in the subject and then creating a stream per partition.
Point taken about wanting to have the 'ordered consumer + persist the offset in a KV' built-in, though it should really not be cumbersome to write. Maybe that could be added to orbit.go (and we definitely welcome well written contributions BTW :)).
> Having to ack individual messages and worry about AckWait timeouts is contrary to that model
Acking/nacking individual messages is the price to pay for being able to have proper queuing functionality on top of streams (without forcing people to have create partitions), including automated re-delivery of messages and one-to-many message consumption flow control.
However it is not mandatory: you can set any ack policy you want on a consumer: ackAll is somewhat like committing an offset in Kafka (it acks the sequence number and all prior sequence numbers), or you can simply use ackNone meaning you forgo completely message acknowledgements (but it will still remember the last sequence number delivered (i.e. the offset) automatically).
For example using a pull consumer with ack policy=none and doing 'fetch' to get batches of messages is exactly what you describe what you want to do (and functionally not different from using an ordered consumer and persisting the offset).
And yes, having acks turned on or off on a consumer does have a significant performance impact: nothing comes for free and explicit individual message acking is a very high quality of service.
As for the max message size you can easily increase that in a server setting. Technically you can set it up all the way to 32 MB if you want to use JetStream and up to 64MB if you want to just use Core NATS. However many would advise you to not increase it over 8 or 16 MB because the large the message are the more the potential for things like latency spikes (think 'head of the line blocking') increased memory management, increased slow consumers, etc...
> Jetstream clusters don't scale to many servers (they recommend max 3, I think)
You can have clusters with many servers in them, 3 is actually the minimum required if you want fault-tolerance, that's how you scale JetStream horizontally: you let the stream (which can be replicated 1, 3 or 5 times) spread themselves over the servers in the cluster.
JetStream consumers create state on the server (or servers if they are replicated) and are either durable with a well known name or 'auto-cleanup after idle time' (ephemeral), and indeed allow you to ack/nack each message individually rather than just having an offset.
However, that is with the exception of 'ordered consumers', which are really the closest equivalent to Kafka consumers in that the state is kept in the client library rather than on the servers. They deliver the message in order to the client code, they take care of re-deliveries and recovering from things like getting disconnected from a server, and only ever deliver messages in order, no need to explicitly ack the messages, and if you want to persist your offset (which is the sequence number of the last message the ordered consumer delivered) just like the consumer group clients in Kafka persist their offset in a stream, you would persist your offset in a NATS KV bucket.
And indeed you can now even go further and use batched direct gets to get very good read speed from the stream and no extra server state in the server besides an entry in the offset KV, performance of the batched direct gets is very high and can match the ordered consumer's speed. Besides no incurring no server state, another advantage of stateless consuming is that all the servers replicating the stream will be used to process direct get requests not just the currently elected leader (don't forget to enable direct gets for the stream, it's not on by default). So you can scale the read throughput horizontally by increasing the number of replicas.
The mechanics of replication: streams and stateful consumers can be replicated using 1, 3 or 5 servers. Servers connect directly together to form a cluster and jetstream assets (streams/consumers) are spread out over the servers in the cluster. Clusters can be connected together for form super-clusters. Super-cluster means that access to JetStream assets is transparent: streams/consumers located in one cluster can be accessed from any other cluster. You can have streams that mirror or source from other streams, those mirrors could be located in other clusters to offer faster local access. You can easily move on the fly JS assets from one cluster to another. Leaf nodes are independent servers (which can be clustered) that connect to a cluster like a client would. Being independent means they have their own security for their own clients to connect to them, they can have their own JS domain and you can source to/from streams between the leaf node's domain and the hub (super-cluster). Leaf nodes can be daisy chained.