Last Month in Nakadi: April 2018

This is the third instalment in the series of blog posts, “Last Month in Nakadi”, where I try to give some more details on new features and bug fixes that were released over the previous month.

New URL

Nakadi now has its own domain name! You can check out https://nakadi.io

[Fixes] Don’t log a complete stack trace when a resource does not exist

Pull Request 1
Pull Request 2

These two fixes are very similar. We found that, when a user tries to perform an operation on a resource that does not exist, Nakadi logs a complete stack trace. This is unnecessary, and can be an issue if Nakadi processed a lot of requests for resources that don’t exist: disks may fill up. The first fix is for non-existing subscriptions, and the second one, for non-existing event types.

Now, we just log one line to describe the error.

[Fix] Fix event parsing during production

Pull Request

So far, Nakadi has been parsing events for validation, then storing them to a stream of bytes for Kafka. The issue was that numbers might change between the user-provided format and the one exported by the json library (e.g., 0.0 would become 0). This fix ensures that Nakadi does not modify the users’ events, except for enrichment of course.

[New] Subscriptions status

Pull Request

It is now possible to get the status of each partition in a subscription, from the subscriptions endpoint. This is useful for monitoring, so users know which partitions are assigned to which consumers, if any. With the introduction in March of the ability to consume from specific partitions, this is even more valuable.

This is not really a new feature, as it was already available in the subscription’s /stats endpoint. However, this one is much faster, as it does not try to compute the number of unconsumed events – an expensive operation.

To use it, just set the show_status flag to true in your request to the subscriptions endpoint.

[Fix] Multiple bug fixes

Pull Request 1 
Pull Request 2
Pull Request 3

Finally, we released 3 bug fixes together with the subscription status feature: the first one fixes a bug that occurred when committing offsets for a new subscription under specific circumstances.

The second one fixes a bug that could occur when users tried to consume from busy event types with large values for batch_limit and max_uncommitted_events.

The third one is for consumers who reach the stream_timeout they have set. At that point, Nakadi should flush the events it has accumulated so far, before closing the stream.

 

And that’s it for April. If you would like to contribute to Nakadi, please feel free to browse the issues on github, especially those marked with the “help wanted” tag. If you would like to implement a large new feature, please open an issue first to discuss it, so we can all agree on what it should look like. We very much welcome all sorts of contributions: not just code, but also documentation, help with the website, etc.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s