Last Month in Nakadi: May 2018

This is the fourth installment in the series of blog posts, “Last Month in Nakadi”, where I try to give some more details on new features and bug fixes that were released over the previous month.

[New] Admins can set unlimited retention time

Pull Request

Every user can set and change the retention time of their event types very easily, up to the maximum retention time set by the Nakadi administrators. From now on, Nakadi administrators can bypass this limitation, and set an arbitrary retention time for event types.

This is useful when a user suffers an incident, that will require them to re-consume data after they fix their software. Sometimes, releasing a fix can take a while, and users can ask administrators to increase the retention time temporarily, to avoid data loss.

[Fix] Improve performance of listings subscriptions with status

Pull Request

Last month, I talked about this new flag in the subscriptions endpoint, that gives the status of each subscription. We have improved the performance of this endpoint by quite a lot, and now the response always comes immediately.

[New] Extend subscription statistics with time-lag information

Pull Request

The subscription stats endpoint (/subscriptions/{subscription_id}/stats) can return a new, optional field: the time lag. The time lag is the time in seconds between when the first unconsumed event in a partition was produced to Nakadi, and the time the request to the stats endpoint is made. To get the time lag, just set `show_time_lag` to `true` in your stats request.

This is really useful for monitoring subscriptions: you can tell how far back you are currently. If the time lag increases for some time, you are probably not catching up with the rate at which events are produced.

[Fix] Provide gzip compression for POST responses if requested

Pull Request

‘GET-with-body’ (so, POST) queries weren’t getting gzip responses if they requested it. This is now fixed.


And that’s it for May. If you would like to contribute to Nakadi, please feel free to browse the issues on github, especially those marked with the “help wanted” tag. If you would like to implement a large new feature, please open an issue first to discuss it, so we can all agree on what it should look like. We very much welcome all sorts of contributions: not just code, but also documentation, help with the website, etc.