Last Month in Nakadi: April 2018

This is the third instalment in the series of blog posts, “Last Month in Nakadi”, where I try to give some more details on new features and bug fixes that were released over the previous month.


Nakadi now has its own domain name! You can check out

[Fixes] Don’t log a complete stack trace when a resource does not exist

Pull Request 1
Pull Request 2

These two fixes are very similar. We found that, when a user tries to perform an operation on a resource that does not exist, Nakadi logs a complete stack trace. This is unnecessary, and can be an issue if Nakadi processed a lot of requests for resources that don’t exist: disks may fill up. The first fix is for non-existing subscriptions, and the second one, for non-existing event types.

Now, we just log one line to describe the error.

[Fix] Fix event parsing during production

Pull Request

So far, Nakadi has been parsing events for validation, then storing them to a stream of bytes for Kafka. The issue was that numbers might change between the user-provided format and the one exported by the json library (e.g., 0.0 would become 0). This fix ensures that Nakadi does not modify the users’ events, except for enrichment of course.

[New] Subscriptions status

Pull Request

It is now possible to get the status of each partition in a subscription, from the subscriptions endpoint. This is useful for monitoring, so users know which partitions are assigned to which consumers, if any. With the introduction in March of the ability to consume from specific partitions, this is even more valuable.

This is not really a new feature, as it was already available in the subscription’s /stats endpoint. However, this one is much faster, as it does not try to compute the number of unconsumed events – an expensive operation.

To use it, just set the show_status flag to true in your request to the subscriptions endpoint.

[Fix] Multiple bug fixes

Pull Request 1 
Pull Request 2
Pull Request 3

Finally, we released 3 bug fixes together with the subscription status feature: the first one fixes a bug that occurred when committing offsets for a new subscription under specific circumstances.

The second one fixes a bug that could occur when users tried to consume from busy event types with large values for batch_limit and max_uncommitted_events.

The third one is for consumers who reach the stream_timeout they have set. At that point, Nakadi should flush the events it has accumulated so far, before closing the stream.


And that’s it for April. If you would like to contribute to Nakadi, please feel free to browse the issues on github, especially those marked with the “help wanted” tag. If you would like to implement a large new feature, please open an issue first to discuss it, so we can all agree on what it should look like. We very much welcome all sorts of contributions: not just code, but also documentation, help with the website, etc.

Last Month in Nakadi: March 2018

This is the second instalment in the series of blog posts, “Last Month in Nakadi”, where I try to give some more details on new features and bug fixes that were released over the previous month.

March saw an important dependency update, as well as a new feature. The former is thanks to our colleague Peter Liske, who has been working on the issue for quite some time.

JSON-schema validation library now uses RE2/J for regex pattern matching

Peter alerted us about the problem, and fixed it upstream. It turns out that a well-crafted regular expression in a schema could become a regex bomb, when used to evaluate a simple string. Peter demonstrated how easy it would be to “kill”one instance of Nakadi with a single message for several minutes – and kill a whole cluster by sending a sufficient number of messages.

The issue is with the default (PCRE) regex matching library used in Java. Peter swapped it for the RE2/J library in the dependency we use for json-schema validation, and now Nakadi can survive evaluating even the nastiest of regular expressions.

New feature: select which partitions to read from in a subscription

In February, we made the decision to deprecate the low-level API in Nakadi. It is currently still supported, but will be removed in the future. The subscription API did not cover one common use case that the low-level API provided: the ability for a consumer to choose which partitions to consume events from. For some users, it is important to make sure that all events in a given partition will be consumed by the same consumer. Perhaps they do some form of de-duplication, aggregation, or re-ordering, and such a feature makes their job a lot easier.

When we announced the deprecation of the low-level API, we promised to implement that feature in the subscriptions API, to allow users to migrate without issues. This is now done, and users can check out the relevant part of the documentation.

Here is a simple example of how it works. The “usual” way to consume from the subscription API is by creating a stream to get events from a subscription. Nakadi will automatically balance partitions between the consumers connected to the subscription, so that each partition is always connected to exactly one consumer. Given a subscription with ID 1234, it works like this:

GET {nakadi}/subscriptions/1234/events

Pretty simple. Now, if you want to specify which specific partitions you want to consume form, you need to send a “GET with body” (so, a POST) request, and specify the partitions you want to the body. For example, if you want to get partitions 0 and 1 from event type my-event-type, you would do something like this:

POST {nakadi}/subscriptions/1234/events -d '{"partitions": [{"event-type": "my-event-type", "partition": "0"},{"event-type": "my-event-type", "partition": "1"}]}'

Simple. And of course, you can have both types of consumers simultaneously consuming from the same connection. In this case, the rebalanced consumers will share the partitions that have not been requested specifically.

And that’s it for March. If you would like to contribute to Nakadi, please feel free to browse the issues on github, especially those marked with the “help wanted” tag. If you would like to implement a large new feature, please open an issue first to discuss it, so we can all agree on what it should look like. We very much welcome all sorts of contributions: not just code, but also documentation, help with the website, etc.

Last Month in Nakadi: February 2018

I’m experimenting with a new series of posts, called “Last Month in Nakadi”. In the Nakadi project, we maintain a changelog, that we update on each release. Each entry in the file is a one-line summary of a change that was implemented, but that alone is not always sufficient to understand what happened. There is still a fair amount of discussion and context that stays hidden inside Zalando, but we are working on changing that too.

Therefore, I will try, once a month, to provide some context on the changes that we released the month before. I hope that users of Nakadi, and people interested in deploying their own Nakadi-based service, will find this summary useful. Let’s start then, with what we released last month, February 2018.


Released on the 15th of February, this version includes one bug fix, and one performance improvement.

Fix: Problem JSON for authorization issues

A user of Nakadi reported that Nakadi does not provide a correct Problem JSON when authorization has failed.

Improvement: subscription rebalance

We found that, when rebalancing a subscription, Nakadi calls Zookeeper several times, which is costly. This improvement reduces the number of calls to Zookeeper when rebalancing subscriptions, improving the speed of rebalances.


Released on the 22nd of February, this version brings a new feature: the ability to allow a set of applications to get read access to all event types, overriding individual event types’ authorization policies, for archival purposes.

At Zalando, we maintain a data lake, where data is stored and made available to authorised users for analysis. One of the preferred ways to get data into the data lake is to push it to our deployment of Nakadi. Events are then consumed by the data lake ingestion applications, and saved there. Over time, we have noticed that event type owners, when setting or updating their event types’ authorisation policies, would on occasion forget to whitelist the data lake applications, causing delays in data ingestion. Another issue we noticed is that, should the data lake team use a different application to ingest data (they actually use several applications, working together), they would have to contact the owners of all event types from which data is ingested – that’s a lot of people, and a huge burden.

So, we decided to allow these applications to bypass the event types’ authorization policies, such that event type owners would not accidentally block the data lake’s read access. In a future release, we could add a way for the event type owner to indicate that they do not want their data ingested into the data lake.

We also added an optional warning header, sent when an event type is created or updated. We use it to remind our users that their data may be archived, even if the archiving application is not whitelisted for their event type. You can choose the message you want – or no message at all.

And that’s it for February. If you would like to contribute to Nakadi, please feel free to browse the issues on github, especially those marked with the “help wanted” tag. If you would like to implement a large new feature, please open an issue first to discuss it, so we can all agree on what it should look like. We very much welcome all sorts of contributions: not just code, but also documentation, help with the website, etc.
















What Has Our Team Been Up To?

Around the start of this year, I created this blog, to replace my static website. Since then, I have mostly been writing about talks I have given, and I have a few posts in preparation that detail what I am working on (in case you didn’t figure it out yet, it’s called Nakadi.)

Some of my colleagues have already written, on Zalando’s tech blog, about some of the things that we do in our team. Not only do we work on Nakadi, but we also operate it as a service, running on AWS. They wrote about some of the challenges we met, and how we tackled them. We are happy to report that, even when on call, we sleep very well at night: our services are pretty resilient, and out-of-hours calls are the exception, rather than the rule.

Last year, Andrey wrote about his work with Kafka and EBS volumes. We keep a lot of data in Kafka. It used to be that, every time we lost a Kafka broker, or every time we had to restart one, the broker would have to collect once again all its data from other replicas in the cluster. This would take a long time, and while the data was being replicated, we would only have two in-sync replicas for each of the partitions on that broker. Upgrading Kafka would take a week, during which brokers would use some bandwidth – and IO – to replicate data. Andrey solved the issue by making sure that Kafka’s data is stored on a permanent EBS volume, that does not get destroyed when the instance it is attached to goes down. He then worked on upgrade and recovery scripts, such that new brokers will automatically attach previously detached volumes, which greatly reduces the amount of data to synchronise: we only need to copy whatever had been written to Kafka since the broker went down. His work saved us, and continues to save us, considerable amounts of time. It also reduces dramatically the amount of time during which a number of partitions have less than 3 in-syn replicas.

Another post on our work was early this year, by Ricardo. He talked about how he solved one of our biggest pain points in terms of operations: Zookeeper. For the longest time, we were absolutely terrified of a Zookeeper node going down: it would come back, of course, but with a different IP address, and Kafka only takes a fixed list of IP addresses for Zookeeper. Losing a Zookeeper node is not the end of the world, of course, since we run an ensemble of 3. But it did require a rolling restart of Kafka (and a redeployment of Nakadi), which is a time-consuming operation. Losing 2 Zookeeper nodes would have been a catastrophe, but fortunately that hasn’t happened. In his work, Ricardo focused on making sure that Zookeeper nodes always get the same, private, IP address (EIPs were not an option for us). So now, when a Zookeeper node goes down, we know that it will be back a couple of minutes later, with the same address. No more rolling restarts of Kafka!

Last, but not least, Sergii very recently started writing about his previous experience with security while working for an airport in Ukraine. Go read it, it is both instructive and funny. I’m really looking forward to episode 3!

Stay tuned for more news from team Aruha (that’s our name!)

Talk: Nakadi at Zalando’s first Kafka meetup

Back in November, a few colleagues and I (actually, our entire team) organised our first Kafka meetup. In this meetup we wanted to bring together engineers and devops who run software around Kafka, or maintain Kafka in production, to exchange knowledge and discuss our experiences. We wanted to talk about successes as well as failures and challenges. No sales pitches, just the truth of what we have struggled with.

For this first edition, all speakers were engineers at Zalando, as we didn’t know how much interest there would be from outside. We had short talks (10 minutes each, plus another 5 minutes for questions), and we had seven of them (yes, seven).

After an introduction by our team’s engineering lead, Himanshu Gahlaut, I talked about Nakadi for a bit. My colleague Ricardo de Cillo then talked about operating Kafka on AWS. He talked about choosing the right EC2 instances, the size of the cluster, the amount of disk space to use; failures, and how to recover from them; and configuring Kafka to run smoothly on virtual machines that could get terminated at any moment.

Dmitry Sorokin then spoke about Bubuku, our open source supervisor for running Kafka on AWS. Bubuku is a very interesting supervisor, with a lot of features. Not only can it control individual brokers, but it can also trigger rolling restarts of an entire cluster, calculate a fair distribution of partitions among brokers and trigger the appropriate rebalance operations, and much more.

After the break, Andrey Dyachkov discussed how we upgrade Kafka brokers without losing the broker’s data, and how the same mechanism can deal with a brokers that gets terminated in the middle of the night. Long story short, it comes back with the same storage, without manual intervention.

The next speaker was Max Schulze, from a team we work with very closely. He works on Zalando’s data lake, and talked about some aspects of how it was built.

Daniel Truemper, from yet another team, talked about how they operate Kafka for communication between microservices.

All the talks were recorded. As far as I know the video is not yet available online, but hopefully that will change soon. Unfortunately, something was wrong with the projector, and the slides were displayed with a very strong shade of alien green. I wish I had taken pictures. You’ll see it on the video. It’s really green.

The meetup turned out to be very successful: the room was full, and the feedback we got during and after the event was very positive (we’ll try to have more, and colder, beer next time). So we decided to have another one. We’ll have a bit less talks, to leave more time for discussions, but we will keep the 10-15 minutes per talk format. Watch out for the annoucement, it should be out around the end of February!

Feb 3rd: At FOSDEM to talk about Nakadi

Back when I was studying in Belgium, I religiously attended FOSDEM – the Free and Open Source Software Developers’ European Meeting, every year, in Brussels. In fact, as a member of the NamurLUG, I was part of the team that recorded the talks at FOSDEM for quite a few years. Initially we recorded with consumer-grade cameras, but we soon upgraded to better quality equipment, and even started streaming the events live, after a couple of years. Since then, another team has taken over, and the quality of the recording has improved quite a lot from our very amateur debuts.

This year, I will be back at FOSDEM, but this time I’ll be on the other side: I will give a Lightning Talk about Nakadi, the Event Broker I work on at Zalando. Nakadi is Open Source Software, and provides a RESTful API on top of Kafka-like queues (we have plans to support Kinesis in the future), as well as a bunch of other features: schema validation with json-schema, schema evolution, per-event type authorization, and more. In this talk I will focus on one of my favourite features: Timelines. What is Timelines? Well, I guess you’ll have to watch my talk to find out (or wait for the blog post explaining it, I am working on one)! If you can’t make it to Brussels for FOSDEM, the talk will be recorded and streamed live.

Some of my colleagues will also speak at FOSDEM, and more will be in attendance. Oleskii Kliukin and Jan Mußler will give a talk called “Blue elephant on-demand: Postgres + Kubernetes” in the Postgres devroom. And Ferit Topcu will talk about “Automating styleguides with DocumentJS” in the “tool the docs” devroom.

See you all in Brussels in February!

Data Natives Panel: Open Source Data Projects in Berlin

Back in November, I was at the Data Natives conference in Berlin, to take part in a panel on ‘Open Source Data projects in Berlin’. The other panelists were Kostas Tzoumas, Co-Founder and CEO of data Artisans, Ines Montani, Founder of Explosion AI and spaCy developer, and Andreas Dewes, Founder at 7scientists GmbH. The panel was moderated by Dr. Kristian Rother, Python Trainer at Academis. I was there, representing Zalando, and in particular my team, that maintains two open source projects, Nakadi and Bubuku.

All of us are involved in open source software development, so it will be a surprise to nobody that we all agreed that open source is a good thing. I guess it’s getting hard to find people that disagree with that statement, these days. Still, our approach to open source software differs. The way we do open source at Zalando, a relatively large tech company, is quite different from the way a two-people consultancy does it.

In our discussion, we had an interesting exchange on a variety of questions, such as the reasons for a company to open source software, licensing, building and managing a community, and many more. The talk was recorded, you can find the video below.