Data Engineering Meetup, 4th Edition: 25 September 2018

The fourth edition of our data engineering meetup is in exactly two weeks’ time, and we have just put up the event program. If you’re in Berlin on the 25th of September, and interested in data engineering, register quickly on the meetup page. Last time, the event was full in a few hours only! For this edition, you will get to hear about BI, serverless data ingestion, streaming platforms, notebooks, and more, with speakers from Ifesca, Valohai, and Zalando. [Read More]

Last Month in Nakadi: July 2018

This is the sixth installment in the series of blog posts, “Last Month in Nakadi”, where I try to give some more details on new features and bug fixes that were released over the previous month. [New] Log-compacted topics Pull Request 1 Pull Request 2  Pull Request 3 Nakadi now supports log-compacted topics. A feature long available in Kafka can now be used from Nakadi. In a nutshell, a log-compacted topic is a topic where events are published as key-value pairs. [Read More]

Call for Submissions: Berlin Data Engineering Meetup, 25 September 2018

Photo by Kane Reinholdtsen on Unsplash The Berlin Data Engineering Meetup is a quarterly meetup organised by a few crazy people from Zalando’s data services department. The meetup is a venue for engineers to present their ideas, exchange best practices, and candidly talk about failures, accidents, and other catastrophes. What brings people together is their interest for all things data engineering: streaming platforms, machine learning, databases, storage formats, stream processing, etc. [Read More]

Last Month in Nakadi: June 2018

This is the fifth installment in the series of blog posts, “Last Month in Nakadi”, where I try to give some more details on new features and bug fixes that were released over the previous month. [Changed] Reduced logging Pull Request Nakadi logs a lot of stuff. It’s very useful, but also comes with a cost. Recently, we were looking at our logs, and noticed that our SLO logging amounts for a large percentage of our logs. [Read More]

Open Sourcing Nakadi-UI

Almost 2 years ago, my colleague Sergii Kamenskyi started working on a web UI for Nakadi. So far it has been used internally at Zalando, providing our users with an easy way to find out about the data that flows through Nakadi. Last Friday, after getting approval from our open source team, Sergii released nakadi-ui with an open source license, and anyone who deploys Nakadi can now deploy the web UI as well. [Read More]

Last Month in Nakadi: May 2018

This is the fourth installment in the series of blog posts, “Last Month in Nakadi”, where I try to give some more details on new features and bug fixes that were released over the previous month. [New] Admins can set unlimited retention time Pull Request Every user can set and change the retention time of their event types very easily, up to the maximum retention time set by the Nakadi administrators. [Read More]

Data Engineering Meetup, 3rd Edition: Data Engineering for AI, 26 June 2018

The third edition of our data engineering meetup is in just about a month, and we have just put up the event program. If you’re in Berlin on the 26th of June, and interested in data engineering, register quickly on the meetup page. Last time, the event was full in a few hours only! The theme for this edition is ‘data engineering for AI’, and I really look forward to listening to what our speakers have to say on the subject. [Read More]

Last Month in Nakadi: April 2018

This is the third instalment in the series of blog posts, “Last Month in Nakadi”, where I try to give some more details on new features and bug fixes that were released over the previous month. New URL Nakadi now has its own domain name! You can check out https://nakadi.io [Fixes] Don’t log a complete stack trace when a resource does not exist Pull Request 1 Pull Request 2 [Read More]

Last Month in Nakadi: March 2018

This is the second instalment in the series of blog posts, “Last Month in Nakadi”, where I try to give some more details on new features and bug fixes that were released over the previous month. March saw an important dependency update, as well as a new feature. The former is thanks to our colleague Peter Liske, who has been working on the issue for quite some time. JSON-schema validation library now uses RE2/J for regex pattern matching Peter alerted us about the problem, and fixed it upstream. [Read More]

Last Month in Nakadi: February 2018

I’m experimenting with a new series of posts, called “Last Month in Nakadi”. In the Nakadi project, we maintain a changelog, that we update on each release. Each entry in the file is a one-line summary of a change that was implemented, but that alone is not always sufficient to understand what happened. There is still a fair amount of discussion and context that stays hidden inside Zalando, but we are working on changing that too. [Read More]