Schedule - FOSDEM PGDay 2016

Streaming huge databases using logical decoding

Date: 2016-01-29
Time: 16:30–17:20
Room: Hotel
Level: Intermediate

In this talk, I will discuss Zalando's open-source data collection prototype that uses PostgreSQL's logical replication streaming capabilities to collect data from the database and stream it for later processing (Data Lake, Operational Data Store, automatic process monitoring, KPI calculation systems). The process includes taking a consistent snapshot of the database in a distributed and efficient manner and streaming the subsequent changes via a logical replication slot.

This approach can be used for online PostgreSQL major version upgrades, to fill event queues, for partial replication, etc. Use of Python makes it easy to support different target systems and to prototype rapidly.

The audience will come away with new ideas for how to use PostgreSQL logical decoding in daily business.

Speaker

Oleksandr Shulgin

Sponsors