Replicating Schema Changes with PostgreSQL
Thursday, October 24 at 16:00–16:50
Postgres is great at what it does (and that is a lot), but sometimes you need to extend its reach by using more specialised tools. Making the data available in near real time to those tools is something that Postgres already supports via replication. There are many tools to help you stream data out of Postgres built on replication, but most on the market don’t support evolving schemas.
Database schemas have a tendency to change over time - if your change data capture tool doesn’t support replicating them, you risk data loss and manual intervention (along with late night pages and subsequent headaches) to fix your pipeline.
This talk presents a new approach to data replication, where schema changes are tracked in Postgres and automatically replicated alongside the data, forming a linearised schema change log that is then applied downstream.
This talk covers the ideas behind this approach, as well as some sample use cases:
- DDL replication leveraging SQL queries and triggers
- Postgres replication into Elasticsearch including DDL changes - data mapping alignment and other headaches
- Beyond triggers, webhook support with Postgres replication
- Finally, an introduction to pgstream, an open-source tool that puts these ideas into practice, showing how it can be used to facilitate data replication in Postgres applications.
