Add support for streaming to built-in logical replication

Enterprise / PostgreSQL - Amit Kapila [postgresql.org] - 3 September 2020 02:24 UTC

To add support for streaming of in-progress transactions into the built-in logical replication, we need to do three things:

- Extend the logical replication protocol, so identify in-progress transactions, and allow adding additional bits of information (e.g. XID of subtransactions).

- Modify the output plugin (pgoutput) to implement the new stream API callbacks, by leveraging the extended replication protocol.

- Modify the replication apply worker, to properly handle streamed in-progress transaction by spilling the data to disk and then replaying them on commit.

We however must explicitly disable streaming replication during replication slot creation, even if the plugin supports it. We don't need to replicate the changes accumulated during this phase, and moreover we don't have a replication connection open so we don't have where to send the data anyway.

Author: Tomas Vondra, Dilip Kumar and Amit Kapila

464824323e Add support for streaming to built-in logical replication.
doc/src/sgml/monitoring.sgml | 16 +
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 11 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 46 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 4 +
src/backend/replication/logical/proto.c | 162 +++-
src/backend/replication/logical/worker.c | 952 ++++++++++++++++++++-
src/backend/replication/pgoutput/pgoutput.c | 367 +++++++-
src/bin/pg_dump/pg_dump.c | 18 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 10 +-
src/include/catalog/catversion.h | 2 +-
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/regress/expected/subscription.out | 63 +-
src/test/regress/sql/subscription.sql | 15 +
src/test/subscription/t/015_stream.pl | 98 +++
src/tools/pgindent/typedefs.list | 3 +
23 files changed, 1766 insertions(+), 74 deletions(-)

Upstream: git.postgresql.org


  • Share