Logical replication support for initial data copy

Enterprise / PostgreSQL - Peter Eisentraut [gmx.net] - 23 March 2017 08:55 EDT

Add functionality for a new subscription to copy the initial data in the tables and then sync with the ongoing apply process.

For the copying, add a new internal COPY option to have the COPY source data provided by a callback function. The initial data copy works on the subscriber by receiving COPY data from the publisher and then providing it locally into a COPY that writes to the destination table.

A WAL receiver can now execute full SQL commands. This is used here to obtain information about tables and publications.

Several new options were added to CREATE and ALTER SUBSCRIPTION to control whether and when initial table syncing happens.

Change pg_dump option --no-create-subscription-slots to--no-subscription-connect and use the new CREATE SUBSCRIPTION ... NOCONNECT option for that.

Author: Petr Jelinek

7c4f524 Logical replication support for initial data copy
contrib/file_fdw/file_fdw.c | 5 +-
doc/src/sgml/catalogs.sgml | 78 ++
doc/src/sgml/config.sgml | 25 +
doc/src/sgml/logical-replication.sgml | 55 +-
doc/src/sgml/monitoring.sgml | 9 +-
doc/src/sgml/protocol.sgml | 9 +-
doc/src/sgml/ref/alter_subscription.sgml | 50 +-
doc/src/sgml/ref/create_subscription.sgml | 38 +
doc/src/sgml/ref/pg_dump.sgml | 15 +-
src/backend/catalog/Makefile | 1 +
src/backend/catalog/heap.c | 6 +
src/backend/catalog/pg_publication.c | 4 +-
src/backend/catalog/pg_subscription.c | 282 +++++++
src/backend/catalog/system_views.sql | 1 +
src/backend/commands/copy.c | 23 +-
src/backend/commands/subscriptioncmds.c | 468 ++++++++++--
src/backend/parser/gram.y | 39 +-
src/backend/postmaster/pgstat.c | 6 +
.../libpqwalreceiver/libpqwalreceiver.c | 203 ++++-
src/backend/replication/logical/Makefile | 2 +-
src/backend/replication/logical/launcher.c | 130 +++-
src/backend/replication/logical/relation.c | 7 +
src/backend/replication/logical/snapbuild.c | 85 ++-
src/backend/replication/logical/tablesync.c | 840 +++++++++++++++++++++
src/backend/replication/logical/worker.c | 203 +++--
src/backend/replication/repl_gram.y | 32 +-
src/backend/replication/repl_scanner.l | 5 +-
src/backend/replication/walsender.c | 104 ++-
src/backend/tcop/postgres.c | 5 +-
src/backend/utils/adt/misc.c | 20 +
src/backend/utils/cache/syscache.c | 14 +-
src/backend/utils/misc/guc.c | 12 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_dump.c | 9 +-
src/bin/pg_dump/t/002_pg_dump.pl | 2 +-
src/include/catalog/catversion.h | 2 +-
src/include/catalog/indexing.h | 7 +-
src/include/catalog/pg_proc.h | 5 +-
src/include/catalog/pg_subscription_rel.h | 78 ++
src/include/commands/copy.h | 5 +-
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 13 +
src/include/nodes/replnodes.h | 9 +
src/include/parser/kwlist.h | 1 +
src/include/pgstat.h | 4 +-
src/include/replication/logical.h | 13 +-
src/include/replication/logicallauncher.h | 1 +
src/include/replication/snapbuild.h | 1 +
src/include/replication/walreceiver.h | 67 +-
src/include/replication/walsender.h | 12 +-
src/include/replication/worker_internal.h | 30 +-
src/include/utils/syscache.h | 1 +
src/test/regress/expected/object_address.out | 3 +-
src/test/regress/expected/rules.out | 3 +-
src/test/regress/expected/sanity_check.out | 1 +
src/test/regress/expected/subscription.out | 45 +-
src/test/regress/sql/object_address.sql | 2 +-
src/test/regress/sql/subscription.sql | 11 +-
src/test/subscription/t/001_rep_changes.pl | 36 +-
src/test/subscription/t/002_types.pl | 6 +
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 159 ++++
62 files changed, 2966 insertions(+), 341 deletions(-)

Upstream: git.postgresql.org


  • Share