Introduce logical decoding

Enterprise / PostgreSQL - Robert Haas [postgresql.org] - 3 March 2014 15:32 UTC

This feature, building on previous commits, allows the write-ahead log stream to be decoded into a series of logical changes; that is, inserts, updates, and deletes and the transactions which contain them. It is capable of handling decoding even across changes to the schema of the effected tables. The output format is controlled by a so-called "output plugin"; an example is included. To make use of this in a real replication system, the output plugin will need to be modified to produce output in the format appropriate to that system, and to perform filtering.

Currently, information can be extracted from the logical decoding system only via SQL; future commits will add the ability to stream changes via walsender.

Andres Freund, with review and other contributions from many other people, including Álvaro Herrera, Abhijit Menon-Sen, Peter Gheogegan, Kevin Grittner, Robert Haas, Heikki Linnakangas, Fujii Masao, Abhijit Menon-Sen, Michael Paquier, Simon Riggs, Craig Ringer, and Steve Singer.

b89e151 Introduce logical decoding.
contrib/Makefile | 1 +
contrib/test_decoding/.gitignore | 4 +
contrib/test_decoding/Makefile | 69 +
contrib/test_decoding/expected/binary.out | 35 +
.../test_decoding/expected/concurrent_ddl_dml.out | 733 +++++
contrib/test_decoding/expected/ddl.out | 651 +++++
.../test_decoding/expected/decoding_in_xact.out | 89 +
contrib/test_decoding/expected/delayed_startup.out | 38 +
contrib/test_decoding/expected/mxact.out | 66 +
contrib/test_decoding/expected/permissions.out | 130 +
contrib/test_decoding/expected/rewrite.out | 107 +
contrib/test_decoding/expected/toast.out | 90 +
contrib/test_decoding/logical.conf | 2 +
.../test_decoding/specs/concurrent_ddl_dml.spec | 94 +
contrib/test_decoding/specs/delayed_startup.spec | 24 +
contrib/test_decoding/specs/mxact.spec | 38 +
contrib/test_decoding/sql/binary.sql | 14 +
contrib/test_decoding/sql/ddl.sql | 337 +++
contrib/test_decoding/sql/decoding_in_xact.sql | 41 +
contrib/test_decoding/sql/permissions.sql | 69 +
contrib/test_decoding/sql/rewrite.sql | 62 +
contrib/test_decoding/sql/toast.sql | 51 +
contrib/test_decoding/test_decoding.c | 404 +++
doc/src/sgml/contrib.sgml | 1 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/test-decoding.sgml | 42 +
src/Makefile.global.in | 2 +
src/backend/access/heap/heapam.c | 18 +-
src/backend/access/heap/pruneheap.c | 37 +-
src/backend/access/heap/rewriteheap.c | 606 +++-
src/backend/access/heap/tuptoaster.c | 26 -
src/backend/access/index/indexam.c | 6 +-
src/backend/access/rmgrdesc/heapdesc.c | 4 +
src/backend/access/transam/xact.c | 10 +-
src/backend/access/transam/xlog.c | 36 +-
src/backend/catalog/index.c | 2 +-
src/backend/catalog/system_views.sql | 34 +
src/backend/commands/analyze.c | 3 +-
src/backend/commands/cluster.c | 4 +-
src/backend/commands/dbcommands.c | 15 +
src/backend/commands/vacuum.c | 8 +-
src/backend/commands/vacuumlazy.c | 5 +-
src/backend/executor/nodeBitmapHeapscan.c | 3 +-
src/backend/replication/Makefile | 2 +
src/backend/replication/logical/Makefile | 19 +
src/backend/replication/logical/decode.c | 826 ++++++
src/backend/replication/logical/logical.c | 920 ++++++
src/backend/replication/logical/logicalfuncs.c | 509 ++++
src/backend/replication/logical/reorderbuffer.c | 3059 ++++++++++++++++++++
src/backend/replication/logical/snapbuild.c | 1885 ++++++++++++
src/backend/replication/slot.c | 289 +-
src/backend/replication/slotfuncs.c | 98 +-
src/backend/replication/walreceiver.c | 2 +-
src/backend/replication/walsender.c | 13 +-
src/backend/storage/ipc/procarray.c | 222 +-
src/backend/storage/ipc/standby.c | 30 +-
src/backend/storage/lmgr/proc.c | 8 +-
src/backend/tcop/postgres.c | 11 +
src/backend/utils/cache/inval.c | 4 +-
src/backend/utils/cache/relcache.c | 57 +-
src/backend/utils/time/snapmgr.c | 102 +-
src/backend/utils/time/tqual.c | 164 +-
src/bin/initdb/initdb.c | 5 +-
src/include/access/heapam.h | 3 +-
src/include/access/heapam_xlog.h | 14 +-
src/include/access/rewriteheap.h | 29 +-
src/include/access/transam.h | 5 +
src/include/access/tuptoaster.h | 27 +-
src/include/access/xlog.h | 1 +
src/include/catalog/catversion.h | 2 +-
src/include/catalog/pg_proc.h | 12 +-
src/include/commands/vacuum.h | 4 +-
src/include/replication/decode.h | 19 +
src/include/replication/logical.h | 100 +
src/include/replication/logicalfuncs.h | 24 +
src/include/replication/output_plugin.h | 98 +
src/include/replication/reorderbuffer.h | 351 +++
src/include/replication/slot.h | 64 +-
src/include/replication/snapbuild.h | 83 +
src/include/storage/itemptr.h | 3 +
src/include/storage/proc.h | 6 +-
src/include/storage/procarray.h | 10 +-
src/include/storage/sinval.h | 2 +
src/include/utils/inval.h | 1 +
src/include/utils/snapmgr.h | 11 +
src/include/utils/snapshot.h | 32 +-
src/include/utils/tqual.h | 15 +-
src/test/regress/expected/rules.out | 4 +-
src/tools/pgindent/typedefs.list | 33 +
89 files changed, 12997 insertions(+), 193 deletions(-)

Upstream: git.postgresql.org

  • Share