Add circular WAL decoding buffer

Enterprise / PostgreSQL - Thomas Munro [postgresql.org] - 8 April 2021 11:20 UTC

Teach xlogreader.c to decode its output into a circular buffer, to support optimizations based on looking ahead.

- XLogReadRecord() works as before, consuming records one by one, and allowing them to be examined via the traditional XLogRecGetXXX() macros.

- An alternative new interface XLogNextRecord() is added that returns pointers to DecodedXLogRecord structs that can be examined directly.

- XLogReadAhead() provides a second cursor that lets you see further ahead, as long as data is available and there is enough space in the decoding buffer. This returns DecodedXLogRecord pointers to the caller, but also adds them to a queue of records that will later be consumed by XLogNextRecord()/XLogReadRecord().

The buffer's size is controlled with wal_decode_buffer_size. The buffer could potentially be placed into shared memory, for future projects. Large records that don't fit in the circular buffer are called "oversized" and allocated separately with palloc().

Discussion: https://postgr.es/m/CA+hUKGJ4VJN8ttxScUFM8dOKX0BrBiboo5uz1cq=AovOddfHpA@mail.gmail.com

f003d9f872 Add circular WAL decoding buffer.
src/backend/access/transam/generic_xlog.c | 6 +-
src/backend/access/transam/xlog.c | 28 +-
src/backend/access/transam/xlogreader.c | 744 ++++++++++++++++++++++++------
src/backend/access/transam/xlogutils.c | 2 +-
src/backend/replication/logical/decode.c | 2 +-
src/bin/pg_rewind/parsexlog.c | 2 +-
src/bin/pg_waldump/pg_waldump.c | 22 +-
src/include/access/xlogreader.h | 128 +++--
8 files changed, 734 insertions(+), 200 deletions(-)

Upstream: git.postgresql.org


  • Share