Get rid of artificial restriction on hash table sizes on Windows

Enterprise / PostgreSQL - Tom Lane [] - 25 July 2021 18:02 UTC

The point of introducing the hash_mem_multiplier GUC was to let users reproduce the old behavior of hash aggregation, i.e. that it could use more than work_mem at need. However, the implementation failed to get the job done on Win64, where work_mem is clamped to 2GB to protect
various places that calculate memory sizes using "long int". As written, the same clamp was applied to hash_mem. This resulted in severe performance regressions for queries requiring a bit more than 2GB for hash aggregation, as they now spill to disk and there's no way to stop that.

Getting rid of the work_mem restriction seems like a good idea, but it's a big job and could not conceivably be back-patched. However, there's only a fairly small number of places that are concerned with the hash_mem value, and it turns out to be possible to remove the restriction there without too much code churn or any ABI breaks. So, let's do that for now to fix the regression, and leave the larger task for another day.

This patch does introduce a bit more infrastructure that should help with the larger task, namely pg_bitutils.h support for working with size_t values.

Per gripe from Laurent Hasson. Back-patch to v13 where the behavior change came in.

Discussion: Discussion:

28d936031a Get rid of artificial restriction on hash table sizes on Windows.
src/backend/executor/execGrouping.c | 6 +-
src/backend/executor/nodeAgg.c | 42 ++++++-----
src/backend/executor/nodeHash.c | 126 +++++++++++++++++----------------
src/backend/executor/nodeMemoize.c | 2 +-
src/backend/optimizer/path/costsize.c | 7 +-
src/backend/optimizer/plan/planner.c | 6 +-
src/backend/optimizer/plan/subselect.c | 6 +-
src/backend/optimizer/prep/prepunion.c | 8 +--
src/backend/optimizer/util/pathnode.c | 3 +-
src/backend/storage/ipc/shm_mq.c | 6 +-
src/include/miscadmin.h | 2 +-
src/include/port/pg_bitutils.h | 50 ++++++++++++-
12 files changed, 153 insertions(+), 111 deletions(-)


  • Share