The existing cryptohash facility was causing problems in some code paths related to MD5 (frontend and backend) that relied on the fact that the only type of error that could happen would be an OOM, as the MD5 implementation used in PostgreSQL ~13 (the in-core implementation is used when compiling with or without OpenSSL in those older versions), could fail only under this circumstance.
The new cryptohash facilities can fail for reasons other than OOMs, like attempting MD5 when FIPS is enabled (upstream OpenSSL allows that up to 1.0.2, Fedora and Photon patch OpenSSL 1.1.1 to allow that), so this would cause incorrect reports to show up.
This commit extends the cryptohash APIs so as callers of those routines can fetch more context when an error happens, by using a new routine called pg_cryptohash_error(). The error states are stored within each implementation's internal context data, so as it is possible to extend the logic depending on what's suited for an implementation. The default implementation requires few error states, but OpenSSL could report
various issues depending on its internal state so more is needed in cryptohash_openssl.c, and the code is shaped so as we are always able to grab the necessary information.
The core code is changed to adapt to the new error routine, painting more "const" across the call stack where the static errors are stored, particularly in authentication code paths on variables that provide log details. This way, any future changes would warn if attempting to free these strings. The MD5 authentication code was also a bit blurry about the handling of "logdetail" (LOG sent to the postmaster), so improve the comments related that, while on it.
The origin of the problem is 87ae969, that introduced the centralized cryptohash facility. Extra changes are done for pgcrypto in v14 for the non-OpenSSL code path to cope with the improvements done by this commit.
b69aba7457 Improve error handling of cryptohash computations
contrib/passwordcheck/passwordcheck.c | 2 +-
contrib/uuid-ossp/uuid-ossp.c | 18 ++++--
src/backend/commands/user.c | 4 +-
src/backend/libpq/auth-sasl.c | 2 +-
src/backend/libpq/auth-scram.c | 5 +-
src/backend/libpq/auth.c | 33 ++++++-----
src/backend/libpq/crypt.c | 38 ++++++-------
src/backend/replication/backup_manifest.c | 11 ++--
src/backend/utils/adt/cryptohashfuncs.c | 25 ++++++---
src/common/cryptohash.c | 57 ++++++++++++++++++-
src/common/cryptohash_openssl.c | 93 +++++++++++++++++++++++++++++++
src/common/md5_common.c | 20 +++++--
src/include/common/cryptohash.h | 1 +
src/include/common/md5.h | 9 ++-
src/include/libpq/crypt.h | 7 ++-
src/include/libpq/sasl.h | 4 +-
src/interfaces/libpq/fe-auth.c | 22 ++++++--
src/tools/pgindent/typedefs.list | 1 +
18 files changed, 275 insertions(+), 77 deletions(-)