If an error occurs during the tracing operation, particularly during a call to lock_and_info() which calls _dl_addr, we may end up calling back into the malloc-subsystem and relock the loader lock and deadlock. For all intents and purposes the call to _dl_addr can call any of the malloc family API functions and so we should disable all tracing before calling such loader functions. This is similar to the strategy that the new malloc tracer takes when calling the real malloc, namely that all tracing ceases at the boundary to the real function and any faults at that point are the purvue of the library (though the new tracer does this on a per-thread basis in an MT-safe fashion). Since the new tracer and the hook deprecation are not yet complete we must fix these issues where we can.
Tested on x86_64 with no regressions.
e621246ec6 malloc: Set and reset all hooks for tracing (Bug 16573)
ChangeLog | 15 ++++++++++++
malloc/mtrace.c | 72 ++++++++++++++++++++++++++++++++++++---------------------
2 files changed, 61 insertions(+), 26 deletions(-)