tpm2: avoid comparison between signed and unsigned ints

Hardware / Coreboot - Vadim Bendebury [chromium.org] - 13 July 2016 16:58 UTC

The marshaling/unmarshaling code is using integer values to represent room left in the buffer, to be able to communicate three conditions: positive number means there is room left in the buffer, zero means that the exact amount of data in the buffer was unmarshaled and negative value means that the result of the operation did not fit into the buffer.

The implementation is wrong though, as it compares directly signed and unsigned values, which is illegal, as signed values get promoted to unsigned by the compiler.

This patch changes the marshaling code to use size_t for the size, and use zero as marshaling failure indication - after all the buffer where the data is marshaled to should definitely be large enough, and it is reasonable to expect at least some room left in it after marshaling.

The unmarshaling situation is different: we sure want to communicate errors to the caller, but do not want to propagate error return values through multiple layers. This patch keeps the size value in int, but checks if it is negative separately, before comparing with positive
values.

BRANCH=none BUG=chrome-os-partner:50645 TEST=with the rest of the patches applied kevin successfully boots up.

Change-Id: Ibfbd1b351e35e37c8925a78d095e4e8492805bad

bc92710 tpm2: avoid comparison between signed and unsigned ints
src/lib/tpm2_marshaling.c | 93 +++++++++++++++++++++++++++++----------------
src/lib/tpm2_marshaling.h | 4 +-
2 files changed, 63 insertions(+), 34 deletions(-)

Upstream: review.coreboot.org

  • Share