test-suite: Add a new mechanism for XFAILs

Graphics / Cairo - Uli Schlachter [znc.in] - 17 April 2021 10:01 UTC

This commit adds a new mechanism to mark tests as expected to fail via an environment variable. For example, if you expect the tests "foo" and "bar" to fail when run under image.argb32, you would set

CAIRO_TEST_IGNORE_image_argb32=foo,bar

The test suite then expects these tests to fail and treats this as xfail. If they do not fail, this is a failure on its own.

This new feature is explicitly not documented much, because it is only used as a stopgap measure to make our CI more useful: Right now the test suite runs on CI, but the result is ignored. This new feature allows to mark the known failures as xfail without too much work. When the situation changes, this will be noticed as a new test suite failure. Thus, these environment variables to not run into the danger of still containing tests that were already fixed.

a31c7395a test-suite: Add a new mechanism for XFAILs
.gitlab-ci.yml | 15 ++++++-
test/cairo-test-runner.c | 101 +++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 114 insertions(+), 2 deletions(-)

Upstream: cgit.freedesktop.org


  • Share