Add new script for plotting string benchmark JSON output

System Internals / glibc - Krzysztof Koch [arm.com] - 13 November 2019 14:18 EST

Add a script for visualizing the JSON output generated by existing glibc string microbenchmarks.

Overview: plot_strings.py is capable of plotting benchmark results in the following formats, which are controlled with the -p or --plot argument: 1. absolute timings (-p time): plot the timings as they are in the input benchmark results file. 2. relative timings (-p rel): plot relative timing difference with respect to a chosen ifunc (controlled with -b argument). 3. performance relative to max (-p max): for each varied parameter
value, plot 1/timing as the percentage of the maximum value out of the plotted ifuncs. 4. throughput (-p thru): plot varied parameter value over timing

For all types of graphs, there is an option to explicitly specify the subset of ifuncs to plot using the --ifuncs parameter.

For plot types 1. and 4. one can hide/expose exact benchmark figures using the --values flag.

When plotting relative timing differences between ifuncs, the first ifunc listed in the input JSON file is the baseline, unless the baseline implementation is explicitly chosen with the --baseline parameter. For the ease of reading, the script marks the statistically insignificant range on the graphs. The default is +-5% but this
value can be controlled with the --threshold parameter.

To accommodate for the heterogeneity in benchmark results files, one can control i.e the x-axis scale, the resolution (dpi) of the generated figures or the key to access the varied parameter value in the JSON file. The corresponding options are --logarithmic,--resolution or --key. The --key parameter ensures that plot_strings.py works with all files which pass JSON schema validation. The schema can be chosen with the --schema parameter.

If a window manager is available, one can enable interactive figure display using the --display flag.

Finally, one can use the --grid flag to enable grid lines in the generated figures.

Implementation: plot_strings.py traverses the JSON tree until a 'results' array is found and generates a separate figure for each such array. The figure is then saved to a file in one of the available formats (controlled with the --extension parameter).

As the tree is traversed, the recursive function tracks the metadata about the test being run, so that each figure has a unique and meaningful title and filename.

While plot_strings.py works with existing benchmarks, provisions have been made to allow adding more structure and metadata to these benchmarks. Currently, many benchmarks produce multiple timing values for the same value of the varied parameter (typically 'length'). Mutiple data points for the same parameter usually mean that some other parameter was varied as well, for example, if memmove's src and dst buffers overlap or not (see bench-memmove-walk.c and bench-memmove-walk.out).

Unfortunately, this information is not exposed in the benchmark output file, so plot_strings.py has to resort to computing the geometric mean of these multiple values. In the process, useful information about the benchmark configuration is lost. Also, averaging the timings for different alignments can hide useful characterstics of the benchmarked ifuncs.

Testing: plot_strings.py has been tested on all existing string microbenchmarks which produce results in JSON format. The script was tested on both Windows 10 and Ubuntu 16.04.2 LTS. It runs on both python 2 and 3 (2.7.12 and 3.5.12 tested).

Useful commands: 1. Plot timings for all ifuncs in bench-strlen.out: $ ./plot_strings.py bench-strlen.out

2. Display help: $ ./plot_strings.py -h

3. Plot throughput for __memset_avx512_unaligned_erms and __memset_avx512_unaligned. Save the generated figure in pdf format to 'results/'. Use logarithmic x-axis scale, show grid lines and expose the performance numbers: $ ./plot_strings.py bench.out -o results/ -lgv -e pdf -p thru \-i __memset_avx512_unaligned_erms __memset_avx512_unaligned

4. Plot relative timings for all ifuncs in bench.out with __generic_memset as baseline. Display percentage difference threshold of +-10%: $ ./plot_strings.py bench.out -p rel -b __generic_memset -t 10

Discussion: 1. I would like to propose relaxing the benchout_strings.schema.json to allow specifying either a 'results' array with 'timings' (as before) or a 'variants' array. See below example:

{ "timing_type": "hp_timing", "functions": { "memcpy": { "bench-variant": "default", "ifuncs": ["generic_memcpy", "__memcpy_thunderx"], "variants": [ { "name": "powers of 2", "variants": [ { "name": "both aligned", "results": [ { "length": 1, "align1": 0, "align2": 0, "timings": [x, y] }, { "length": 2, "align1": 0, "align2": 0, "timings": [x, y] },

{ "length": 65536, "align1": 0, "align2": 0, "timings": [x, y] }] }, { "name": "dst misaligned", "results": [ { "length": 1, "align1": 0, "align2": 0, "timings": [x, y] }, { "length": 2, "align1": 0, "align2": 1, "timings": [x, y] },


'variants' array consists of objects such that each object has a 'name' attribute to describe the configuration of a particular test in the benchmark. This can be a description, for example, of how the parameter was varied or what was the buffer alignment tested. The 'name' attribute is then followed by another 'variants' array or a 'results' array.

The nesting of variants allows arbitrary grouping of benchmark timings, while allowing description of these groups. Using recusion, it is possible to proceduraly create titles and filenames for the figures being generated.

15740788d7 Add new script for plotting string benchmark JSON output
benchtests/scripts/plot_strings.py | 395 +++++++++++++++++++++++++++++++++++++
1 file changed, 395 insertions(+)

Upstream: sourceware.org


  • Share