Benchmarks

The galois library comes with benchmarking tests. They are contained in the benchmarks/ folder. They are pytest tests using the pytest-benchmark extension.

Install dependencies

First, pytest and pytest-benchmark must be installed on your system. Easily install them by installing the development dependencies.

$ python3 -m pip install -r requirements-dev.txt

Create a benchmark

To create a benchmark, invoke pytest on the benchmarks/ folder or a specific test set (e.g., benchmarks/test_field_arithmetic.py). It is also advised to pass extra arguments to format the display --benchmark-columns=min,max,mean,stddev,median and --benchmark-sort=name.

$ python3 -m pytest benchmarks/test_field_arithmetic.py --benchmark-columns=min,max,mean,stddev,median --benchmark-sort=name
============================================================================== test session starts ===============================================================================platform linux -- Python 3.8.10, pytest-4.6.9, py-1.8.1, pluggy-0.13.0
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /mnt/c/Users/matth/repos/galois, inifile: setup.cfg
plugins: requests-mock-1.9.3, cov-3.0.0, benchmark-3.4.1, typeguard-2.13.3, anyio-3.5.0
collected 56 items

benchmarks/test_field_arithmetic.py ........................................................                                                                               [100%]

------------------- benchmark "GF(2) Array Arithmetic: shape=(100_000,), ufunc_mode='jit-calculate'": 8 tests -------------------
Name (time in us)                    Min                 Max                Mean             StdDev              Median
---------------------------------------------------------------------------------------------------------------------------------
test_add                         14.6070 (1.20)     246.2890 (1.42)      18.5947 (1.27)      8.1284 (1.18)      16.2800 (1.24)
test_additive_inverse            12.2030 (1.0)      205.2510 (1.19)      14.6355 (1.0)       6.9034 (1.0)       13.1750 (1.0)
test_divide                      31.0980 (2.55)     411.8500 (2.38)      37.0098 (2.53)     16.5464 (2.40)      32.5610 (2.47)
test_multiplicative_inverse      20.9890 (1.72)     253.5650 (1.47)      28.9587 (1.98)     14.3950 (2.09)      22.5020 (1.71)
test_multiply                    14.1260 (1.16)     305.0260 (1.76)      17.1140 (1.17)     11.2444 (1.63)      14.8480 (1.13)
test_power                      144.7910 (11.87)    173.0440 (1.0)      156.4390 (10.69)    11.8957 (1.72)     153.3870 (11.64)
test_scalar_multiply            521.6480 (42.75)    823.1500 (4.76)     551.7997 (37.70)    34.6098 (5.01)     545.2820 (41.39)
test_subtract                    14.5470 (1.19)     564.8020 (3.26)      19.6030 (1.34)     14.3437 (2.08)      15.9500 (1.21)
---------------------------------------------------------------------------------------------------------------------------------

---------------------- benchmark "GF(257) Array Arithmetic: shape=(100_000,), ufunc_mode='jit-calculate'": 8 tests -----------------------
Name (time in us)                      Min                   Max                  Mean              StdDev                Median
------------------------------------------------------------------------------------------------------------------------------------------
test_add                           73.2680 (1.45)       161.8240 (1.0)         77.5348 (1.38)       6.5241 (1.0)         74.7200 (1.44)
test_additive_inverse              50.6750 (1.0)        228.7990 (1.41)        56.0843 (1.0)       14.0921 (2.16)        52.0480 (1.0)
test_divide                     3,110.9100 (61.39)    3,194.0960 (19.74)    3,150.1173 (56.17)     22.0701 (3.38)     3,148.5205 (60.49)
test_multiplicative_inverse     3,129.6760 (61.76)    3,176.1430 (19.63)    3,155.8828 (56.27)     12.4384 (1.91)     3,153.8910 (60.60)
test_multiply                     190.8880 (3.77)       211.8770 (1.31)       198.7725 (3.54)       7.5219 (1.15)       197.5500 (3.80)
test_power                      3,093.3480 (61.04)    3,941.4670 (24.36)    3,360.6968 (59.92)    368.8893 (56.54)    3,150.8450 (60.54)
test_scalar_multiply              503.8040 (9.94)       750.9060 (4.64)       521.2844 (9.29)      13.8751 (2.13)       520.4705 (10.00)
test_subtract                      73.3680 (1.45)       294.1580 (1.82)        78.2542 (1.40)       8.8710 (1.36)        75.0700 (1.44)
------------------------------------------------------------------------------------------------------------------------------------------

-------------------- benchmark "GF(257) Array Arithmetic: shape=(100_000,), ufunc_mode='jit-lookup'": 8 tests -------------------
Name (time in us)                    Min                 Max                Mean             StdDev              Median
---------------------------------------------------------------------------------------------------------------------------------
test_add                         74.2090 (1.44)      94.9680 (1.06)      81.2720 (1.46)      7.0786 (1.05)      78.1520 (1.48)
test_additive_inverse            51.3860 (1.0)       89.6480 (1.0)       55.8487 (1.0)       8.6054 (1.28)      52.6985 (1.0)
test_divide                     198.2020 (3.86)     252.7840 (2.82)     209.8239 (3.76)     12.5969 (1.87)     207.0735 (3.93)
test_multiplicative_inverse     135.6840 (2.64)     172.2930 (1.92)     144.4076 (2.59)      9.8891 (1.47)     141.1450 (2.68)
test_multiply                   187.7320 (3.65)     288.9800 (3.22)     200.3634 (3.59)     11.2736 (1.67)     198.4620 (3.77)
test_power                      255.5790 (4.97)     295.4040 (3.30)     269.9318 (4.83)     10.7527 (1.60)     266.8300 (5.06)
test_scalar_multiply            502.5610 (9.78)     757.1780 (8.45)     524.1787 (9.39)     13.9009 (2.06)     524.0170 (9.94)
test_subtract                    74.7700 (1.46)      94.8770 (1.06)      79.6818 (1.43)      6.7367 (1.0)       75.6520 (1.44)
---------------------------------------------------------------------------------------------------------------------------------

------------------------- benchmark "GF(2^8) Array Arithmetic: shape=(100_000,), ufunc_mode='jit-calculate'": 8 tests --------------------------
Name (time in us)                       Min                    Max                   Mean                StdDev                 Median
------------------------------------------------------------------------------------------------------------------------------------------------
test_add                            14.1770 (1.16)         63.0380 (1.0)          15.8036 (1.04)         3.4018 (1.0)          14.8980 (1.13)
test_additive_inverse               12.2530 (1.0)         231.6340 (3.67)         15.2299 (1.0)          5.7599 (1.69)         13.1950 (1.0)
test_divide                     18,110.1600 (>1000.0)  23,570.4150 (373.91)   20,069.0754 (>1000.0)  2,438.1073 (716.71)   18,794.8630 (>1000.0)
test_multiplicative_inverse     16,511.2540 (>1000.0)  16,933.0940 (268.62)   16,692.3108 (>1000.0)    175.2790 (51.53)    16,691.8020 (>1000.0)
test_multiply                    1,092.8780 (89.19)     1,158.5510 (18.38)     1,126.8230 (73.99)       24.4832 (7.20)      1,136.4800 (86.13)
test_power                      16,858.2620 (>1000.0)  18,860.9460 (299.20)   17,547.2790 (>1000.0)    906.1260 (266.37)   16,982.8170 (>1000.0)
test_scalar_multiply               835.2460 (68.17)     1,118.3870 (17.74)       862.5502 (56.64)       14.4826 (4.26)        861.3440 (65.28)
test_subtract                       14.2560 (1.16)        203.0710 (3.22)         16.1151 (1.06)         4.4977 (1.32)         15.0290 (1.14)
------------------------------------------------------------------------------------------------------------------------------------------------

--------------------- benchmark "GF(2^8) Array Arithmetic: shape=(100_000,), ufunc_mode='jit-lookup'": 8 tests --------------------
Name (time in us)                    Min                   Max                Mean             StdDev              Median
-----------------------------------------------------------------------------------------------------------------------------------
test_add                         14.1270 (1.15)       234.6490 (4.07)      16.0570 (1.18)      4.4345 (1.31)      14.9080 (1.17)
test_additive_inverse            12.2530 (1.0)         90.4190 (1.57)      13.6181 (1.0)       3.3801 (1.0)       12.7340 (1.0)
test_divide                     204.2830 (16.67)      251.3910 (4.36)     219.1132 (16.09)    16.4107 (4.86)     212.2180 (16.67)
test_multiplicative_inverse     136.0750 (11.11)      170.1890 (2.95)     143.8333 (10.56)    10.7180 (3.17)     138.2690 (10.86)
test_multiply                   189.9960 (15.51)      213.2000 (3.70)     200.5345 (14.73)     7.8811 (2.33)     201.4980 (15.82)
test_power                      318.4670 (25.99)      346.3900 (6.01)     336.9693 (24.74)     8.8297 (2.61)     337.2825 (26.49)
test_scalar_multiply            795.9620 (64.96)    1,107.1750 (19.21)    829.7765 (60.93)    28.8501 (8.54)     825.2110 (64.80)
test_subtract                    14.3570 (1.17)        57.6480 (1.0)       16.2090 (1.19)      4.3581 (1.29)      15.0080 (1.18)
-----------------------------------------------------------------------------------------------------------------------------------

------------------------- benchmark "GF(3^5) Array Arithmetic: shape=(1_000,), ufunc_mode='jit-calculate'": 8 tests -------------------------
Name (time in us)                       Min                    Max                   Mean             StdDev                 Median
---------------------------------------------------------------------------------------------------------------------------------------------
test_add                           267.9320 (1.61)        340.8490 (1.53)        290.3090 (1.62)     20.4666 (1.53)        281.4870 (1.66)
test_additive_inverse              166.5620 (1.0)         222.3970 (1.0)         179.3101 (1.0)      15.4222 (1.16)        169.9880 (1.0)
test_divide                     11,050.2190 (66.34)    11,139.6170 (50.09)    11,085.6634 (61.82)    36.7729 (2.76)     11,067.3310 (65.11)
test_multiplicative_inverse     10,000.1510 (60.04)    10,081.4430 (45.33)    10,039.3364 (55.99)    36.0173 (2.70)     10,049.1930 (59.12)
test_multiply                      861.3440 (5.17)        896.6900 (4.03)        875.0538 (4.88)     13.3343 (1.0)         874.5490 (5.14)
test_power                       9,822.0570 (58.97)    10,007.8650 (45.00)     9,937.5374 (55.42)    69.6998 (5.23)      9,945.8890 (58.51)
test_scalar_multiply               731.6210 (4.39)      1,445.9930 (6.50)        776.0280 (4.33)     67.8114 (5.09)        758.5320 (4.46)
test_subtract                      265.7590 (1.60)        633.8150 (2.85)        301.0594 (1.68)     68.9719 (5.17)        282.0190 (1.66)
---------------------------------------------------------------------------------------------------------------------------------------------

------------------- benchmark "GF(3^5) Array Arithmetic: shape=(1_000,), ufunc_mode='jit-lookup'": 8 tests -------------------
Name (time in us)                   Min                 Max               Mean             StdDev             Median
------------------------------------------------------------------------------------------------------------------------------
test_add                        12.7240 (1.28)      33.4230 (1.13)     15.9908 (1.24)      5.6576 (1.50)     14.2765 (1.30)
test_additive_inverse           10.1190 (1.01)      34.8550 (1.18)     12.9293 (1.0)       5.9502 (1.57)     11.2965 (1.03)
test_divide                     10.9400 (1.10)      29.5050 (1.0)      13.0403 (1.01)      3.7818 (1.0)      12.1275 (1.10)
test_multiplicative_inverse      9.9790 (1.0)      106.0190 (3.59)     15.9433 (1.23)     19.9987 (5.29)     10.9910 (1.0)
test_multiply                   10.7700 (1.08)      34.5450 (1.17)     13.2007 (1.02)      5.5236 (1.46)     11.5815 (1.05)
test_power                      14.0560 (1.41)      51.4260 (1.74)     18.7223 (1.45)      8.8523 (2.34)     15.5290 (1.41)
test_scalar_multiply            17.5130 (1.75)     260.2270 (8.82)     20.3556 (1.57)      6.9250 (1.83)     18.9950 (1.73)
test_subtract                   14.1060 (1.41)      33.1720 (1.12)     16.8235 (1.30)      5.8832 (1.56)     14.4920 (1.32)
------------------------------------------------------------------------------------------------------------------------------

Legend:
Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
OPS: Operations Per Second, computed as 1 / Mean
==================================================================== 56 passed, 16 warnings in 16.14 seconds =====================================================================

Compare with a previous benchmark

If you would like to compare the performance impact of a branch, first run a benchmark on master using the --benchmark-save option. This will save the file .benchmarks/0001_master.json.

$ git checkout master
$ python3 -m pytest benchmarks/test_field_arithmetic.py --benchmark-save=master --benchmark-columns=min,max,mean,stddev,median --benchmark-sort=name

Next, run a benchmark on the branch under test while comparing against the benchmark from master.

$ git checkout branch
$ python3 -m pytest benchmarks/test_field_arithmetic.py --benchmark-compare=0001_master --benchmark-columns=min,max,mean,stddev,median --benchmark-sort=name

Or, save a benchmark run from branch and compare it explicitly against the one from master. This benchmark run will save the file .benchmarks/0001_branch.json.

$ git checkout branch
$ python3 -m pytest benchmarks/test_field_arithmetic.py --benchmark-save=branch --benchmark-columns=min,max,mean,stddev,median --benchmark-sort=name
$ python3 -m pytest-benchmark compare 0001_master 0001_branch

Last update: May 09, 2022