Benchmarks¶
The galois library comes with benchmarking tests. They are contained in the benchmarks/ folder. They are pytest
tests using the pytest-benchmark extension.
Install dependencies¶
First, pytest and pytest-benchmark must be installed on your system. Easily install them by installing the development dependencies.
$ python3 -m pip install -r requirements-dev.txt
Create a benchmark¶
To create a benchmark, invoke pytest on the benchmarks/ folder or a specific test set (e.g., benchmarks/test_field_arithmetic.py). It is also
advised to pass extra arguments to format the display --benchmark-columns=min,max,mean,stddev,median and --benchmark-sort=name.
$ python3 -m pytest benchmarks/test_field_arithmetic.py --benchmark-columns=min,max,mean,stddev,median --benchmark-sort=name
===================================================================== test session starts =====================================================================platform linux -- Python 3.8.10, pytest-4.6.9, py-1.8.1, pluggy-0.13.0
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /mnt/c/Users/matth/repos/galois, inifile: setup.cfg
plugins: requests-mock-1.9.3, cov-3.0.0, benchmark-3.4.1, typeguard-2.13.3, anyio-3.5.0
collected 56 items
benchmarks/test_field_arithmetic.py ........................................................ [100%]
-------------------- benchmark "GF(2) Array Arithmetic: shape=(100_000,), ufunc_mode='jit-calculate'": 8 tests --------------------
Name (time in us) Min Max Mean StdDev Median
-----------------------------------------------------------------------------------------------------------------------------------
test_add 16.3810 (1.21) 218.3280 (1.22) 18.9455 (1.17) 5.4959 (1.07) 17.3620 (1.22)
test_additive_inverse 13.5850 (1.0) 206.5360 (1.15) 16.1445 (1.0) 5.9249 (1.16) 14.2670 (1.0)
test_divide 132.0870 (9.72) 191.0680 (1.07) 149.6357 (9.27) 16.9537 (3.31) 145.4920 (10.20)
test_multiplicative_inverse 91.4410 (6.73) 179.0050 (1.0) 102.6590 (6.36) 18.8467 (3.68) 94.4670 (6.62)
test_multiply 16.0400 (1.18) 229.4400 (1.28) 18.3296 (1.14) 5.1267 (1.0) 16.9010 (1.18)
test_power 150.2410 (11.06) 212.2870 (1.19) 168.8103 (10.46) 16.4850 (3.22) 166.2860 (11.66)
test_scalar_multiply 543.3970 (40.00) 714.2870 (3.99) 562.2968 (34.83) 12.4125 (2.42) 559.1370 (39.19)
test_subtract 16.3110 (1.20) 2,233.8710 (12.48) 19.2938 (1.20) 23.4038 (4.57) 17.2520 (1.21)
-----------------------------------------------------------------------------------------------------------------------------------
---------------------- benchmark "GF(257) Array Arithmetic: shape=(100_000,), ufunc_mode='jit-calculate'": 8 tests ----------------------
Name (time in us) Min Max Mean StdDev Median
-----------------------------------------------------------------------------------------------------------------------------------------
test_add 78.2860 (1.37) 311.2620 (1.40) 87.9984 (1.26) 12.1680 (1.04) 81.7530 (1.36)
test_additive_inverse 57.0860 (1.0) 281.9070 (1.27) 69.7403 (1.0) 17.0927 (1.47) 60.0520 (1.0)
test_divide 3,274.0860 (57.35) 3,351.6220 (15.09) 3,309.5920 (47.46) 27.4510 (2.36) 3,307.3240 (55.07)
test_multiplicative_inverse 3,245.1620 (56.85) 4,295.9590 (19.34) 3,350.8016 (48.05) 96.3332 (8.26) 3,321.1050 (55.30)
test_multiply 197.1090 (3.45) 305.5620 (1.38) 218.1805 (3.13) 20.9767 (1.80) 213.6600 (3.56)
test_power 3,270.7210 (57.29) 3,520.5480 (15.85) 3,349.1942 (48.02) 91.3962 (7.84) 3,329.6105 (55.45)
test_scalar_multiply 544.0880 (9.53) 1,182.1140 (5.32) 575.6227 (8.25) 42.0059 (3.60) 562.4830 (9.37)
test_subtract 77.6160 (1.36) 222.1760 (1.0) 88.3242 (1.27) 11.6562 (1.0) 82.8905 (1.38)
-----------------------------------------------------------------------------------------------------------------------------------------
-------------------- benchmark "GF(257) Array Arithmetic: shape=(100_000,), ufunc_mode='jit-lookup'": 8 tests -------------------
Name (time in us) Min Max Mean StdDev Median
---------------------------------------------------------------------------------------------------------------------------------
test_add 79.0580 (1.37) 393.6670 (2.39) 86.7954 (1.26) 12.6945 (1.0) 81.4630 (1.34)
test_additive_inverse 57.9080 (1.0) 164.6380 (1.0) 69.0218 (1.0) 21.7213 (1.71) 60.6330 (1.0)
test_divide 228.7890 (3.95) 280.8050 (1.71) 243.1431 (3.52) 16.6688 (1.31) 241.0210 (3.98)
test_multiplicative_inverse 263.8140 (4.56) 348.4620 (2.12) 290.6663 (4.21) 20.8113 (1.64) 284.3620 (4.69)
test_multiply 193.5820 (3.34) 475.2490 (2.89) 216.4317 (3.14) 24.6557 (1.94) 212.2370 (3.50)
test_power 311.6030 (5.38) 389.2180 (2.36) 328.9333 (4.77) 18.9217 (1.49) 326.1145 (5.38)
test_scalar_multiply 539.7710 (9.32) 973.1410 (5.91) 573.4538 (8.31) 49.0047 (3.86) 557.7030 (9.20)
test_subtract 80.3500 (1.39) 270.0450 (1.64) 97.6062 (1.41) 37.3127 (2.94) 89.1270 (1.47)
---------------------------------------------------------------------------------------------------------------------------------
------------------------ benchmark "GF(2^8) Array Arithmetic: shape=(100_000,), ufunc_mode='jit-calculate'": 8 tests -------------------------
Name (time in us) Min Max Mean StdDev Median
----------------------------------------------------------------------------------------------------------------------------------------------
test_add 16.6110 (1.21) 218.1990 (1.09) 19.4288 (1.21) 5.8745 (1.11) 17.4830 (1.22)
test_additive_inverse 13.6750 (1.0) 200.7150 (1.0) 16.0465 (1.0) 5.2959 (1.0) 14.3070 (1.0)
test_divide 13,280.4310 (971.15) 13,367.6440 (66.60) 13,340.0968 (831.34) 36.5738 (6.91) 13,354.6500 (933.43)
test_multiplicative_inverse 11,842.1600 (865.97) 15,404.4870 (76.75) 12,129.1417 (755.88) 529.9702 (100.07) 12,015.3740 (839.82)
test_multiply 1,079.0300 (78.91) 1,137.0780 (5.67) 1,098.1473 (68.44) 18.6741 (3.53) 1,092.5140 (76.36)
test_power 12,832.8340 (938.41) 13,115.7640 (65.35) 12,942.1951 (806.54) 92.9381 (17.55) 12,928.9640 (903.68)
test_scalar_multiply 883.2930 (64.59) 1,192.1310 (5.94) 928.3991 (57.86) 44.9582 (8.49) 912.0860 (63.75)
test_subtract 16.6210 (1.22) 1,334.7780 (6.65) 19.7528 (1.23) 15.2536 (2.88) 17.4330 (1.22)
----------------------------------------------------------------------------------------------------------------------------------------------
--------------------- benchmark "GF(2^8) Array Arithmetic: shape=(100_000,), ufunc_mode='jit-lookup'": 8 tests --------------------
Name (time in us) Min Max Mean StdDev Median
-----------------------------------------------------------------------------------------------------------------------------------
test_add 16.0900 (1.23) 277.5990 (3.89) 18.8739 (1.24) 5.6347 (1.35) 17.1720 (1.24)
test_additive_inverse 13.1050 (1.0) 71.3340 (1.0) 15.1649 (1.0) 4.1860 (1.0) 13.8860 (1.0)
test_divide 215.6730 (16.46) 271.6490 (3.81) 233.7595 (15.41) 16.0094 (3.82) 229.9500 (16.56)
test_multiplicative_inverse 152.3150 (11.62) 207.4480 (2.91) 167.0589 (11.02) 12.9483 (3.09) 166.4220 (11.98)
test_multiply 199.3430 (15.21) 250.2580 (3.51) 220.8079 (14.56) 17.1620 (4.10) 216.1740 (15.57)
test_power 331.7910 (25.32) 401.3410 (5.63) 348.8168 (23.00) 17.4759 (4.17) 348.8730 (25.12)
test_scalar_multiply 850.2810 (64.88) 1,128.3010 (15.82) 884.6499 (58.34) 29.6705 (7.09) 876.5800 (63.13)
test_subtract 16.0400 (1.22) 83.5460 (1.17) 18.2685 (1.20) 4.4904 (1.07) 16.8610 (1.21)
-----------------------------------------------------------------------------------------------------------------------------------
--------------------- benchmark "GF(3^5) Array Arithmetic: shape=(100_000,), ufunc_mode='jit-lookup'": 8 tests --------------------
Name (time in us) Min Max Mean StdDev Median
-----------------------------------------------------------------------------------------------------------------------------------
test_add 313.4770 (2.04) 358.2300 (1.58) 328.4561 (1.85) 12.1327 (1.0) 326.4100 (1.87)
test_additive_inverse 153.6980 (1.0) 226.6550 (1.0) 177.9128 (1.0) 19.6890 (1.62) 174.6160 (1.0)
test_divide 222.3460 (1.45) 284.7130 (1.26) 235.5486 (1.32) 15.5184 (1.28) 232.1795 (1.33)
test_multiplicative_inverse 165.4600 (1.08) 241.2010 (1.06) 186.5927 (1.05) 23.5185 (1.94) 178.2130 (1.02)
test_multiply 202.1690 (1.32) 327.1620 (1.44) 231.3098 (1.30) 30.2870 (2.50) 219.7315 (1.26)
test_power 361.5260 (2.35) 447.0060 (1.97) 385.7585 (2.17) 28.6975 (2.37) 375.4475 (2.15)
test_scalar_multiply 756.5460 (4.92) 1,014.9590 (4.48) 792.1778 (4.45) 29.3465 (2.42) 786.1765 (4.50)
test_subtract 383.7790 (2.50) 461.3640 (2.04) 411.7450 (2.31) 26.7056 (2.20) 403.7260 (2.31)
-----------------------------------------------------------------------------------------------------------------------------------
-------------------------- benchmark "GF(3^5) Array Arithmetic: shape=(10_000,), ufunc_mode='jit-calculate'": 8 tests --------------------------
Name (time in us) Min Max Mean StdDev Median
------------------------------------------------------------------------------------------------------------------------------------------------
test_add 876.9310 (1.57) 1,635.8940 (1.52) 936.2487 (1.48) 76.1260 (3.84) 915.1175 (1.58)
test_additive_inverse 557.6440 (1.0) 1,945.0700 (1.81) 632.3527 (1.0) 257.9239 (13.01) 578.4425 (1.0)
test_divide 90,022.6490 (161.43) 96,282.8560 (89.50) 92,257.7516 (145.90) 2,808.8230 (141.69) 90,481.3870 (156.42)
test_multiplicative_inverse 82,011.9590 (147.07) 83,817.2670 (77.91) 82,897.2702 (131.09) 471.2330 (23.77) 82,992.5040 (143.48)
test_multiply 6,847.6130 (12.28) 6,894.3920 (6.41) 6,872.3102 (10.87) 19.8231 (1.0) 6,876.2980 (11.89)
test_power 77,322.3730 (138.66) 78,040.5270 (72.54) 77,650.6814 (122.80) 267.5041 (13.49) 77,693.8380 (134.32)
test_scalar_multiply 6,049.4100 (10.85) 7,260.1360 (6.75) 6,184.4565 (9.78) 146.6458 (7.40) 6,153.1895 (10.64)
test_subtract 888.4720 (1.59) 1,075.8030 (1.0) 944.4420 (1.49) 47.1406 (2.38) 936.5830 (1.62)
------------------------------------------------------------------------------------------------------------------------------------------------
Legend:
Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
OPS: Operations Per Second, computed as 1 / Mean
=========================================================== 56 passed, 16 warnings in 19.54 seconds ===========================================================
Compare with a previous benchmark¶
If you would like to compare the performance impact of a branch, first run a benchmark on main using the --benchmark-save option.
This will save the file .benchmarks/0001_master.json.
$ git checkout main
$ python3 -m pytest benchmarks/test_field_arithmetic.py --benchmark-save=main --benchmark-columns=min,max,mean,stddev,median --benchmark-sort=name
Next, run a benchmark on the branch under test while comparing against the benchmark from main.
$ git checkout branch
$ python3 -m pytest benchmarks/test_field_arithmetic.py --benchmark-compare=0001_master --benchmark-columns=min,max,mean,stddev,median --benchmark-sort=name
Or, save a benchmark run from branch and compare it explicitly against the one from main. This benchmark run will save the file .benchmarks/0001_branch.json.
$ git checkout branch
$ python3 -m pytest benchmarks/test_field_arithmetic.py --benchmark-save=branch --benchmark-columns=min,max,mean,stddev,median --benchmark-sort=name
$ python3 -m pytest-benchmark compare 0001_master 0001_branch
Last update:
Nov 04, 2023