COMPILER PERFORMANCE TESTS ON FREEBSD 10.0-CURRENT, SEPTEMBER 2012
==================================================================
INTRODUCTION
------------
The compilers tested were:
- gcc 4.2.1, the system compiler in FreeBSD, which is compiled by gcc 4.2.1.
- gcc 4.7.1, from the official gcc.gnu.org release, compiled via a three-stage
bootstrap, so the final compiler has been compiled by gcc 4.7.1.
- clang 3.1 (branches/release_31 156863), which is the default version of clang
in FreeBSD 10-CURRENT before r239462. The used executable was compiled by a
previous copy of itself.
- clang 3.2 (trunk 162107), which is the default version of clang in FreeBSD
10.0-CURRENT, after r239462. The used executable was compiled by a previous
copy of itself.
All tests were run on ref10-amd64.freebsd.org, which is a Dell 2950, 1.86GHz
Core2 Xeon, 2x4 Core, 16G RAM. It runs FreeBSD/amd64 10.0-CURRENT #0 r231914:
Sun Feb 19 17:24:37 UTC 2012.
Each build was repeated 6 times, after cleaning out the object directories, and
syncing. Each build was timed using the system time(1) command, using the -l
argument to obtain rusage information.
The programs tested by compilation were:
- A large C++ program: clang 3.2, as it occurs in the FreeBSD 10.0-CURRENT
source tree as of r239532.
- A medium-large C program: gcc 4.2.1, as it occurs in the FreeBSD 10.0-CURRENT
source tree as of r239532.
- A large C++ library: boost 1.50.0, the officially released version from
.
Building a large C++ program (clang 3.2) single-threaded
========================================================
The optimization flags used for all four compilers were the default ones that
FreeBSD uses for building clang 3.2, as a part of its source tree:
-O2 -pipe -fno-strict-aliasing
Using clang 3.1:
----------------
N Min Max Median Avg Stddev
real 6 2283.69 2288.46 2285.74 2285.505 1.6470064
user 6 2145.2 2147.2 2146.18 2146.0567 0.68266146
sys 6 128.3 132.08 130.65 130.54833 1.256653
maxrss 6 179264 179264 179264 179264 0
ixrss 6 21407 21436 21420 21419.833 9.6211572
idrss 6 3628 3632 3630 3629.8333 1.3291601
isrss 6 252 252 252 252 0
minflt 6 12485556 12485556 12485556 12485556 0
majflt 6 0 0 0 0 0
nswap 6 0 0 0 0 0
inblock 6 0 0 0 0 0
oublock 6 2058 2106 2103 2081.3333 25.216397
msgsnd 6 18 18 18 18 0
msgrcv 6 0 0 0 0 0
nsignals 6 1878 1878 1878 1878 0
nvcsw 6 16288 16357 16333 16320.667 29.615311
nivcsw 6 2071535 3998751 3057756 2966314 635381.66
Using clang 3.2:
----------------
N Min Max Median Avg Stddev
real 6 2358.61 2362.84 2362.67 2361.22 1.7831321
user 6 2215.33 2221.13 2218.72 2218.57 2.0094278
sys 6 130.78 134.63 133.41 132.99833 1.4702301
maxrss 6 177796 177796 177796 177796 0
ixrss 6 21388 21413 21408 21400.833 11.052903
idrss 6 3702 3707 3706 3704.6667 2.2509257
isrss 6 253 253 253 253 0
minflt 6 12583827 12583827 12583827 12583827 0
majflt 6 0 0 0 0 0
nswap 6 0 0 0 0 0
inblock 6 0 0 0 0 0
oublock 6 2036 2074 2071 2054.8333 19.589963
msgsnd 6 18 26 26 23.333333 4.1311822
msgrcv 6 0 0 0 0 0
nsignals 6 1878 1878 1878 1878 0
nvcsw 6 16266 16391 16354 16327.667 53.909801
nivcsw 6 2118900 3891231 3534528 3168715.7 673236.29
Using gcc 4.2.1:
----------------
N Min Max Median Avg Stddev
real 6 4238.49 4241.76 4240.78 4240.1867 1.2375244
user 6 3903.48 3908.6 3907.58 3906.5583 1.8932661
sys 6 358.38 361.43 359.94 359.94667 1.1494984
maxrss 6 568592 568592 568592 568592 0
ixrss 6 6348 6353 6350 6350.3333 1.6329932
idrss 6 3495 3498 3497 3496.5 1.0488088
isrss 6 146 146 146 146 0
minflt 6 47304156 47304184 47304175 47304172 10.545141
majflt 6 0 0 0 0 0
nswap 6 0 0 0 0 0
inblock 6 0 0 0 0 0
oublock 6 2620 2754 2732 2683.6667 68.730391
msgsnd 6 0 0 0 0 0
msgrcv 6 0 0 0 0 0
nsignals 6 1878 1878 1878 1878 0
nvcsw 6 67561 67763 67674 67648 75.198404
nivcsw 6 3994087 5821442 4846679 4810028.2 597366.52
Using gcc 4.7.1:
----------------
N Min Max Median Avg Stddev
real 6 3818.41 3974.54 3820.49 3846.7417 62.715466
user 6 3506.86 3591.97 3509.96 3522.8283 33.896088
sys 6 333.58 364.34 338.93 340.70833 11.839839
maxrss 6 480724 480736 480724 480727.33 5.316639
ixrss 6 12173 12198 12194 12188.333 9.4375138
idrss 6 1520 1523 1522 1521.8333 1.1690452
isrss 6 134 134 134 134 0
minflt 6 38406568 38406673 38406592 38406599 38.768544
majflt 6 0 90 0 20.333333 36.45088
nswap 6 0 0 0 0 0
inblock 6 0 4775 0 1233.3333 2028.0327
oublock 6 2266 2301 2286 2284.3333 13.662601
msgsnd 6 30 31 30 30.166667 0.40824829
msgrcv 6 0 0 0 0 0
nsignals 6 1878 1878 1878 1878 0
nvcsw 6 59792 67936 60369 61859.833 3186.3204
nivcsw 6 2867702 4546665 4361653 3753550.8 769382.51
Summary:
--------
For building this specific large C++ program, gcc 4.2.1 is ~86% slower than
clang 3.1 in real time, ~82% slower in user time, and ~176% slower in system
time. The maximum resident set size during building is ~217% larger, and it
causes ~279% more page reclaims.
Though gcc 4.7.1 is faster than its older version, it is still ~68% slower than
clang 3.1 in real time, ~64% slower in user time, and ~161% slower in system
time. The maximum resident set size during building is ~220% larger, and it
causes ~208% more page reclaims.
Finally, clang 3.2 is ~3% slower than clang 3.1 in both real time and user time,
and ~2% slower in system time. The maximum resident set size and the number of
page reclaims during building are approximately equal.
Conclusion:
-----------
Clang 3.1 is clearly the fastest compiler for building this specific large C++
program, with clang 3.2 trailing closely behind. Both are significantly faster,
and use much less memory than either version of gcc.
Building a medium-large C program (gcc 4.2.1) single-threaded
=============================================================
The optimization flags used for all four compilers were the default ones that
FreeBSD uses for building gcc 4.2.1, as a part of its source tree:
-O2 -pipe
Using clang 3.1:
----------------
N Min Max Median Avg Stddev
real 6 303.31 304.06 303.65 303.67167 0.24991332
user 6 275.42 277.09 275.99 276.11167 0.57766484
sys 6 24.92 26.15 25.6 25.656667 0.44643775
maxrss 6 177876 177876 177876 177876 0
ixrss 6 20529 20559 20544 20542.833 12.38413
idrss 6 3618 3623 3621 3620.3333 1.9663842
isrss 6 247 247 247 247 0
minflt 6 2214250 2214250 2214250 2214250 0
majflt 6 0 0 0 0 0
nswap 6 0 0 0 0 0
inblock 6 0 0 0 0 0
oublock 6 677 677 677 677 0
msgsnd 6 18 18 18 18 0
msgrcv 6 0 0 0 0 0
nsignals 6 883 883 883 883 0
nvcsw 6 5705 5837 5819 5793.6667 49.33018
nivcsw 6 205418 467152 449398 371699.67 114414.58
Using clang 3.2:
----------------
N Min Max Median Avg Stddev
real 6 330.22 331.23 330.95 330.69833 0.43687145
user 6 301.29 302.59 302.3 302.05667 0.49649438
sys 6 26.12 27.19 27.06 26.875 0.39747956
maxrss 6 186260 186260 186260 186260 0
ixrss 6 20639 20674 20660 20656.833 14.469508
idrss 6 3699 3705 3703 3702.3333 2.1602469
isrss 6 316 319 318 317.33333 1.2110601
minflt 6 2290933 2290934 2290934 2290933.7 0.51614557
majflt 6 0 0 0 0 0
nswap 6 0 0 0 0 0
inblock 6 0 0 0 0 0
oublock 6 668 669 668 668.16667 0.40824829
msgsnd 6 18 18 18 18 0
msgrcv 6 0 0 0 0 0
nsignals 6 883 883 883 883 0
nvcsw 6 5783 5822 5801 5799 17.944358
nivcsw 6 111115 520961 396082 316725.33 164041.32
Using gcc 4.2.1:
----------------
N Min Max Median Avg Stddev
real 6 422.68 425.44 423.23 423.47333 1.0273396
user 6 389.1 391.67 390.58 390.41333 0.82734918
sys 6 36.85 39.2 38.65 38.23 0.83840324
maxrss 6 392560 392560 392560 392560 0
ixrss 6 5529 5542 5534 5534.5 6.0580525
idrss 6 3915 3924 3919 3919 4.1472883
isrss 6 142 142 142 142 0
minflt 6 4055461 4055464 4055463 4055462.7 1.21063
majflt 6 0 4 0 0.66666667 1.6329932
nswap 6 0 0 0 0 0
inblock 6 0 730 0 121.66667 298.02125
oublock 6 659 693 662 667 12.884099
msgsnd 6 0 0 0 0 0
msgrcv 6 0 0 0 0 0
nsignals 6 883 883 883 883 0
nvcsw 6 15645 16454 15874 15888 298.36019
nivcsw 6 121293 661776 414611 363556.83 207101.28
Using gcc 4.7.1:
----------------
N Min Max Median Avg Stddev
real 6 461.58 462.55 462.01 461.98333 0.40287302
user 6 425.22 426.36 425.92 425.835 0.43825791
sys 6 40.83 42.94 41.99 41.925 0.71034499
maxrss 6 445624 445624 445624 445624 0
ixrss 6 10781 10816 10801 10797.5 12.405644
idrss 6 2427 2433 2430 2430 2.1908902
isrss 6 178 178 178 178 0
minflt 6 3883735 3883740 3883739 3883738 2.3664319
majflt 6 0 0 0 0 0
nswap 6 0 0 0 0 0
inblock 6 0 14 0 2.3333333 5.7154761
oublock 6 677 681 679 679 1.4142136
msgsnd 6 20 20 20 20 0
msgrcv 6 0 0 0 0 0
nsignals 6 883 883 883 883 0
nvcsw 6 16411 16660 16532 16542.333 98.544744
nivcsw 6 284414 901533 384379 449845.33 241447.41
Summary:
--------
For building this specific medium C program, gcc 4.2.1 is ~40% slower than clang
3.1 in real time, ~41% slower in user time, and ~49% slower in system time. The
maximum resident set size during building is ~121% larger, and it causes ~83%
more page reclaims.
For C, gcc 4.7.1 is even slower than its older version; it is ~52% slower than
clang 3.1 in real time, ~54% slower in user time, and ~63% slower in system
time. The maximum resident set size during building is ~151% larger, and it
causes ~75% more page reclaims.
Finally, clang 3.2 is ~9% slower than clang 3.1 in both real time and user
time, and ~5% slower in system time. The maximum resident set size during
building is ~5% larger, and it causes ~4% more page reclaims.
Conclusion:
-----------
Clang 3.1 is clearly the fastest compiler for building this specific medium-
large C program, with clang 3.2 somewhat behind. Both are significantly faster,
and use much less memory than either version of gcc.
Building a large C++ library (boost 1.50.0) single-threaded
===========================================================
The optimization flags used for all four compilers were the defaults that Boost
1.50.0 uses for gcc-compatible compilers, from tools/build/v2/tools/gcc.jam:
-ftemplate-depth-128 -O3 -finline-functions
Using clang 3.1:
----------------
N Min Max Median Avg Stddev
real 6 1056.69 1060.49 1059.09 1058.6783 1.5028695
user 6 975.49 978.88 978.53 977.55 1.4653464
sys 6 73.75 76.42 74.87 74.95 1.0609618
maxrss 6 212324 216712 213668 214260.67 1774.6309
ixrss 6 22472 22549 22525 22514.5 31.232995
idrss 6 3793 3806 3802 3800.1667 5.492419
isrss 6 276 277 277 276.5 0.54772256
minflt 6 9543701 9543702 9543701 9543701.3 0.51234754
majflt 6 0 0 0 0 0
nswap 6 0 0 0 0 0
inblock 6 0 0 0 0 0
oublock 6 1453 1461 1457 1455.8333 3.3714487
msgsnd 6 115 115 115 115 0
msgrcv 6 0 0 0 0 0
nsignals 6 0 0 0 0 0
nvcsw 6 7352 7834 7576 7567.1667 167.70023
nivcsw 6 27478 2350999 1699745 1337037.8 1040439
Using clang 3.2:
----------------
N Min Max Median Avg Stddev
real 6 1075.33 1077.94 1076.39 1076.4267 0.93958856
user 6 995.14 997.61 995.43 995.88833 0.9489661
sys 6 72.34 74.67 74.23 73.843333 0.81563881
maxrss 6 208552 211148 210436 209936 921.08458
ixrss 6 22437 22484 22458 22459 19.768662
idrss 6 3869 3878 3873 3873.3333 3.8815804
isrss 6 275 275 275 275 0
minflt 6 9351477 9351478 9351478 9351477.5 0.54772256
majflt 6 0 0 0 0 0
nswap 6 0 0 0 0 0
inblock 6 0 0 0 0 0
oublock 6 1448 1454 1449 1449.6667 2.4221203
msgsnd 6 115 115 115 115 0
msgrcv 6 0 0 0 0 0
nsignals 6 0 0 0 0 0
nvcsw 6 10481 12934 11049 11105.333 936.9249
nivcsw 6 975292 2383586 1633650 1615797.3 605542.76
Using gcc 4.2.1:
----------------
N Min Max Median Avg Stddev
real 6 1037.86 1047.78 1039.71 1040.21 3.8054592
user 6 938.74 944.49 941.52 941.55667 1.8382999
sys 6 86.37 92.84 89.89 89.57 2.1105639
maxrss 6 560256 560316 560272 560274 21.428952
ixrss 6 6435 6453 6441 6443 7.2663608
idrss 6 3563 3573 3566 3567.5 4.0373258
isrss 6 136 136 136 136 0
minflt 6 12360490 12360492 12360491 12360491 0.63245553
majflt 6 0 0 0 0 0
nswap 6 0 0 0 0 0
inblock 6 0 4283 0 713.83333 1748.5274
oublock 6 2648 2656 2655 2653.3333 2.9439203
msgsnd 6 44 51 51 49.833333 2.857738
msgrcv 6 0 0 0 0 0
nsignals 6 0 0 0 0 0
nvcsw 6 7897 12281 8004 8696 1757.1741
nivcsw 6 19915 1989580 897003 957452.5 764999.83
Using gcc 4.7.1:
----------------
N Min Max Median Avg Stddev
real 6 1038.13 1041.29 1040.98 1039.92 1.3837774
user 6 937.73 943.59 941.35 941.14167 2.0323919
sys 6 89.19 95.1 91.61 91.588333 2.0745739
maxrss 6 361268 361268 361268 361268 0
ixrss 6 12431 12474 12469 12457.333 17.385818
idrss 6 1547 1552 1551 1549.8333 1.9407902
isrss 6 129 129 129 129 0
minflt 6 10455489 10455489 10455489 10455489 0
majflt 6 0 0 0 0 0
nswap 6 0 0 0 0 0
inblock 6 0 162 0 27 66.136223
oublock 6 2537 2540 2539 2538.5 1.0488088
msgsnd 6 113 113 113 113 0
msgrcv 6 0 0 0 0 0
nsignals 6 0 0 0 0 0
nvcsw 6 7778 7975 7880 7874.1667 78.036957
nivcsw 6 27055 2302383 2187401 1478808.3 946760.86
Summary:
--------
For building this specific large C++ library, clang 3.1 is ~2% slower than gcc
4.2.1 in real time, ~4% slower in user time, but ~20% faster in system time.
The maximum resident set size during building is ~162% smaller, and it causes
~30% less page reclaims.
As before, clang 3.2 is slower than its older version; it is ~3% slower than
gcc 4.2.1 in real time, ~6% slower in user time, but ~21% faster in system
time. The maximum resident set size is ~167% smaller, and it causes ~32% less
page reclaims.
Finally, gcc 4.7.1 is equally fast as gcc 4.2.1 in real time and user time, and
~2% slower in system time. The maximum resident set size is ~36% smaller, and
it causes ~15% less page reclaims.
Conclusion:
-----------
Both gcc 4.2.1 and 4.7.1 are the fastest compilers for building this specific
large C++ library, but both versions of clang are not far behind. Both versions
of gcc use quite a bit more memory than either version of clang.
================================================================================
Copyright (c) 2012 Dimitry Andric
Verbatim copying and redistribution of this entire text are permitted, provided
this notice is preserved.
================================================================================