Publications :: Various Papers :: Quick Benchmark of Mono CLR, Sun JVM and MS CLR
2002/15/09, version 1.2, Daniel Mettler
Changes (see the markup)
Version 1.2, 2002/15/09
Added a link to the Portable.NET Benchmark (by Rhys Weatherley) which is "loosely based on the techniques used by the CaffeineMark". Note that, according to Rhys Weatherley, "the purpose of this tool is not to compare Portable.NET with other CLR implementations. Rather, it is intended to identify areas of Portable.NET that may need further attention."
Version 1.1, 2002/13/09
Did some clarifications and added a link to the Gfx_NET 3D bench. Thanks to the people on mono-list@ximian.com for their feedback.
Version 1.0, 2002/11/09
Initial release.
Introduction
With regard to the upcoming MONOtalk by Miguel de Icaza I decided to do some very basic performance tests with the current implementations of the Mono CLR (Linux, Windows XP), the Sun JVM (Linux, Windows XP) and the Microsoft CLR (Windows XP).
Test Setup
All tests were carried out on the following box:
AMD Athlon Thunderbird 1.0 GHz
512 MB SDRAM
Nvidia Geforce 2 GTS graphics board
IBM-DTLA-307045 hard disk
Windows XP Professional SP-1
Linux 2.4.19-gentoo-r7 (including low-latency patch [1])
The following runtime environments were used:
- mono 0.16 (Linux & Windows XP, Mono/Ximian)
- mint 0.16 (Linux & Windows XP, Mono/Ximian)
- java 1.4.1_01 (Linux & Windows XP, Java/Sun)
- clr 1.0.3705 (Windows XP, .NET/Microsoft)
Compilers:
- Linux & Windows XP, Mono: mcs 0.16
- Linux & Windows XP, Java: javac 1.4.1_01
- Windows XP, .NET: csc 7.00.9466
The following sources were used:
LoopMono.cs
LoopJava.java
Assumptions:
- The above source code works correctly with all tested runtime environments
- Particularly the time measurement methods are assumed to work correctly with all tested runtime environments
- Default heap and stack sizes for all runtime environments
- Default "tuning" (none at all) of all runtime environments
- No other significant background activity during the tests (no logging, swapping etc.). Therefore the tests on Linux were done in single user mode, syslog and other daemons were stopped. On Windows XP, sys-tray applications were unloaded from memory (if possible) and all optional system services were stopped prior to the testing. All tests on Windows XP were done using a 80x25 terminal window.
- Negligible use of RAM (no swapping etc.)
Bytecode/CIL code:
Results
mono_bench_data.txt (raw data)
Timings in milliseconds (lower is better) for 100'000 loops. Standard deviation based on 'sample'.
|
|
Linux 2.4.19-gentoo-r7
|
Windows XP SP-1
|
|
mono 0.16
|
5148 +/- 30
|
9119 +/- 71
|
|
mint 0.16
|
29767 +/- 31
|
27690 +/- 193
|
|
java 1.4.1_01 (mixed mode)
|
4268 +/- 1
|
12149 +/- 97
|
|
java 1.4.1_01 (int mode)
|
8965 +/- 11
|
13410 +/- 14
|
|
clr 1.0.3705
|
n/a
|
29552 +/- 45
|
Conclusions
- Disclaimer: This benchmark test only measures how long it takes to write "Hello World!" a 100'000 times to the system console. In consequence, the following conclusions are only valid in this particular context (which is not representative for a general context at all).
For well-balanced, overall performance measurements another benchmark, possibly consisting of a defined mix of common method calls, variable initializations, GUI tasks etc. would be required. Eventually, measuring the performance of a real world application which is available for both runtime environments would be suitable for this purpose[2].
- Apart from the results obtained with 'mint', Linux runtime environments show a significantly better execution performance than those running on Windows XP. Note: This point might be a bit misleading. Of course, such a small, highly specific bench is not suitable for comparing the performances of the runtime environments of different OS platforms. Especially if the bench involves console/terminal output (which is known to be pretty slow on Windows "by nature"). The purpose of these tests is to compare the relative performances of the runtime environments of a particular OS platform regarding this very specific task.
- On Linux, the JRE 1.4.1_01 slightly beats the Mono CLR 0.16.
- On Windows XP, the Mono CLR 0.16 performs significantly better than the JRE 1.4.1_01. Microsoft .NET CLR 1.0.3705 is bottom of the league here.
- Comparing runtime environments on the same platform, mixed mode runtime environments (partly using JIT compilation technologies) perform better than interpretation-only runtime environments in general (with the exception of the Microsoft .NET CLR).
- For detailed comparisons, take a look at the figures in the results table. Another interesting task might be a comparison of the (obviously different) CIL code generated by the Mono and Microsoft .NET compiler respectively.
References
The MONO Project
Sun J2SE
Microsoft .NET Framework SDK
Results of the J2EE vs. .NET bench by "The Middleware Company"
Gfx_NET.tar.gz, a 3D bench for .NET by Sergey Chaban
Portable.NET benchmark
Annotations
[1] The low-latency patch usually slows down benchmarks considerably
[2] Recently, results of performance comparisons between .NET and J2EE using a .NET port of the Java Pet Store application have been published. Critics complain that the two versions of the Pet Store did not undergo the same efforts regarding speed optimization. Consequently I would suggest to repeat the tests in two different ways. One test should be done with fully optimized variants without any restrictions (allowing the use of native code and other "dirty" tricks), another one with fully optimized variants but not allowing any native code. Both Microsoft and Sun should be responsible for the optimization of their respective Pet Store application variants. The conformity of the applications and the benchmark tests should be supervised by an independent jury/organisation.
The same should be done for a client-side benchmark using a typical GUI application.
|
