
Permalink for comment
by Rayiner Hashem on Thu 8th Jan 2004 20:57 UTC

Well, these benchmarks aren't really very indicative. The I/O benchmark is, well, I/O bound, which is why interpreted Python performed as fast as compiled C++. The numeric benchmarks are just that, numeric benchmarks. Numerics are really the best case for an optimizer, because they are so low level. All the JIT compilers should have compiled the loop once to native code, and gotten out of the way. This is fine if all you are doing is inner-loop numeric code (some scientific stuff, graphics) but not really a good indicator of general performance. Even for scientific code, this benchmark probably isn't representative, because you often need proper mathematical semantics for your calculations, which C/C++/Java/C# don't provide.

A more telling test would be to get higher-level language features involved. Test virtual C++ function calls vs Java method calls (which are automatically virtual). Test the speed of memory allocation. Test the speed of iterators in various languages. Do an abstraction benchmark (like Stepanov for C++) to test how well the compiler optimizes-away abstraction.

@Brian: I can tell you how a Common Lisp result of the same benchmark would turn out. Given proper type declarations, and a good compiler (SBCL, CMUCL), you will get arbitrarily close to C++ for this task. The compiler should generate more or less the same code. See this thread for some good numbers:

Note that cmucl is very competitive with gcc. Intel C++ blew both cmucl and gcc away, but that has nothing to do with the language. Intel C++ has an auto-vectorizer that will automatically generate SSE code if it finds algorithms (like the dot-prod and scale in this benchmark) that can be vectorized. GCC and CMUCL don't support his feature.

Interestingly, there is evidence that Lisp performs extremely well for large programs:

See this links:

In the study, the fastest programs were C++, but the average of the Lisp programs was faster than the average of the C++ programs. The Java stats on the study are a bit outdated, because it was done with JDK 1.2.