rtyler

Comparing IronPython and CPython

· mono · software development ·
First a little background to help explain some of the terms, etc. "Python" is a language, similar to how "Java" is a language; unlike Java wherein the language is also relatively synonymous with the actual implementation of that language, Python has multiple implementations. If you've run python(1) from the command line, you're most likely running the CPython implementation of the Python language, in effect, Python implemented in C. Other implementations of Python exist, like Jython (implemented on top of the Java virtual machine), PyPy (Python implemented in Python), and IronPython (Python implemented on top of the .NET CLR).

I was talking with some of the guys from the #mono channel on GIMPNet about IronPython versus CPython as far as performance is concerned and I decided that I would refine my testing (using pybench) for more similar versions of the respective implementations, in as controlled of an environment as possible.

I ran pybench.py on a "quiet" (i.e. not-busy) machine sitting in a remote datacenter not too far from Novell, the machine is a Pentium III (i386) based machine running openSUSE 10.3. Since IronPython reports it's "implementation version" as Python 2.4.0, I decided to build and run CPython 2.4 against it. IronPython is running on top of the recently released Mono 1.2.6 which I also built from source (I got IronPython from the IPCE package in YaST however). pybench reported the various implementation details for both as such:

CPython 


       Implementation: 2.4.4


       Executable:     /home/tyler/basket/bin/python


       Version:        2.4.4


       Compiler:       GCC 4.2.1 (SUSE Linux)


       Bits:           32bit


       Build:          Dec 18 2007 23:00:48 (#1)


       Unicode:        UCS2


IronPython 


       Implementation: 2.4.0


       Executable:     /usr/lib/IPCE/ipy.exe


       Version:        2.4.0


       Compiler:       .NET 2.0.50727.42


       Bits:           32bit


       Build:           (#)


       Unicode:        UCS2


IronPython did alright, but it got pretty thrashed on a lot of the benchmarks. Unfortunately it's hard to tell whether it's Mono getting beaten up, or whether it's IronPython itself that's losing the battle here, running similar tests on the .NET 2.0 CLR would be beneficial but not something I am curious enough to boot a Windows virtual machine for. Regardless, here are the results, I've highlighed the rows where IronPython performs better than CPython.



































































Test Minimum Run-time Average Run-time

CPython IronPython Diff CPython IronPython Diff
BuiltinFunctionCalls:448ms357ms+25.4%450ms405ms+11.0%
BuiltinMethodLookup:530ms1329ms-60.1%536ms1390ms-61.4%
CompareFloats:380ms129ms+194.3%381ms132ms+187.7%
CompareFloatsIntegers:377ms93ms+306.1%378ms97ms+291.2%
CompareIntegers:436ms160ms+172.5%437ms161ms+170.6%
CompareInternedStrings:425ms443ms-4.1%426ms445ms-4.3%
CompareLongs:360ms292ms+23.3%361ms293ms+23.0%
CompareStrings:423ms330ms+28.0%423ms337ms+25.6%
CompareUnicode:377ms243ms+54.7%377ms245ms+54.2%
ConcatStrings:726ms9452ms-92.3%823ms10071ms-91.8%
ConcatUnicode:711ms5687ms-87.5%756ms6039ms-87.5%
CreateInstances:508ms761ms-33.2%518ms815ms-36.4%
CreateNewInstances:451ms3475ms-87.0%458ms3581ms-87.2%
CreateStringsWithConcat:473ms2650ms-82.1%475ms2833ms-83.2%
CreateUnicodeWithConcat:482ms1008ms-52.1%508ms1092ms-53.4%
DictCreation:405ms2944ms-86.2%407ms3057ms-86.7%
DictWithFloatKeys:552ms934ms-40.9%553ms944ms-41.5%
DictWithIntegerKeys:423ms1118ms-62.2%426ms1137ms-62.5%
DictWithStringKeys:413ms1186ms-65.1%414ms1317ms-68.6%
ForLoops:412ms189ms+118.5%413ms217ms+90.7%
IfThenElse:372ms128ms+191.8%374ms141ms+165.8%
ListSlicing:311ms4033ms-92.3%315ms4230ms-92.6%
NestedForLoops:488ms349ms+39.7%489ms382ms+28.1%
NormalClassAttribute:430ms1080ms-60.2%432ms1104ms-60.9%
NormalInstanceAttribute:401ms427ms-6.1%404ms442ms-8.7%
PythonFunctionCalls:393ms302ms+30.1%402ms352ms+14.3%
PythonMethodCalls:478ms643ms-25.7%536ms673ms-20.3%
Recursion:547ms158ms+245.9%659ms159ms+313.6%
SecondImport:476ms1383ms-65.6%481ms1432ms-66.4%
SecondPackageImport:501ms1425ms-64.8%503ms1482ms-66.1%
SecondSubmoduleImport:589ms1916ms-69.3%592ms1990ms-70.2%
SimpleComplexArithmetic:475ms729ms-34.9%476ms758ms-37.3%
SimpleDictManipulation:424ms1009ms-58.0%427ms1020ms-58.2%
SimpleFloatArithmetic:416ms455ms-8.7%422ms480ms-12.0%
SimpleIntFloatArithmetic:345ms161ms+113.8%346ms162ms+112.9%
SimpleIntegerArithmetic:345ms161ms+114.7%345ms161ms+113.9%
SimpleListManipulation:346ms497ms-30.4%350ms501ms-30.1%
SimpleLongArithmetic:402ms1120ms-64.1%403ms1130ms-64.3%
SmallLists:417ms1693ms-75.4%421ms1717ms-75.5%
SmallTuples:450ms3839ms-88.3%453ms3915ms-88.4%
SpecialClassAttribute:431ms1104ms-60.9%432ms1133ms-61.8%
SpecialInstanceAttribute:608ms423ms+43.8%610ms437ms+39.5%
StringMappings:443ms2255ms-80.3%448ms2311ms-80.6%
StringPredicates:503ms1058ms-52.5%504ms1066ms-52.7%
StringSlicing:527ms2880ms-81.7%562ms3008ms-81.3%
TryExcept:418ms21ms+1905.2%418ms39ms+985.6%
TryRaiseExcept:587ms6670ms-91.2%591ms6733ms-91.2%
TupleSlicing:390ms1817ms-78.5%397ms1863ms-78.7%
UnicodeMappings:362ms1323ms-72.7%365ms1347ms-72.9%
UnicodePredicates:438ms860ms-49.0%439ms912ms-51.8%
UnicodeProperties:400ms0msn/a401ms0msn/a
UnicodeSlicing:624ms2491ms-75.0%666ms2638ms-74.7%



The results are disappointing but not all that surprising, especially with regards to string manipulation. I attempted to run the same pybench.py tool on top of Jython but Jython doesn't appear to support the "platform" module, so I don't have a really good baseline for "managed/virtual machine-based Python implementations" right now. However, given the lack of evidence otherwise, I'll just go ahead and assume IronPython blew the doors off of Jython :). In general though this isn't the be-all end-all benchmark for IronPython, especially on Mono, but it does give a nice hint of where some improvements could be made both in the Mono runtime and IronPython. I'll have to run the benchmarks again with the newer versions of both implementations of Python to see where they're improving or degrading but by all means don't let this deter you from checking out IronPython! I'll be writing up a few code samples over the next couple weeks that I hope will be helpful to those "unenlightened" among us; dynamic languages on the CLR, what has the world come to.
comments powered by Disqus