Why is PyPy translate so slow?

Nominated for the understatement of the month. I don't quite understand the details of the translation process myself, but even I can tell you that each of its several analysis and optimization passes does a whole lot of very complex work to a whole lot of code.

Disclaimer: I'm not an expert on PyPy - in particular, I don't understand the details of the RPython translation, I'm only citing docs and what overheared at the mailing list and in the blog. "A CPU-intensive task"? Nominated for the understatement of the month.

I don't quite understand the details of the translation process myself, but even I can tell you that each of its several analysis and optimization passes does a whole lot of very complex work to a whole lot of code. It starts to run the Python code normally, e.g. Importing modules, executing module-level variable definitions, defining functions and classes, applying decorators to them, etc.At some point, when "sufficent staticness" should be achieved, it stops and continues the actual translation process. It takes the frozen in-memory in-progress Python program and runs all of it in a special object space that simulates flow control, possible values of variables, etc. It's essentially running the interpreter symbolically!

It does this to perform type inference (everything but easy in a language like Python) and additional analysis. The results are converted into low-level code. Optionally, lots of optimizations (enabled by default, I'd guess) and a complex transformation for stackless support (disabled by default, I'd guess) follow.

Then it's lowering all that stuff into a type system that fits the designated backend, generates millions of lines of code (from a recent mailing list entry it seems that there are at least 19 . C files and at least one of it contains at least 247,560 lines - just so you have an idea of the order of magnitude we're talking about). All that code is compiled with gcc -O2 or similar, which of course has a lot of parsing and checking to do and will itself have many many analysis and optimization passes to do.So yeah, it's a pretty freaking HUGE task.

No wonder your puny CPU it lost. For reference, the PyPy guys used a Intel Xeon W3580 (3.33 GHz) when benchmarking the translation process in November 2010.It still took them about 76 minutes, even though they also had 12 GB of RAM - which leads to the next issue: A lot of RAM is needed in the process (2.3 GB on 64 bit Ubuntu back then, don't know how the numbers translate to your situation). I'm pretty sure you end up exceeding your physical RAM - enter excessive swapping and associated kick to performance's groin.

That, combined with the fact that you propably had several other programs running stealing CPU time and RAM, explains your experience pretty well in my opinion. Translating PyPy is reserved for more powerful computers. If there's something that can fundamentally improve these times, outsiders like us are unlikely to find it.

I'd leave these worrys to the developers.

Thanks for the quick answer. Since I don't have much memory, I use the official python to run the translate, which probably make the process even longer than it should be according to the doc, but I did get pypy built successfully on my poor machine. – jaimechen May 27 at 18:24.

I cant really gove you an answer,but what I can give you is a way to a solution, that is you have to find the anglde that you relate to or peaks your interest. A good paper is one that people get drawn into because it reaches them ln some way.As for me WW11 to me, I think of the holocaust and the effect it had on the survivors, their families and those who stood by and did nothing until it was too late.