[Scons-users] Announcement: SCons core changes ahead...

[Dirk Bächle]

Hi there,

after releasing SCons v2.3.2, the developer team is now planning the 
next steps. In the background we're already working on a version that 
runs under Python 2 *and* 3, and have started redesigning the handling 
of Tools a little bit. Both of these items will need some more time, so 
we'd like to take care of SCons' performance instead for the next release.

We have prepared two major additions that will hit the development repo 
at Bitbucket soon. They will probably have very little impact (none, for 
the standard user), but might break some of your SConstructs/SConscripts 
and Builders. That's why we're making this announcement as early as 
possible...to give you enough time, if you should need to prepare for 
the oncoming changes:

1.) Speed: A series of profilings (see 
http://scons.org/wiki/WhySconsIsNotSlow ) has uncovered that clean 
builds in SCons suffer badly from the fork() call, which is used by the 
subprocess module to spawn separate processes when executing single 
build commands. For Posix-like systems we'll add a wrapper module, that 
redirects the subprocess calls to the more lightweight posix_spawn().

     This change can have an impact on your local code if you're 
redirecting the SPAWN method in SCons environments your own way.

2.) Memory: SCons is known to be a memory hog, but by having raised the 
floor version of Python to 2.4 first, and now to 2.7, we can use more 
modern idioms against it. We'd like to switch the Node class, and its 
supporting NodeInfo and BuildInfo classes, to using "slots".

     Again, the changes are transparent for the "normal" user and no 
modification of your local build descriptions should be necessary. Only 
if you're accessing Node attributes or adding/replacing methods 
directly, you might have to adapt to the transitions described in the 
appendix below.


In return to some slight "inconveniences", we get back a linear runtime 
behaviour even for very large clean builds (like > 40000 files or so). 
Memory consumption will get reduced, without affecting speed...big 
projects save more, as the following figures show (viewed best with a 
fixed-width font):


Clean builds
============

Project     | Default (MB) | Slots (MB) | Saved
-----------------------------------------------
wonderbuild |     391      |     274    |  30%
ascend      |      92      |      80    |  13%
sconsbld    |     434      |     288    |  34%
questfperf  |     319      |     230    |  28%
mapnik      |      74      |      69    |   7%

Updates
=======

Project     | Default (MB) | Slots (MB) | Saved
-----------------------------------------------
wonderbuild |     367      |     243    |  34%
ascend      |      91      |      78    |  14%
sconsbld    |     453      |     296    |  35%
questfperf  |     330      |     233    |  29%
mapnik      |      74      |      69    |   7%


If this pre-announcement left any questions open, just ask please. When 
a branch with these changes is available we'll inform you again, such 
that you can then test it against your builds to detect problems early.


Best regards,

Dirk



Appendix: Short list of changes to the Node* classes
====================================================

- Introduction of decorators for memoizer debugging/counting (we can't 
use metaclasses anymore).
- Adding interface for accessing node attributes that may get 
initialized lazily:
     t.abspath -> t.get_abspath()
     t.labspath -> t.get_labspath()
     t.tpath -> t.get_tpath()
     t.path_elements -> t.get_path_elements()
     t.path -> t.get_internal_path()

- Methods like t.exists(), t.rexists() and t.get_contents() can't be 
re-assigned directly anymore. They'll be realized via function maps 
internally ( {int : func} ).

- NodeInfo and BuildInfo will also get switched to __slots__, and get a 
new internal version number (1.0 -> 2.0). This means that after the 
switch *all* targets are regarded to be out-of-date on the first build.