Thu 31 January 2019

A few years ago, while working on software build tooling, I needed to work out what was the optimal setting for parallelising make/compile instructions.

Using some 64-core AWS machines I plotted what happened for building some large projects using different settings for MAKEFLAGS -j. The results were interesting, for example:

For machines with up to 8 cores, I concluded that it makes sense to set

-j = number_of_cores

Once we're beyond 8-10 cores, though, the law of diminishing returns set in, so it seemed (and still seems) to me that we'd be better off using the extra cores for other work.

    if 'max-jobs' not in config:
        config['max-jobs'] = cpu_count()

    if 'instances' not in config:
        # based on some testing (mainly on AWS), maximum effective
        # max-jobs value seems to be around 8-10 if we have enough cores
        # users should set values based on workload and build infrastructure
        # FIXME: more testing :)
        if cpu_count() >= 10:
            config['instances'] = 1 + (cpu_count() / 10)
            config['max-jobs'] = cpu_count() / config['instances']

For the the general case, I'm told I should have also looked at memory bandwidth and number of independent memory channels, so ymmv.