A few years ago, while working on software build tooling, I needed to work out
what was the optimal setting for parallelising make/compile instructions.
Using some 64-core AWS machines I plotted what happened for building some
large projects using different settings for MAKEFLAGS -j. The results were
interesting, for example:
For machines with up to 8 cores, I concluded that it makes sense to set
-j = number_of_cores
Once we're beyond 8-10 cores, though, the law of diminishing returns set in,
so it seemed (and still seems) to me that we'd be better off using the extra
cores for other work.
if 'max-jobs' not in config:
config['max-jobs'] = cpu_count()
if 'instances' not in config:
# based on some testing (mainly on AWS), maximum effective
# max-jobs value seems to be around 8-10 if we have enough cores
# users should set values based on workload and build infrastructure
# FIXME: more testing :)
if cpu_count() >= 10:
config['instances'] = 1 + (cpu_count() / 10)
config['max-jobs'] = cpu_count() / config['instances']
For the the general case, I'm told I should have also looked at memory
bandwidth and number of independent memory channels, so ymmv.
TL; DR We need to figure out how to guarantee that software can be trusted
In the over-simple graphic above, I've tried to map some of the key technical
roles being performed in the delivery and maintenance of complex systems and
services. I'm struggling to work out who precisely sits …
GENIVI Members need to make attractive, useful software to run reliably in cars for a long time. The amount of software needed is increasing faster than our capacity to bring in extra engineers, and our ability to increase the productivity of the engineers already engaged. Given the economics, automotive organizations …
In my experience many organisations experience systemic difficulties in getting work 'done' to an acceptable standard. The Scrum solution for this is 'agreed definition of done', which typically leads to a checklist eg
Actually it is surprisingly difficult... Do I mean true as in fact or true as in 'I believe this'? Maybe we're in the matrix, and everything is a simulation... maybe the people you trust are lying.. or maybe they're just wrong …
After a typically ignored secretive softlaunch, I'm finally taking the wraps off devcurmudgeon. The idea is to be funny, incisive... and maybe change the world one hacker software developer at a time. But I'm not optimistic. This project may fail, like so many others :-)
Developing complex software well is actually very difficult to do.
Seems to me it requires intelligence way above average, combined with enough pig-headedness to refuse to be defeated by dumb chips and other people's code spitting random error codes at you - often for hours or days at a time.
After years of advising folks NOT to blog, tweet or display their backsides innermost secrets on facebook, here I am poised to break my own rules. No doubt I'll come to regret it later, but really, I need to vent.