A few years ago, while working on software build tooling, I needed to work out
what was the optimal setting for parallelising make/compile instructions.
Using some 64-core AWS machines I plotted what happened for building some
large projects using different settings for MAKEFLAGS -j. The results were
interesting, for example:
For machines with up to 8 cores, I concluded that it makes sense to set
-j = number_of_cores
Once we're beyond 8-10 cores, though, the law of diminishing returns set in,
so it seemed (and still seems) to me that we'd be better off using the extra
cores for other work.
if 'max-jobs' not in config:
config['max-jobs'] = cpu_count()
if 'instances' not in config:
# based on some testing (mainly on AWS), maximum effective
# max-jobs value seems to be around 8-10 if we have enough cores
# users should set values based on workload and build infrastructure
# FIXME: more testing :)
if cpu_count() >= 10:
config['instances'] = 1 + (cpu_count() / 10)
config['max-jobs'] = cpu_count() / config['instances']
For the the general case, I'm told I should have also looked at memory
bandwidth and number of independent memory channels, so ymmv.
It's a year since I wrote the "People vs Processes and Tools" article. Apart
from being busy with other things, the main reason I don't write very much
here is that my criteria for a post are:
cover something that I want (or need) to assert in public
I recently attended a demo of a software tool for requirements management.
After a few minutes showing the menus and screens, the demo became a
presentation of a slide deck about the software.
It turned out that the engineer demonstrating the tool does not use it himself.
His weapon of …
TL; DR We need to figure out how to guarantee that software can be trusted
In the over-simple graphic above, I've tried to map some of the key technical
roles being performed in the delivery and maintenance of complex systems and
services. I'm struggling to work out who precisely sits …
"I go off on this tangent to raise the whole bucket of pain involved in building a trustworthy system like this. Not that we necessarily want to try to build a 100% trustworthy …
For a whole load of reasons I think we need to get to trustable software, and
would appreciate comments and suggestions, preferably on the
mailing list,
which is public.
Note, I'm not "marketing" or "selling" anything here.
The aim is to trigger interest in a cross-industry set of grown-up engineers …
Many large projects are late. Most of the time the the actual journey turns
out to be much harder and much longer than it looked when we started out.
From a management perspective we are constantly looking for validation that
we're hitting the plan, whether it's traditional milestones and deliverables …
"It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so." [Mark Twain]
If you've been in involved in large-scale software projects over the last few
decades you've probably learned the hard way that upgrading the platform (operating system, libraries …
GENIVI Members need to make attractive, useful software to run reliably in cars for a long time. The amount of software needed is increasing faster than our capacity to bring in extra engineers, and our ability to increase the productivity of the engineers already engaged. Given the economics, automotive organizations …
I've had a stormy relationship with the automotive industry - love-hate doesn't begin to cover it.
It's not that I'm a car guy - I can barely manage an oil change. But cars these days contain a staggering amount of software. And in spite of what some may say, I am a …
In my experience many organisations experience systemic difficulties in getting work 'done' to an acceptable standard. The Scrum solution for this is 'agreed definition of done', which typically leads to a checklist eg
Actually it is surprisingly difficult... Do I mean true as in fact or true as in 'I believe this'? Maybe we're in the matrix, and everything is a simulation... maybe the people you trust are lying.. or maybe they're just wrong …
After a typically ignored secretive softlaunch, I'm finally taking the wraps off devcurmudgeon. The idea is to be funny, incisive... and maybe change the world one hacker software developer at a time. But I'm not optimistic. This project may fail, like so many others :-)
I recently demonstrated my desperation gravitas to a roomful of undergraduates with the immortal line "My briefcase is older than you." Disappointingly they continued fiddling with their gadgets.
Even my rollercoaster antics with Prezi failed to amuse...
But the exercise was not entirely a waste of time.
Maybe I'm alone on this one ... but aren't Open Source and Agile pretty much directly opposed?
Agile = 5-9 people in a team, locked up together, everything thrashed out in face-to-face conversations, daily standups, timeboxed iterative deliveries.
Open Source = no fixed team size, distributed around the world, everything thrashed out via …
I used to be in a band, when I still had enough hair for a stupid haircut.
We had real instruments. None of this electronic bleepy noises nonsense. We used to lug huge amplifiers everywhere, because the guitars always had to be loud enough to shake the building.
I need to get with the program. All these years working on secret projects with someone else's name on the mastheads have created an obvious problem - nobody knows who the hex I am.
But wait!
There's an obvious solution too.
Why don't I just take my old spaghetti code - all …
Developing complex software well is actually very difficult to do.
Seems to me it requires intelligence way above average, combined with enough pig-headedness to refuse to be defeated by dumb chips and other people's code spitting random error codes at you - often for hours or days at a time.
My first brush with programming was on the Sinclair Cambridge Programmable Calculator. A mathematics teacher got me hooked - if I remember well he was later knocked down and killed by a milk float.
After years of advising folks NOT to blog, tweet or display their backsides innermost secrets on facebook, here I am poised to break my own rules. No doubt I'll come to regret it later, but really, I need to vent.
