Measuring MAKEFLAGS -j performance

Thu 31 January 2019

A few years ago, while working on software build tooling, I needed to work out what was the optimal setting for parallelising make/compile instructions.

Using some 64-core AWS machines I plotted what happened for building some large projects using different settings for MAKEFLAGS -j. The results were interesting, for example:

For machines with up to 8 cores, I concluded that it makes sense to set

-j = number_of_cores

Once we're beyond 8-10 cores, though, the law of diminishing returns set in, so it seemed (and still seems) to me that we'd be better off using the extra cores for other work.

    if 'max-jobs' not in config:
        config['max-jobs'] = cpu_count()

    if 'instances' not in config:
        # based on some testing (mainly on AWS), maximum effective
        # max-jobs value seems to be around 8-10 if we have enough cores
        # users should set values based on workload and build infrastructure
        # FIXME: more testing :)
        if cpu_count() >= 10:
            config['instances'] = 1 + (cpu_count() / 10)
            config['max-jobs'] = cpu_count() / config['instances']

For the the general case, I'm told I should have also looked at memory bandwidth and number of independent memory channels, so ymmv.

Tools vs Processes and People (again)

Thu 05 July 2018

It's a year since I wrote the "People vs Processes and Tools" article. Apart from being busy with other things, the main reason I don't write very much here is that my criteria for a post are:

  • cover something that I want (or need) to assert in public
  • include something …
Read More

People vs Processes and Tools

Mon 10 July 2017

I recently attended a demo of a software tool for requirements management. After a few minutes showing the menus and screens, the demo became a presentation of a slide deck about the software.

It turned out that the engineer demonstrating the tool does not use it himself. His weapon of …

Read More

Who can we trust for software?

Thu 15 September 2016

TL; DR We need to figure out how to guarantee that software can be trusted

In the over-simple graphic above, I've tried to map some of the key technical roles being performed in the delivery and maintenance of complex systems and services. I'm struggling to work out who precisely sits …

Read More

The Dalton Cycle

Fri 29 July 2016

Thanks to Niall Dalton for this comment at the end of his post on the trustable-software list

"I go off on this tangent to raise the whole bucket of pain involved in building a trustworthy system like this. Not that we necessarily want to try to build a 100% trustworthy …

Read More

Trustable Software - Help Wanted!

Tue 14 June 2016

For a whole load of reasons I think we need to get to trustable software, and would appreciate comments and suggestions, preferably on the mailing list, which is public.

Note, I'm not "marketing" or "selling" anything here. The aim is to trigger interest in a cross-industry set of grown-up engineers …

Read More

Are we there yet?

Sat 16 April 2016

Many large projects are late. Most of the time the the actual journey turns out to be much harder and much longer than it looked when we started out.

From a management perspective we are constantly looking for validation that we're hitting the plan, whether it's traditional milestones and deliverables …

Read More

Technical debt for whole systems

Sat 19 March 2016

"It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so." [Mark Twain]

If you've been in involved in large-scale software projects over the last few decades you've probably learned the hard way that upgrading the platform (operating system, libraries …

Read More

Software Engineering As Legitimate Engineering Discipline

Wed 20 January 2016

For 2016 I've decided it's time I take my public communications seriously.

Perhaps some of my customers and colleagues realize that my previous attempts at wit are born from decades of struggle on difficult software projects.

But most of the internet has no idea who I am, and my previous …

Read More

Why Should GENIVI Members Contribute, Again?

Tue 23 June 2015

GENIVI Members need to make attractive, useful software to run reliably in cars for a long time. The amount of software needed is increasing faster than our capacity to bring in extra engineers, and our ability to increase the productivity of the engineers already engaged. Given the economics, automotive organizations …

Read More

Leading Tools in GENIVI

Sat 02 May 2015

I've had a stormy relationship with the automotive industry - love-hate doesn't begin to cover it.

It's not that I'm a car guy - I can barely manage an oil change. But cars these days contain a staggering amount of software. And in spite of what some may say, I am a …

Read More

Three Times Done

Sun 11 January 2015

In my experience many organisations experience systemic difficulties in getting work 'done' to an acceptable standard. The Scrum solution for this is 'agreed definition of done', which typically leads to a checklist eg

  • the work has been checked in to the VCS
  • it builds ok
  • the automated tests pass
  • documentation …
Read More

Refactoring is an anti-pattern

Fri 13 December 2013

I thought I'd written more than enough words on refactoring already, but maybe I was too cryptic.

So here's my version 2.

Let's start by noting that most of the folks who claim that they are 'refactoring' don't actually know what it means.

I, on the other hand, found the …

Read More

What does "BSP" stand for?

Sun 23 June 2013

During a late-night drinking session technical discussion this week there was confusion about the meaning of "BSP" for Linux and Android.

Some folks think BSP stands for "Board Support Package".

But, as with SCRUM, the popular view is wrong.

In the real world, BSP means Bull Shit Project.

BSPs are …

Read More

The Worst Metric

Wed 06 February 2013

Lines Of Code is an awful way to measure software - but look at the pretty picture...

Read More

The software commandments

Fri 15 July 2011

OK, I admit it - some of the Agile thinking did influence the recent Software Commandments 2.0 update.

I accept short timeboxed sprints.

Agreed definition of DONE - great.

Prioritised list of requirements - awesome.

Daily standups - fine if you are all in the same room, and you stick to the rules …

Read More

The best revenge...

Thu 14 July 2011

Over the years I've worked with many amazing people, but from time to time I've also had to deal with some complete assholes unpleasantness.

Frankly it's amazing at how downright dishonest, manipulative and plain stupid some folks can be.

Actually plain stupid applies to me too.

I've been repeatedly robbed …

Read More

I predict the future of embedded Linux

Fri 06 May 2011

I've been reading the excellent Future Babble by Dan Gardner.

Perhaps unsurprisingly, it confirms that we're all wired as complete idiots when it comes to looking into the future.

The science shows that the more confidently someone makes a prediction, the more likely they're talking utter balls. But sadly, it …

Read More

I believe in aeroplanes

Thu 27 January 2011

How can you be sure that something is true?

Actually it is surprisingly difficult... Do I mean true as in fact or true as in 'I believe this'? Maybe we're in the matrix, and everything is a simulation... maybe the people you trust are lying.. or maybe they're just wrong …

Read More


Sun 16 January 2011

After a typically ignored secretive softlaunch, I'm finally taking the wraps off devcurmudgeon. The idea is to be funny, incisive... and maybe change the world one hacker software developer at a time. But I'm not optimistic. This project may fail, like so many others :-)

Mostly I expect to be ranting …

Read More

El Reg "not your grandad's project management"

Tue 14 December 2010

I confess, I get a lot of my worldview from The Register... saves me coming up with my own jokes.

Anyway, was a little disappointed watching this...

Had to turn off when the Microsoft guy said "you have to build the business …

Read More

My briefcase is older than you

Tue 30 November 2010

I recently demonstrated my desperation gravitas to a roomful of undergraduates with the immortal line "My briefcase is older than you." Disappointingly they continued fiddling with their gadgets.

Even my rollercoaster antics with Prezi failed to amuse...

But the exercise was not entirely a waste of time.

A spot of …

Read More

Picking a fight...

Sun 14 November 2010

Maybe I'm alone on this one ... but aren't Open Source and Agile pretty much directly opposed?

Agile = 5-9 people in a team, locked up together, everything thrashed out in face-to-face conversations, daily standups, timeboxed iterative deliveries.

Open Source = no fixed team size, distributed around the world, everything thrashed out via …

Read More

Musical differences...

Thu 04 November 2010

I used to be in a band, when I still had enough hair for a stupid haircut.

We had real instruments. None of this electronic bleepy noises nonsense. We used to lug huge amplifiers everywhere, because the guitars always had to be loud enough to shake the building.

Whereas the …

Read More

Open Sores

Fri 22 October 2010

I need to get with the program. All these years working on secret projects with someone else's name on the mastheads have created an obvious problem - nobody knows who the hex I am.

But wait!

There's an obvious solution too.

Why don't I just take my old spaghetti code - all …

Read More

Apple vs Contortionists redux

Fri 01 October 2010

Previously I asked "Does Apple use Scrum?"...

An ex underling just slipped me the recipe for Apple's secret sauce....

Wanna hear? Of course you do... you too could make gazillions... but you've got to promise not to tell anybody else.

It goes like this...

  1. Three independent design teams produce mockups …
Read More

Software Chaos Ridiculously Under Managed

Sat 28 August 2010

So this post is just a place-holder for me to launch my new methodology - SCRUM.

I'd hate to have some fan copy my ideas (again) and find myself unable to prove categorically that I thought it first, so here goes...

Google tells me (so it must be true) that no-one …

Read More

Emperor's New Scrum... iteration 1

Fri 27 August 2010

I've been wading through the literature on Scrum/Agile/Lean/Kanban/Hokey-kokey for months now, and I'm still no nearer to making sense of it...

How on earth can folks seriously believe that the whole commercial world should turn itself upside down so programmers can have an easy life?

No …

Read More

On time and on budget - yeah right!

Sat 07 August 2010

Developing complex software well is actually very difficult to do.

Seems to me it requires intelligence way above average, combined with enough pig-headedness to refuse to be defeated by dumb chips and other people's code spitting random error codes at you - often for hours or days at a time.

It's …

Read More

Some background...

Thu 05 August 2010

After years of advising folks NOT to blog, tweet or display their backsides innermost secrets on facebook, here I am poised to break my own rules. No doubt I'll come to regret it later, but really, I need to vent.

Later posts will explain why...

Read More