Inquiry Phase 2: Star Witness – Dave gives it both barrels

David McDonnell, a former Deputy Development Manager on the Horizon IT project with Fujitsu/ICL gave evidence during the Horizon IT inquiry on the morning of Wed 16 November.

What he had to say was devastating. In October 1998 McDonnell co-wrote a report on the Horizon EPOS system with Jan Holmes, a Fujitsu/ICL internal auditor (and a very interesting witness in the afternoon of 16 November). The Task Force report was put together to try to address the serious number errors in the Horizon project at the time. You can read it here.

The Task Force report has surfaced already during the inquiry. It was damning, and included this choice phrase:

‘Whoever wrote this code clearly has no understanding of elementary mathematics or the most basic rules of programming.’

Over three hours McDonnell gave some extraordinary evidence. You can listen to the highlights on Episode 22 of Investigating the Post Office Scandal (which includes clips from the last four witnesses), or you can watch the entire evidence session on video and read along with the full transcript, by clicking on the inquiry website here

McDonnell describes the EPOS programming team as like ‘the Wild West’.

When asked what he meant, he replied: ‘There were no standards in place, there were no design documents. The culture of the development team was – I wouldn’t say it was a holiday camp, but it was free format. There was no structure, no discipline; it was crazy, never seen anything like it.’

Of the code itself, he said:

‘it was so bad. It was beyond anything I’ve ever seen. Even in the 25/30 years since that project, I’ve never seen anything like that before. Some of the stuff that we found buried in the code was unbelievable. There was unreachable code… It was a mess.’

Among techies at Fujitsu/ICL, McDonnell quickly discovered the EPOS team were the ‘joke of the building’, telling the inquiry ‘everybody knew, specifically the test team who, when I spoke to those guys, they would make it very clear that the quality of code that was being delivered was to such a bad, poor level that they’re wasting their time testing it.’

The solution

McDonnell’s rather obvious solution was to get some better coders in and re-write the cash account from scratch. He was overruled by the Horizon Programme Manager, Terry Austin. McDonnell found the resistance to his solution odd, and believes the way it has been portrayed by other witnesses at the inquiry as too big a problem to realistically deal with, ‘betrayed a basic misunderstanding of how the EPOS system was built or even potentially suggests an attempt to obfuscate the issue.’

He described it using a Lego analogy:

‘if you understood that it was built out of Lego bricks, you could replace the Lego bricks one at a time starting
with the most critical, the most important, which I would argue was the cash account. Here, you could even — because it was a batch process that wasn’t part of the counter client/customer interaction, you could rewrite that as a separate module and have it running as a shadow process on the counter. You could run the cash account twice at the end of the day or whenever, as a secondary confirmation, and use the replacement module to check the validity of the first one. Once you’d proved that it worked, you could take the old one out and just continue with the new one. This was not a large task. It was not something that – I couldn’t understand why they didn’t do it, because it was such a – it’s not a small piece of work but relatively small, and you could have done it without introducing any danger to anything else on the counter.’

The CSR+ solution

McDonnell says matters reached a head when he was called into Terry Austin’s office and offered a promotion. He accepted on the condition the EPOS cash account was re-written. McDonnell says Austin became ‘frustrated’ by his insistence:

‘He wasn’t very happy with me putting a condition on that acceptance. It was clear that the cash account wasn’t going to get written. That conversation was very quickly brought to a halt, and I was ushered out of the office, and I never really spoke to Terry after that again.’

Fujitsu, in its wisdom, chose to try to fix the code, which took a year, and by November 1999, after the system had been accepted, they were getting a similar number of errors and bugs. Fujitsu got acceptance from the Post Office by agreeing to write a new bit of code, known as the CSR+ release which would monitor the cash accounting discrepancies within the system. McDonnell had been moved off the project by that state, but described the CSR+ release as a ‘big bone of contention at the time.’ The way he saw it:

‘At the end of the Task Force they were given the report that we co-authored detailing what the senior engineers, senior auditing guy, and all of the experienced people around the project were saying, detailing the problems. It’s like the captain of the ship’s been told that there’s a hole in the boat and it’s filling with water by the engineers. Instead of fixing the hole, what they did was they went away and constructed this CSR+ release, which is akin to painting a plimsoll line on the outside of the boat so that they could measure how fast it was sinking.

‘The whole context of this CSR+ release was about being able to detect discrepancies between the counter and the middle and back office, the APS systems and such, and highlight where there was a difference between the number of transactions or the balance between the two being different. That’s just building a dipstick instead of actually fixing the hole in the boat. They spent a year, an inordinate amount of time and resource, on this release instead of fixing the problem.’

The only question left unanswered at the end of McDonnell’s three hour session was why he didn’t come forward sooner. Mr McDonnell declined an interview after giving evidence.

My work on the Post Office Horizon IT inquiry is crowdfunded. If you’d like to contribute, please click on the widget you should be seeing to the right of this text (or below if you’re reading it on a mobile). To find out more before donating, please go to my tip jar web page. All contributors will be added to the ‘secret’ email newsletter, which offers irregular, and at times, irreverent insight into the machinations of the inquiry and the wider scandal.

Subscribe For Latest Blog Updates

11 responses to “Inquiry Phase 2: Star Witness – Dave gives it both barrels”

  1. […] Inquiry Phase 2: Star Witness – Dave gives it both barrels – Post Office Scandal […]

  2. Oh dear, listening to Richard Roll (electronics technician?) constantly confusing “lines of code” for records in a database (Oracle?) was in stark contrast to the credible evidence of a software engineering professional such as McDonnell.

    A Venn diagram of competent/incompetent, honest/liar would help perhaps. In my experience, there are many honest incompetents who have very superficial knowledge about software intensive systems-of-systems (SISoS).

    Dunning-Kruger is an explanation, objective competence management is the solution!

  3. Horizon is a clear reminder that UK institutions and computer technology. Don’t belong in the same sentence!

  4. On names alone, did the fact that POCL chose “Dick” rather than “Tom” or “Harry” seal the fate for Horizon as forever an IT systems how-not-to, synonymous with the Great Escape team choosing the obviously weakest tunnel.

  5. Terry Austin dropped several bombs during his hand-wringing inquiry appearance, for example:

    * RAD was “immature” and was “not successful”, therefore teams switched to “traditional waterfall”……Frying-pan to fire!
    * There was no “Functional Specification”……Yeehaw, cowboy-coder time!
    * Cowboy-code (EPOS) had to be “reverse engineered”…….That old chestnut!
    * There were developers “who were not very professional”……You don’t say?
    * ICL “seriously underestimated” Horizon installation, training and contract (PFI) risk……Bingo, failed at thr first hurdle!

  6. As a professional, I have witnessed this pattern dozens of times:

    PRODUCT: unreliable prototype; severe technical issues; years of chaos, recriminations and expense.
    PROCESS: non-standard lifecycle with improvised processes.
    PEOPLE: clueless cowboy-coders; no relevant qualifications or training.

    Tellingly, in 40+ years working in defence, aviation, Air Traffic Management, IT and automotive, I can count on one hand the number of organisations I encountered with an objective competence management scheme.

    1. I did visit ICL a few years ago…..unsurprisingly, I was not impressed.

    2. Only moronic IT amateurs would adopt Rapid Application Development (RAD), then expect a robust product to emerge then, when it was an obvious (EPOS) turd, polish it in vain to get a diamond shine.

    3. Do people realise that ICL/Fujitsu were involved in other monumental £multi-million IT fiascos, including Libra (MoJ) and Lorenzo (NHS).

    4. Who were they?

  7. […] I hope it will be of use to the public inquiry. I have already suggested that in the light of Dave McConnell’s evidence that Terry Austin might be recalled to the inquiry. I hope that in the light of what John Murray […]

Leave a Reply

Your email address will not be published. Required fields are marked *