Yesterday I commented on the status for the Property Registration System and mentioned, that another major IT failure seemed to have been hiding from the public eye – namely the wreckage of the Danish Emergency Management System. Normally I don't claim to possess a crystal ball, but today the news hit the World: Under the heading: ' The Prime Minister knew that the Emergency Management System was high risc', a paper from the Ministry of Finance dated May 16 2007 documented that the Minister of Finance, the current Prime Minister knew that the Emergency Management Project, a Dkr. 2 B + project, was evaluated at high risk, 4 out of a scale 1 to 5.
This document was originally confidential, confidentiality raised in August 2007, when the contract with the consortium led by Motorola was signed. But the contract – although a major investment – did not cover the 'control room SW', which was out for tender and not signed before 2008.
This contract was won by Terma, a Danish company until then only known for it's supply of high tech military systems, radar controlled defense systems and firearms control systems for helicopters, fighters and ships. The contract value was at the time estimated to be a mere 200 – 250 mio Dkr. For the Control Room SW including 4 years of operation, as opposed to a total budget for the entire emergency management system at 2.718 Mio Dkr.
At the time of the evaluation, the Ministry of Finance could not include Terma and the contracted solution (that also included Intergraph and a couple of smaller, Danish companies) in the risk evaluation. So the reason a '4' was the result, was a combination of 4 for organizational complexity, 4 for technical solutions, 3 for vendor risk and 5 for complexity of stakeholder interest.
As a contingency plan to reduce/meet the risk, 193 Mio was set aside as a buffer, and the conclusion was, that if any of risks materialized, the consequences were thought only to impact the project plan and the budget.
At this point in time it is clear beyond any doubt, that although the control room SW contract only covered a mere 10% of the total project cost, this part of the project has failed completely, Terma's Staff for has been fired, and the department Public Safety, created for the occasion, has been dismantled. The Government Attorney is currently handling the case.
But as I mentioned in a blog, the Government Attorney is no programmer, and this will not create anything but a financial resolution of the conflict between Terma and the Government.
The secretariat set up to handle the emergency system – SINE – has been radio dead around this.
See their homepage at http://www.sikkerhedsnet.dk/om+sine.
So how can we determine what went wrong? If you look at the SINE homepage you will find a link to European Tetra-net projects that more or less all seem to have succeeded. What went wrong in Denmark? What is different and why couldn't we just adopt this seemingly uninteresting piece of technology and enable the fire brigade to talk to the police and the rescuing service to the civil defense units? There are several answers to this question:
Underestimation of the importance of the 'control room SW'.
A very complicated and partly conflicting group of stakeholders
A lack of a systemic overview of both tenders before call for tender
An steering committee consisting of gatekeepers instead of decision makers
Lack of understanding of the competence level of the winning company (Terma)
Lack of understanding of technological improvements in the emergency mgmt arena
As is clear from the international references, the Tetra net is not a very complicated technology – it has a rather small bandwidth, but in case of emergency you could deal with only limited amount of data and voice interchange. Also Motorola is known to have skills and competence in this area, but the exchange between different organizations was explicitly placed in the second call – 'control room SW' – that also covered integration with GIS (Intergraph) as well as links to all the different organizations variety of communication media – plus an integration layer to back end systems like the Central Personal Registration system, interface to local office systems, hospital exchange, and interfaces to privately operated ambulance services and in some areas also fire brigades.
The complicated mixture of stakeholders were supposed to be coordinated by the Police, that although in practice more than 80% of all cases of (day-to-day) emergency management calls do not involve the police. And the police was in the middle of a major reorganisation, where this system was just another brick in the wall. The tender was headed by the Agency of Economy, part of the ministry of Finance, and the Emergency Management organization was merged with the Ministry of Defense a few years earlier and was probably looked upon as a sort of step child here. The complication also included the above mentioned private ambulances and fire brigades, covering about 50% or more of the local Government emergency services.
The lack of systemic overview before the 2 tenders calls for a reevaluation of how and why Gartner Group, hired as a consultant company to prepare the tender, could overlook the key role of 'Control Room SW' and underplay the importance of technical interoperability between the different stakeholders and not least the level and complexity of change management to ensure the take-up of the solution by all major players.
It seemed that the steering committee probably covered the right institutions: Ministry of Justice (which includes the Police) acting as chairman, Ministry of Finance, Ministry of Defense, Ministry of Welfare, Ministry of Science and Technology, Association of Municipalities and Danish Regions Association. Until proven wrong it is my hypothesis that the steering committee meetings concentrated more on status of progress, spending of money rather than risk management and change management like acceptance and enacting of the various groups 'in the fire' literally speaking. If stakeholder committees act as gatekeepers, it is at best an advisory committee, not a steering committee. In stead the head of this group should have been a dedicated 'CZAR' with sufficient decision power to change events – if possible by changing the contract.
Lack of understanding on the proper level of competencies and vendor capabilities. Terma had never before worked in the field of Public Safety; nor had they ever participated in major SW integration tasks with administrative stakeholders – their use of standard systems was and is 'Java'. When the contract negotiations squeezed Terma, they skipped partners that could have provided the skills and muscle, that Terma obviously did not possess when the signals of crisis were raised. And because of the nature of public tenders, they were allowed by a steering committee to continue for too long.
The lack of understanding of concurrent technological development was another reason, why the course was not changed; over the last few years the terminology 'Unified Communications' and 'Collaboration Services' has been greatly improved, mainly led by the defense forces, but using standard SW components and basing communication of the Internet. To day at least 2 vendors offer SW capability for multi-platform same-time support of communication systems supporting both Tetra Net (Motorola), standard mobile communication channels as well as highly sophisticated and encrypted VPN tools for same-time video conferencing, exchange and overlay of Geographic information. ( See CISCO Solution, IBM solution )But nobody was allocated to watch this field while the hand coding of Terma progressed. This could actually have saved the project – if attention to the acceptance of the solution by all stakeholders also had been in focus.