Data centre failure left practices without SystmOne

Tags: BT CfH Contract CSC EMIS GP Solution SystmOne TPP

04 Oct 2007

A fault at a BT-run data centre left GP practices using TPP’s SystmOne experiencing performance problems with their system for three hours last month.

EHI Primary Care understands that the failure on September 18 affected all TPP’s SystmOne users from early morning until about 11am.

The problem is the second time in just over a year that large numbers of NHS organisations have had access to remotely-hosted systems disrupted in the North of England.

CSC, local service provider for the North Midlands and East (NME) Programme for IT, supplies TPP’s SystmOne to more than 600 practices as part of its contract with CfH and has overall responsibility for the service provided.

A joint statement issued to EHI Primary Care by CSC and TPP said: “On the 18th September during proactive monitoring, TPP became aware of a slow-down in performance for SystmOne clients and fully investigated the issue. The slow-down in performance lasted for three hours, with approximately 10 minutes outage while the affected devices were rebooted to resolve the fault.

"SystmOne is hosted in a data centre provided by BT which was where the issue occurred. The hosting arrangement is between TPP and BT and forms the basis of the SystmOne solution provided by CSC.”

Two SystmOne practices contacted by EHI Primary Care confirmed that they had problems with their systems until about 11am on September 18.

Stephen Blackman, practice manager at the Earls Barton Medical Centre in Northamptonshire, said: “We came in and everything was going slow and we were told it was a national problem to do with BT. It did cause us problems because the system kept collapsing and there was very limited functionality.”

Blackman said the system came back on at about 11am. He added: “It wasn’t a major problem and when we had EMIS before we had similar things where our server went down. Now it can be sorted out remotely so I would still rather be with TPP.”

Dr Mark Shenton, a GP at Stow Health Centre in Newmarket, said he understood the system at his practice had also not been usable for a couple of hours.

He told EHI primary Care: “At the end of the day these things happen and it just involved working in the same way as if there was a cut in the electricity supply. We have got a paper system for if there is ever a computer problem or an electricity supply problem and then the paper is scanned in at the end of the day. You just have to be a bit more vigilant when you see patients.”

A spokesperson for CfH told EHI Primary Care: “It can be confirmed that a fault in the data centre for one of the GP systems caused some performance issues for users of that system for a period of approximately two hours. This was not an N3 fault, or with the N3 connection to the data centre and was diagnosed and remedied by the support teams for that system."

The problem is the second time CSC has been involved in a data centre failure. Last summer 80 trusts including eight acute hospitals were left without access to patient data due to a technical failure at a data centre run by CSC.

Link

CSC failure leaves 80 trusts without IT systems

Reader's Comments

Remote vs local servers

maryhawking@tigers.demon.co.uk

05 Oct 07 14:10

I suppose this re-opens the same old argument. If your practice data is on a remote server, it may be better backed up and the server may be better managed than if you had a server in the practice. On the other hand, your practice is vulnerable to interruption of connection to the server - any thing from a JBC, the Manchester tunnel fire, N3 overload and remote server problems. In the larger scale of things, if a single practice server has problems no-one would notice, but if the problem affects a central server, presumably all the practices with their records on that server will be equally affected. Surely the EMIS solution of practice server with mirroring in real time to a secure remote server gives the best of both worlds? It *is* worrying that remote LSP servers seem to be prone to problems: or is this about the level of problems one might expect?

Perceptions

09 Oct 07 10:10

The slowdowns and 10 minute outage were what the Datacentre experienced, this translates into an unusable system to the users on the ground for a much longer period. Presumably the targets and default payments are based on the centralist view.

Locally hosted servers can have problems, but their impact is much more limited, and I think the frequency and impact of these Data Centre failures undermines the alleged benefits.

Resilience

gji@nhs.net

09 Oct 07 19:10

The arguments in favour of a well hosted remote server outweigh those against for the vast majority of practices. However, a system where there is not inbuilt resilience like this example, or where patient confidentiality is not respected in other PCT hosted examples undermine the confidence of the profession (and the public) in them.

Dr Grant Ingrams

Is that the main reason?

12 Oct 07 11:10

I believe that the real reason for EMIS suggesting this "mirroring" may be for other reasons too.

EMIS LV (about 5000 practices nationally) doesn't seem to be capable of being centrally hosted to obtain GPSoC level 4 compliance (and thus obtain IM&T DES Component 4 payments). Also, EMIS PCS (EMIS upgrade path for LV users) does not not appear to be as functionally rich or as well liked by it's users as LV is.

A catch 22 for EMIS is that they will lose business (and brand loyalty) if they force LV users onto PCS, but they will also start losing business if practices with LV are left behind and can't integrate or claim additional DES payments.

I can see why they are pushing this "mirroring" as a solution, as this would allow them to keep LV on the local Server, and stream data centrally into their EMIS Web product to meet DES payment and future intergration requirements.

Interestingly too, most PCTs only pay maintenance costs to EMIS (and other suppliers too) for the main clinical server. The average cost of this is around £1,000 per practice per year (so my guess is that this represents a revenue stream of about £5m for EMIS each year). In addition, the EMIS AV update service costs around £100 per practice per year, even though the licence cost from McAfee is only £10). Multiplied up, this represents another revenue stream of about £450K pa for EMIS. Understandably if this support is not taken by a practice or PCT and the practice server goes down, EMIS (and other suppliers) will not support the practice to get them up and running again - it's a sort of recognised insurance policy.

It appears to me that without this local "mirrored solution" in practices, EMIS is likely to lose a lot of its customer base, and will also lose a revenue stream of around £5.5m p.a. Are we sure this is only about minimising risks to patient safety?

Re: Is that the main reason?

rsullivan@ingleboro.com

16 Oct 07 00:10

After such a long and imaginative comment, I think it only fair to correct a few things you say.

The recommended upgrade path from LV is to LV web and then EMIS web, for PCS it is PCS web and then EMIS web. Practices are not under any pressure to upgrade in the foreseeable future.

Local mirroring is an option a practice can take on if they wish. PCTs will have the choice of whether they pay for it or not - it isn't in the GPSoC infrastructure agreement as far as I know. If the PCT doesn't pay, the individual practice can decide to pay for it if they wish, it's up to them, and it will have no bearing on the service or licence costs paid by PCTs and CfH.

In other words local mirroring is an extra service to improve business continuity and disaster recovery that EMIS will offer. PCTs and practices will take a view of whether they want to pay for it.

Even practices with a fully hosted system need anti-virus software and it will be paid for by CfH as a core item in the infrastructure agreement at prices agreed by CfH with all suppliers.

Re: Re: Is that the main reason?

23 Oct 07 14:10

I stand corrected on the upgrade path, in that I was not aware that the LV to PCS upgrade route was no longer a strategy for EMIS.

None the less, the comments about "mirroring" still seem to stand given that LV Web and PCS Web are still reliant on a local clinical server being present in the practice which streams data to a central repositiry. Also, AV is still not part of the GPSoC programme, so is an additional local cost.

I am not saing that I either approve or dissaprove of a local mirrored solution in practices. I was however trying to point out that it is likely there are other factors at play here other than the righteous cause of patient safety.

What I find interesting, which has also been mentioned in another response on this site, is the power of the EMIS PR machine and it's ability to influence...

EMIS PR machine.

25 Oct 07 12:10

The previous poster is correct to comment on the power of the EMIS PR machine. The reason that its power is so often underestimated is that it is made up not only of many thousands of GPs who are satisfied EMIS customers, but also very many (like myself) who are dissatisfied customers of their competitors.

Streaming is not Hosting

26 Oct 07 17:10

I cannot see how the EMIS local server streaming data to a centrally hosted datawarehouse (which does not seem to have any functionality as a full clinical system) can be considered as meeting the requirements for the IMT DES component 4 - which requires pratices to migrate to a CFH accredited hosted system.

Replicating the data centrally is not the same as a hosted system !

Streaming to EMIS Web

27 Oct 07 11:10

EMIS Web is still under development and streaming is one way only from LV to EMIS Web. I agree that this does not constitute hosting, but the intention is that EMISWeb will become the hosted solution and at some point, when the product is fully functional, practices will be able to switch off their server. The advantage of this route is that the learning curve can be gradual and the migration can occur at a rate suitable to individual Practices.