Free EMR Newsletter Want to receive the latest news on EMR, Meaningful Use, ARRA and Healthcare IT sent straight to your email? Join thousands of healthcare pros who subscribe to EMR and EHR for FREE!

Major EHR System Downtime Causing Issues

Posted on January 28, 2014 I Written By

John Lynn is the Founder of the HealthcareScene.com blog network which currently consists of 10 blogs containing over 8000 articles with John having written over 4000 of the articles himself. These EMR and Healthcare IT related articles have been viewed over 16 million times. John also manages Healthcare IT Central and Healthcare IT Today, the leading career Health IT job board and blog. John is co-founder of InfluentialNetworks.com and Physia.com. John is highly involved in social media, and in addition to his blogs can also be found on Twitter: @techguy and @ehrandhit and LinkedIn.

For many years, I’ve been writing about the potential damage that EHR down time can have on an organization. I’ve also been writing a number of things to try and help organizations battle against EHR downtime. For example, here’s a few of the articles I’ve written: Cost of EHR Down Time, Reasons Your EHR Will Go Down, and SaaS EHR Down Time vs. In House EHR Down Time. The reality is that EHR down time is going to happen and as more organizations adopt EHR it’s going to happen more frequently.

The past week I’ve gotten a number of people emailing me about the pain it’s been having their EHR down. Here’s one message I got from a doctor:

Today, the whole system ground to a halt and froze screens for 10 to 15 seconds at a time, which made it impossible to get our documentation done on time when trying to see 15 to 20 patients each in a day. Both myself and my colleague weren’t able to finish any of our work. My colleague called the system “unbearable” today. Hence, we get to spend the weekend working to catch up.

I guess it could be worse. We could be paying for it.

Another doctor on the same system wrote:

I have had nothing but trouble with my EHR the last two days. I can’t enter new notes. Their Help Desk takes 10 minutes to get respond on line. Not happy at all.

The former doctor had a similar complaint about the EHR helpdesk, but described the help desk’s response as “denying any problems exist even though it’s obvious when the problems do exist.”

Sometimes it’s not even the EHR vendor’s fault that the EHR is down, but it still illustrates the pain of not being able to access your EHR. HIStalk broke the news of the Epic EHR downtime (caused by a network issue) that occurred at Martin Health System in Florida. Here’s the CIO’s response to HIStalk’s inquiry:

Martin Health System had a hardware failure that has resulted in our network being down. The failure occurred the evening of Jan. 22 and we are continuing to work on rectifying the situation. Epic is among the systems being impacted by this hardware failure, however, it was not the genesis of the problem. We are continuing operations as scheduled, while strictly monitoring any potential patient safety concerns or issues that would require appropriate care determinations to be made. Our patient care teams are following downtime procedures and protocols to ensure patient safety and proper documentation is provided.

HIStalk offered more insight on the downtime a few days later:

From Scooper: “Re: Martin Hospital. You scooped the main media on their EHR crash.” I just happened to have a reader with a friend who was admitted at the time and he passed the information along to me. CIO Ed Collins was nice to provide a response. The contact said it was chaos in the hospital, with confused employees assigning random numbers to patients, runners delivering paper copies of everything, medication errors occurring, and unhappy family members threatening to sue everything that moved (all unverified, of course.) The hospital says the problem was hardware, not Epic, and claims (as hospitals always do) that patient care wasn’t impacted. Of course patient care was impacted – the $80 million system that runs everything went down hard. It would be interesting for Joint Commission or state regulators to show up during one of these hospital outages anywhere in the country to provide an impartial view of how well the downtime process works. All that aside, downtime happens and the key is preparing for it, just like Interstate Highway construction and lane-closing accidents. It’s not a reason to drive a horse and buggy.

It amazes me that it took them from Wednesday night until Friday morning to recover from the downtime. That seems like a failure of downtime procedures. I do find all of this EHR downtime really interesting in light of the recent video interview I did with Jason Mendenhall discussing healthcare in the cloud and data centers. They guarantee 100% uptime for power and connectivity. However, even in a 100% uptime data center, that doesn’t mean the application software might not have its own issues. Although, it does remove some points of failure.

HIStalk is right that EHR downtime happens. The key really is being prepared for when it does happen with proper downtime procedures. Although, that doesn’t mean healthcare and EHR vendors can’t do more than we’re doing now to make it happen less often.

My issue isn’t with EHR downtime, but with preventable EHR downtime. Plus, let’s own up to when it happens and learn from the experience. I know how hard it is on a call with EHR support to explain when a software is “down.” Sure, the server might be up and running, but end users know when something isn’t running smoothly in their EHR. Trying to convince low level EHR support people that it’s indeed an issue is a real challenge. It’s so much easier to point fingers than to try and fix the problem.

Working Offline When Your EHR Isn’t Available

Posted on April 23, 2012 I Written By

John Lynn is the Founder of the HealthcareScene.com blog network which currently consists of 10 blogs containing over 8000 articles with John having written over 4000 of the articles himself. These EMR and Healthcare IT related articles have been viewed over 16 million times. John also manages Healthcare IT Central and Healthcare IT Today, the leading career Health IT job board and blog. John is co-founder of InfluentialNetworks.com and Physia.com. John is highly involved in social media, and in addition to his blogs can also be found on Twitter: @techguy and @ehrandhit and LinkedIn.

Many of you will likely remember my series of posts on EHR down time: Cost of EHR Down Time, Reasons Your EHR Will Go Down, and SaaS EHR Down Time vs. In House EHR Down Time. Needless to say, it’s pretty much inevitable that sooner or later you’re going to encounter EHR down time. The key to EHR down time is to think ahead about how you’re going to deal with your EHR being inaccessible.

I started thinking about this a bit more when I came across this FAQ item on Practice Fusion’s EMR user forums.

When there’s a planned maintenance ahead:
•Print your daily calendar for the next day’s schedule
•Know your offline alternatives for handling labs and prescriptions
•Have a plan to document your patient visits so you can input them in the EMR later
•Clear out your To do list and complete any pending Rx refill requests the day before
•Update your web browser and Adobe Flash to the current version

Preparing your office:
•Have a prepaid wireless 3G hub or other back-up internet system ready to go in the event your main internet is down
•Use laptops with good batteries and connect computers to surge protectors and battery back-ups for short term power interruptions
•Identify a second location that you could use temporarily in the case of a serious, long-term outage such as a fire or flood

I’ll always remember the reaction of the director of the health center where I first implemented an EMR to the discussion about “What do we do if the EMR is down?” She basically said, “We can still take care of the patient. We just might have to ask a few more questions.”

Now I’m sure there are cases where a physician might choose not to treat a patient without access to their EHR. There are certainly also cases where you can treat a patient better, faster and with more information with an EHR, but those can either be rescheduled if that’s the case. It’s certainly bad customer service and you should employ techniques to minimize EHR downtime as much as possible. My point is that it’s usually not life or death when the EHR is down. Think about how many patients are treated in an ER every day with no access to the patient’s medical record.

With that said, it is a disruption to the clinic and will be a BIG disruption to your clinic if you don’t have a solid plan of attack for when (not if) your EMR is inaccessible.

I’d focus your efforts in two areas:
•Minimize EMR Down Time
•Plan of Action for When Your EMR Goes Down

Most people do a pretty decent job with the first part. The second part people don’t often give much thought. You can start with some of the comments from Practice Fusion above to build out your plan. I also think it’s worth making a plan for short down time versus long down time. It’s quite different to deal with 5 minutes of down time than 5 days. You should consider both options.

SaaS EHR Down Time vs. In House EHR Down Time

Posted on August 9, 2011 I Written By

John Lynn is the Founder of the HealthcareScene.com blog network which currently consists of 10 blogs containing over 8000 articles with John having written over 4000 of the articles himself. These EMR and Healthcare IT related articles have been viewed over 16 million times. John also manages Healthcare IT Central and Healthcare IT Today, the leading career Health IT job board and blog. John is co-founder of InfluentialNetworks.com and Physia.com. John is highly involved in social media, and in addition to his blogs can also be found on Twitter: @techguy and @ehrandhit and LinkedIn.

As part of my continuing series of posts about EHR Down time (see my previous Cost of EHR Down Time and Reasons Your EHR Will Go Down posts), I thought it would be interesting to look at how a SaaS EHR down time is different from an in house EHR down time.

I’ll use the list of reasons your EHR go down as my discussion points for how it’s different with a SaaS EHR versus an in house EHR. On each point, I’ll see if either approach has an advantage over the other.

Power Outage – Certainly a power outage will impact both types of EHR implementations. If your computer or router doesn’t have power, then it doesn’t matter where your EHR is hosted. However, many clinics use laptops which can run for quite a while without being plugged in. Plus, a small UPS for your network equipment is pretty cheap and easy to implement.

However, a good UPS for your own server will cost a bit more to implement. Plus, the UPS won’t likely last very long. Most UPS are there to give you enough time to power down your system properly or to handle a short power outage. Of course, in this case we’re talking about a small clinic implementation. I have done an EMR implementation where we had some nice UPS and even a backup generator. However, this is the exception.

Conclusion: Slight Advantage for the SaaS EHR

Hard Drive Failure – Certainly the failure of a hard drive in your desktop machine will affect both types of EHR install equally. So, that part is a wash. However, the hard drive failure on your local server is much more of an issue than a SaaS EHR vendor. At least, I’ve never heard of a hard drive failure causing an issue for any SaaS software vendor of any type. Both in house and SaaS EHR implementations can implement redundant hard drives, but SaaS EHR vendors have to implement redundant servers.

Conclusion: Advantage SaaS EHR

Power Supply Failure – This one is similar to the Hard Drive failure. I know a lot of EHR vendors that have their clinics buy an in house server that doesn’t have redundant power supplies. I can’t imagine a SaaS EHR vendor buying a server without redundant power supplies even if the redundancy is across servers.

Conclusion: Advantage SaaS EHR

Network Cable – Cables can get pulled out of switches just as easily as servers. So, I conclude that it will affect SaaS EHR and in house EHR the same.

Conclusion: Tie

Switch/Router – Loss of a switch/router will cause either a SaaS EHR or in house EHR to go down.

Conclusion: Tie

Motherboard Failure – An in house server only has one motherboard. If that motherboard fails, you better hope you have a great tech support contract to get a motherboard to you quickly (For example, Dell has a 4 hour support contract which is amazing, but pricey). Certainly a motherboard can fail for a SaaS EHR as well, but since they likely have multiple servers, they can just roll the users over to another server while they replace the motherboard.

Conclusion: Advantage SaaS EHR

EHR Software Issue – This is a hard one to analyze since a software issue like this could happen on either type of EHR install. It really has more to do with the EHR vendor’s development and testing process than it has to do with the way the EHR software is delivered.

You could argue that because the SaaS EHR is all hosted by he company, they will be able to see the issues you’re having first hand and will have tested on the hardware they have in place. A client server/in house EHR install could be on a variety of EHR systems that the EHR vendor didn’t know about and couldn’t test as they developed and deployed the system. So, I could see a slight benefit for the SaaS EHR system.

However, one disadvantage to the SaaS EHR system is that they are hosting it across dozens of servers and so when something goes wrong on a server it’s sometimes hard to figure out what’s going wrong since all the servers are the same. Maybe that’s a bit of a stretch, but we’ve all seen times when certain users of a service are down, but not others.

Conclusion: Maybe a slight advantage to SaaS EHR

Internet Outage – This one is the most clear cut benefit to an in house server. When your internet connection goes down, the in house server keeps plugging along no problem. Loss of your internet connection with a SaaS EHR is terrible. No doubt that’s often the greatest weakness of a SaaS EHR. Although, it can be partially mitigated with multiple internet connections (ie. wired internet and wireless broadband internet).

Conclusion: Advantage In House EHR

I have to admit that I didn’t realize going into this analysis that it was going to be a landslide for the SaaS EHR. Although, that’s quite clear from this analysis. When it comes to EHR down time, the SaaS EHR is much better. Unless, you live in an area where the internet connection is unreliable and slow. Then, you don’t really have much choice since SaaS EHR needs a reliable internet connection.

It’s also worth noting that this article only talks about how EHR down time relates to SaaS EHR versus in house EHR. There are certainly plenty of other arguments that could be made for and against either implementation method such as: speed, privacy, security, cost, etc.

Reasons Your EHR Will Go Down

Posted on July 26, 2011 I Written By

John Lynn is the Founder of the HealthcareScene.com blog network which currently consists of 10 blogs containing over 8000 articles with John having written over 4000 of the articles himself. These EMR and Healthcare IT related articles have been viewed over 16 million times. John also manages Healthcare IT Central and Healthcare IT Today, the leading career Health IT job board and blog. John is co-founder of InfluentialNetworks.com and Physia.com. John is highly involved in social media, and in addition to his blogs can also be found on Twitter: @techguy and @ehrandhit and LinkedIn.

My previous post about the cost of EHR down time prompted me to think of all the ways that your EHR can go down. This might not be an exhaustive list, but hopefully it gives you an idea of the many many ways that your EMR can go down. With that knowledge hopefully you’ll be inspired to plan for EMR down time.

Reasons Your EMR Could Go Down:
Power Outage – Very few EMRs are setup to be able to handle a power outage. Sure, maybe you have a UPS attached to your server and some battery life in your laptop, but that’s only going to last so long. Plus, I bet your router/network switch isn’t on a UPS. What about your printer? You get the idea. EMR and power are friends.

Hard Drive Failure – At some point I asked someone why it was so common for hard drives to crash. They then described that hard drives are moving parts and anytime something is moving, there’s a higher chance that something will go wrong. Of course, now they have solid state hard drives that you can get. Either way, I’ve seen a lot of hard drives fail in my time. Of course, if you have a nice RAID setup for your hard drives, then you often won’t notice. Until they have to replace the failed hard drive with a new one. This could be the hard drive of your server or your computer. Most computers don’t have a RAID configuration.

Power Supply Failure – Most servers have redundant power supply. Why? Because they’ve been known to fail. If your server doesn’t have a redundant power supply, then be ready for down time. Most desktops don’t have redundant power supply and so they could easily fail too.

Network Cable – I don’t think I’ve ever seen a network cable go bad. However, I’ve certainly seen plenty of cables to servers bumped, moved, ripped out of the server. Everyone wonders why the server went down. Takes far too long to see that the network cable just got pulled out a little too much.

Switch/Router – Switches and Routers are what’s used in most offices to connect all your computers to the internet. Certainly these can go down if the power goes out. However, they can have other issues too. It’s not that common, but can sometimes cause down time for your office. Of course, wirelesss routers in particular can cause clinics lots of headaches when they go down.

Motherboard Failure – Might as well cover another common hardware failure: the motherboard. No motherboard and you can’t use your EMR.

EHR Software Issue – I’ve seen where a poorly tested and implemented EMR system would slowly use up all the memory on a server. Once it used up all the memory on the server, the EMR would take forever to do the simplest task. While not technically down since the server is up, it’s still a form of down time since users start refusing to use it in this state.

Internet Outage – This is particularly applicable to a SaaS EHR. Unless you have spent a lot of money to get redundant internet lines to your office (which in some locations is impossible), your internet will go down sooner or later. I don’t know how many times I’ve heard of some construction guy digging up the internet lines and causing outages for doctors offices. Happens all the time.

Much of the above can be applied to servers, desktops or laptops. Sometimes it causes a partial outage (ie. one laptop dies). Other times it takes down your whole EMR system (server dies). Either way, you should consider all these possibilities. Then, you see how you can minimize them (likely with the help of your IT support). Then, you think about what you’re going to do when the EMR down time happens.

Speaking from first hand experience, having a plan for EMR down time made all the difference when the event occurred.

Cost of EHR Down Time

Posted on July 19, 2011 I Written By

John Lynn is the Founder of the HealthcareScene.com blog network which currently consists of 10 blogs containing over 8000 articles with John having written over 4000 of the articles himself. These EMR and Healthcare IT related articles have been viewed over 16 million times. John also manages Healthcare IT Central and Healthcare IT Today, the leading career Health IT job board and blog. John is co-founder of InfluentialNetworks.com and Physia.com. John is highly involved in social media, and in addition to his blogs can also be found on Twitter: @techguy and @ehrandhit and LinkedIn.

Mark Anderson has an interesting article on Sys-con media about the cost of EHR down time. Among other analysis he makes the following assertion:
“One hour of software downtime can cost a practice almost $488.00 per physician”

EHR down time is something that I don’t think most doctors put much thought into when they are selecting an EHR. I think that putting a dollar sign next to it will help many doctors to really consider the impact of EHR down time on their clinic.

I believe it’s incredibly valuable for a clinic to seriously consider the impact of EHR down time will have on their clinic. I worked at one clinic where, while it was rather annoying, the patient care wasn’t terribly impacted by down time. They had to ask a few more questions, but in almost every case they could provide the patient care that was needed whether they had access to the EHR or not. Although, they did still have the cost of entering the data after the fact once the EHR came back. Thankfully, they never had more than a couple minutes down time at any point in the 5 years I worked with them.

The challenge of EHR down time is also greatly impacted by your choice of EHR. If you choose an in house EHR software, you’re down time planning will be very different than if you’re talking about a SaaS EHR. In fact, I think this topic is so important that I’m going to save it for a future post.

I’d be interested to hear if other people have tried to put a number on the cost of EHR down time. Did you get something similar to the $488/physician/hour that Mark Anderson suggests? If you have an EHR, how much down time have you experienced with your EHR? What were the causes of your EHR down time?