Our Disaster Recovery “Fire Drill”

Last Friday our practice had an opportunity to practice our disaster recovery protocol.  This was actually good news; we are replacing our 6 year old servers with virtual servers and a storage area network (SAN).  The implementation plan required more down time than a weekend would provide so we added a Friday to it.

Unlike a real disaster we had the opportunity to prepare immediately before, knowing exactly when the system was going down.  We printed our clinic schedules and printed the clinic note from the last visit for patients who had appointments.  We did not cancel appointments or reduce the number of patients seen.  For appointments requiring other documentation (i.e., pathology reports for post-surgery visits) we printed those.  We also printed a generous number of our most common handouts and got our paper prescription pads ready.

The day was free of major problems and the staff performed very well.  The experience was very interesting:

  1. My description above is a little too rosy.  We ended up searching for some documents at the last minute.  We ran behind schedule and there was a mild degree of disorganization throughout the day.
  2. Patients were very understanding about the delay and about our EMR being down.  Everyone understands that computers go down and I was thankful to see that our patients remembered that.  In some cases I had to ask patients to refresh my memory regarding prior visits since I didn’t have the entire chart available.  No one seemed to mind.
  3. I was still able to use my PC and Dragon Speech. Dragon runs locally on my desktop.  I dictated notes in Notepad and saved a separate note for each patient.  On Monday, my assistant will create Friday’s chart notes in the EMR, enter their data (vital signs etc.) and route the notes to my desktop.  Then I will copy / paste my notes into the EMR chart.
  4. Dictating unstructured chart notes into my PC was refreshing.  It was also an impressive reminder of how much garbage CPT forces us to add to our notes.  Even for complicated, new patients I was able to record everything relevant in less than two-thirds of a page.  Adding the CPT-required material almost doubles the size of the note without adding any relevant, useful data.
  5. With the system down the front office could focus on patient service without having to obsess over data entry.
  6. With all the extra paper floating around the back office was a mess.  I can’t imagine going back to paper charts.
  7. We were still compromised operationally when we had a patient who needed to schedule surgery.  Without the EMR workflow engine we could not print customized surgical packets.  Handwriting surgical consents is not acceptable.  We will have to catch up on this workflow next week.

Take home lessons:

We need to improve our disaster readiness, but at the moment our readiness is not too bad.  Our new SAN is configured to perform incremental backups every night and full backups every weekend.  The virtual servers and SAN will allow us to redirect workload in the event of a server or hard drive failure.  For a network failure we can follow the protocol we just rehearsed, but we should take the time to update our hard copies of handouts and surgical consents.  This is interesting because on more than one occasion we have had IT vendors try to push their way in to our practice using our presumed lack of disaster readiness as their excuse.

Once our new servers and SAN are settled in I may look into an automated method of copying appointment schedules and recent chart notes for patients with appointments to a local PC in each office rather than wasting all that time and paper every day.

I understand that all that front office data entry is a necessary evil.  But I never realized how much it distracts the front office from tending to patients.  I am going to double my efforts to make it attractive for patients to enter their own data online in advance of the visit – or at least have them use the patient portal to enter their own data in the waiting room.  As much as we have tried so far, our patient portal still has not caught on with our patients in the waiting room, and there has been only moderate use from home.  I don’t think our web portal is good enough yet.

About the author

Dr. Michael Koriwchak

Dr. Michael J. Koriwchak received his medical degree from Duke University School of Medicine in 1988. He completed both his Internship in General Surgery and Residency in Otolaryngology-Head and Neck Surgery at Vanderbilt University Medical Center. Dr. Koriwchak continued at Vanderbilt for a fellowship in Laryngology and Care of the Professional Voice. He is board certified by the American Board of Otolaryngology-Head and Neck Surgery.
After training Dr. Koriwchak moved to Atlanta in 1995 to become one of the original physicians in Ear, Nose and Throat of Georgia. He has built a thriving practice in Laryngology, Care of the Professional Voice, Thyroid/Parathyroid Surgery, Endoscopic Sinus Surgery and General Otolaryngology. A singer himself, many of his patients are people who depend on their voice for their careers, including some well-known entertainers. Dr. Koriwchak has also performed thousands of thyroid, parathyroid and head and neck cancer operations.
Dr. Koriwchak has been working with information technology since 1977. While an undergraduate at Bucknell University he taught a computer-programming course. In medical school he wrote his own software for his laboratory research. In the 1990’s he adapted generic forms software to create one the first electronic prescription applications. Soon afterward he wrote his own chart note templates using visual BASIC script. In 2003 he became the physician champion for ENT of Georgia’s EMR implementation project. This included not only design and implementation strategy but also writing code. In 2008 the EMR implementation earned the e-Technology award from the Medical Association of Georgia.
With 7 years EMR experience, 18 years in private medical practice and over 35 years of IT experience, Dr. Koriwchak seeks opportunities to merge the information technology and medical communities, bringing information technology to health care.

3 Comments

  • The above example is not really disaster recovery, but planned downtime. The problem for most practices is that is their server goes down, there is no copy of the last note or of the schedule to revert back to.

    Real Disaster Recovery requires a way to get functional in the case of a server failure or facility problem. Virtualization is the best way to ensure uptime in the case of hardware problems, but it still does not address what happens if the power goes out or a pipe breaks causing flooding.

  • Preparedness in the event of an outage is essential and – amidst the rush to EHR adoption – it’s often done too late, after a failure. In that regard Dr. Koriwchak’s approach and insights are excellent. So too is the comment from Mr. Bletnitsky regarding the implications of downtime.

    But more emphasis should be placed on reducing the risk of downtime in the first case. Call it preventive medicine for EHR. Yes, you need effective back up, recovery and continuity measures, like you need a doctor when you’re sick. But common sense is to try and avoid being sick. Or in this case, reduce the risk up front of downtime. Ironically, healthcare reform is ultimately trying to achieve this, so should the implementation of an EHR system.

    Virtualization doesn’t ensure uptime in the case of hardware problems. It efficiently uses resources but still requires an application and associated processes to restart – and if your SQL server is one of those processes that went down you’re looking at hours and not minutes of recovery even if everything goes perfectly. Too often things don’t however.

    Ensuring uptime requires three things – resilient technologies, proactive monitoring to identify and mitigate failures before they occur, and best practices. These can be achieved working with companies like Stratus and achieve lower costs with far less IT staff time and keep your EHR system healthy.

  • Sorry I didn’t read this blog until today Mike, and then only because, while starting my first blog on which students can complete assignments, I ran across you on “Blogger” and that I “follow” your blog. Thought you might find it interesting–scary even?–that we happened to pick the same “watermark” background. I chose it due to my affinity for Richard Bach’s short novel “Jonathan Livingston Seagull”. Hope all’s going well with you and yours.

    Henry

Click here to post a comment
   

Categories