Free EMR Newsletter Want to receive the latest news on EMR, Meaningful Use, ARRA and Healthcare IT sent straight to your email? Join thousands of healthcare pros who subscribe to EMR and EHR for FREE!

What is Quality in Health Care? (Part 2 of 2)

Posted on February 10, 2016 I Written By

Andy Oram is an editor at O'Reilly Media, a highly respected book publisher and technology information provider. An employee of the company since 1992, Andy currently specializes in open source, software engineering, and health IT, but his editorial output has ranged from a legal guide covering intellectual property to a graphic novel about teenage hackers. His articles have appeared often on EMR & EHR and other blogs in the health IT space. Andy also writes often for O'Reilly's Radar site (http://oreilly.com/) and other publications on policy issues related to the Internet and on trends affecting technical innovation and its effects on society. Print publications where his work has appeared include The Economist, Communications of the ACM, Copyright World, the Journal of Information Technology & Politics, Vanguardia Dossier, and Internet Law and Business. Conferences where he has presented talks include O'Reilly's Open Source Convention, FISL (Brazil), FOSDEM, and DebConf.

The first part of this article described different approaches to quality–and in fact to different qualities. In this part, I’ll look at the problems with quality measures, and at emerging solutions.

Difficulties of assessing quality

The Methods chapter of a book from the National Center for Biotechnology Information at NIH lays out many of the hurdles that researchers and providers face when judging the quality of clinical care. I’ll summarize a few of the points from the Methods chapter here, but the chapter is well worth a read. The review showed how hard it is to measure accurately many of the things we’d like to know about.

For instance, if variations within a hospital approach (or exceed) the variations between hospitals, there is little benefit to comparing hospitals using that measure. If the same physician gets wildly different scores from year to year, the validity of the measure is suspect. When care is given by multiple doctors and care teams, it is unjust to ascribe the outcome to patient’s principal caretaker. If random variations outweigh everything, the measure is of no use at all. One must also keep in mind practical considerations, such as making sure the process of collecting data would not cost too much.

Many measures apply to a narrow range of patients (for instance, those with pneumonia) and therefore may be skewed for doctors with a relatively small sample of those patients. And a severe winter could elevate mortality from pneumonia, particularly if patients have trouble getting adequate shelter and heat. In general, “For most outcomes, the impacts of random variation and patient factors beyond providers’ control often overwhelm differences attributable to provider quality.” ACMQ quality measures “most likely cannot definitively distinguish poor quality providers from high quality providers, but rather may illuminate potential quality problems for consideration of further investigation.”

The chapter helps explain why many researchers fall back on standard of care. Providers don’t trust outcome-based measures because of random variations and factors beyond their control, including poverty and other demographics. It’s hard even to know what contributed to a death, because in the final months it may not have been feasible to complete the diagnoses of a patient. Thus, doctors prefer “process measures.”

Among the criteria for evaluating quality indicators we see, “Does the indicator capture an aspect of quality that is widely regarded as important?” and more subtly, “subject to provider or public health system control?” The latter criterion heed physicians who say, “We don’t want to be blamed for bad habits or other reasons for noncompliance on the part of our patients, or for environmental factors such as poverty that resist quick fixes.”

The book’s authors are certainly aware of the bias created by gaming the reimbursement system: “systematic biases in documentation and coding practices introduced by awareness that risk-adjustment and reimbursement are related to the presence of particular complications.” The paper points out that diagnosis data is more trustworthy when it is informed by clinical information, not just billing information.

One of the most sensitive–and important–factors in quality assessment is risk adjustment, which means recognizing which patients have extra problems making their care more difficult and their recovery less certain. I have heard elsewhere the claim that CMS doesn’t cut physicians enough slack when they take on more risky patients. Although CMS tries to take poverty into account, hospital administrators suspect that institutions serving low-income populations–and safety-net hospitals in particular–are penalized for doing so.

Risk adjustment criteria are sometimes unpublished. But the most perverse distortion in the quality system comes when hospitals fail to distinguish iatrogenic complications (those introduced by medical intervention, such as infections incurred in the hospital) from the original diseases that the patient brought. CMS recognizes this risk in efforts such as penalties for hospital-acquired conditions. Unless these are flagged correctly, hospitals can end up being rewarded for treating sicker patients–patients that they themselves made sicker.

Distinguishing layers of quality

Theresa Cullen,associate director of the Regenstrief Institute’s Global Health Informatics Program, suggests that we think of quality measures as a stack, like those offered by software platforms:

  1. The bottom of the stack might simply measure whether a patient receive the proper treatment for a diagnosed condition. For instance, is the hemoglobin A1C of each diabetic patient taken regularly?

  2. The next step up is to measure the progress of the first measure. How many patients’ A1C was under control for their stage of the disease?

  3. Next we can move to measuring outcomes: improvements in diabetic status, for instance, or prevention of complications from diabetes

  4. Finally, we can look at the quality of the patient’s life–quality-adjusted life years.

Ultimately, to judge whether a quality measure is valid, one has to compare it to some other quality measure that is supposedly trustworthy. We are still searching for measures that we can rely on to prove quality–and as I have already indicated, there may be too many different “qualities” to find ironclad measures. McCallum offers the optimistic view that the US is just beginning to collect the outcomes data that will hopefully give us robust quality measures, Patient ratings serve as a proxy in the interim.

When organizations claim to use quality measures for accountable care, ratings, or other purposes, they should have their eyes open about the validity of the validation measures, and how applicable they are. Better data collection and analysis over time should allow more refined and useful quality measures. We can celebrate each advance in the choices we have for measures and their meanings.

What is Quality in Health Care? (Part 1 of 2)

Posted on February 9, 2016 I Written By

Andy Oram is an editor at O'Reilly Media, a highly respected book publisher and technology information provider. An employee of the company since 1992, Andy currently specializes in open source, software engineering, and health IT, but his editorial output has ranged from a legal guide covering intellectual property to a graphic novel about teenage hackers. His articles have appeared often on EMR & EHR and other blogs in the health IT space. Andy also writes often for O'Reilly's Radar site (http://oreilly.com/) and other publications on policy issues related to the Internet and on trends affecting technical innovation and its effects on society. Print publications where his work has appeared include The Economist, Communications of the ACM, Copyright World, the Journal of Information Technology & Politics, Vanguardia Dossier, and Internet Law and Business. Conferences where he has presented talks include O'Reilly's Open Source Convention, FISL (Brazil), FOSDEM, and DebConf.

Assessing the quality of medical care is one of the biggest analytical challenges in health today. Every patient expects–and deserves–treatment that meets the highest standards. Moreover, it is hard to find an aspect of health care reform that does not depend on accurate quality measurement. Without a firm basis for assessing quality, how can the government pay Accountable Care Organizations properly? How can consumer choice (the great hope of many reformers) become viable? How can hospitals and larger bodies of researchers become “learning health systems” and implement continuous improvement?

Ensuring quality, of course, is crucial in a fee-for-value system to ensure that physicians don’t cut costs just by withholding necessary care. But a lot of people worry that quality-based reimbursement plans won’t work. As this article will show, determining what works and who is performing well are daunting tasks.

A recent op-ed claims that quality measures are adding unacceptable stress to doctors, that the metrics don’t make a difference to ultimate outcomes, that the variability of individual patients can’t be reflected in the measures, that the assessments don’t take external factors adequately into account, and that the essential element of quality is unmeasurable.

Precision medicine may eventually allow us to tailor treatments to individual patients with unique genetic prints. But in the meantime, we’re guessing a lot of the time we prescribe drugs.

The term quality originally just distinguished things of different kinds, like the Latin word qualis from which it is derived. So there are innumerable different qualities (as in “The quality of mercy is not strained”). It took a while for quality to be seen as a single continuum, as in an NIH book I’ll cite later, which reduces all quality measures to a single number by weighting different measures and combining them. Given the lack of precision in individual measures and the subjective definitions of quality, it may be a fool’s quest to seek a single definition of quality in health care.

Many qualities in play
Some of the ways to measure quality and outcomes include:

  • Longitudinal research: this tracks a group of patients over many years, like the famous Framingham Heart Study that changed medical care. Modern “big data” research carries on this tradition, using data about patients in the field to supplement or validate conventional clinical research. In theory, direct measurement is the most reliable source of data about what works in public health and treatment. Obvious drawbacks include:

    • the time such studies take to produce reliable results

    • the large numbers of participants needed (although technology makes it more feasible to contact and monitor subjects)

    • the risk that unknown variations in populations will produce invalid results

    • inaccuracies introduced by the devices used to gather patient information

  • Standard of care: this is rooted in clinical research, which in turn tries to ensure rigor through double-blind randomized trials. Clinical trials, although the gold standard in research, are hampered by numerous problems of their own, which I have explored in another article. Reproducibility is currently being challenged in health care, as in many other areas of science.

  • Patient ratings: these are among the least meaningful quality indicators, as I recently explored. Patients can offer valuable insights into doctor/patient interactions and other subjective elements of their experience moving through the health care system–insights to which I paid homage in another article–but they can’t dissect the elements of quality care that went into producing their particular outcome, which in any case may require months or years to find out. Although the patient’s experience determines her perception of quality, it does not necessarily reflect the overall quality of care. The most dangerous aspect of patient ratings, as Health IT business consultant Janice McCallum points out, comes when patients’ views of quality depart from best practices. Many patients are looking for a quick fix, whether through pain-killers, antibiotics, or psychotropic medications, when other interventions are called for on the basis of both cost and outcome. So the popularity of ratings among patients just underscores how little we actually know about clinical quality.

Quality measures by organizations such as the American College of Medical Quality (ACMQ) and National Committee for Quality Assurance (NCQA) depend on a combination of the factors just listed. I’ll look more closely at these in the next part of this article.

Clinical Decision Support Should Be Open Source

Posted on January 26, 2015 I Written By

Andy Oram is an editor at O'Reilly Media, a highly respected book publisher and technology information provider. An employee of the company since 1992, Andy currently specializes in open source, software engineering, and health IT, but his editorial output has ranged from a legal guide covering intellectual property to a graphic novel about teenage hackers. His articles have appeared often on EMR & EHR and other blogs in the health IT space. Andy also writes often for O'Reilly's Radar site (http://oreilly.com/) and other publications on policy issues related to the Internet and on trends affecting technical innovation and its effects on society. Print publications where his work has appeared include The Economist, Communications of the ACM, Copyright World, the Journal of Information Technology & Politics, Vanguardia Dossier, and Internet Law and Business. Conferences where he has presented talks include O'Reilly's Open Source Convention, FISL (Brazil), FOSDEM, and DebConf.

Clinical decision support is a long-standing occupant of the medical setting. It got in the door with electronic medical records, and has recently received a facelift under the term “evidence based medicine.” We are told that CDS or EBM is becoming fine-tuned and energized through powerful analytics that pick up the increasing number of patient and public health data sets out in the field. But how does the clinician know that the advice given for a treatment or test is well-founded?

Most experts reaffirm that the final word lies with the physician–that each patient is unique, and thus no canned set of rules can substitute for the care that the physician must give to a patient’s particular conditions (such as a compromised heart or a history of suicidal ideation) and the sustained attention that the physician must give to the effects of treatment. Still, when the industry gives a platform to futurists such as Vinod Khosla who suggest that CDS can become more reliable than a physician’s judgment, we have to start demanding a lot more reliability from the computer.

It’s worth stopping a moment to consider the various inputs to CDS. Traditionally, it was based on the results of randomized, double-blind clinical trials. But these have come under scrutiny in recent years for numerous failings: the questionable validity of extending the results found on selected test subjects to a broader population, problems reproducing results for as many as three quarters of the studies, and of course the bias among pharma companies and journals alike for studies showing positive impacts.

More recently, treatment recommendations are being generated from “big data,” which trawl through real-life patient experiences instead of trying to isolate a phenomenon in the lab. These can turn up excellent nuggets of unexpected impacts–such as Vioxx’s famous fatalities–but suffer also from the biases of the researches designing the algorithms, difficulties collecting accurate data, the risk of making invalid correlations, and the risk of inappropriately attributing causation.

A third kind of computerized intervention has recently been heralded: IBM’s Watson. However, Watson does not constitute CDS (at least not in the demo I saw at HIMSS a couple years ago). Rather, Watson just does the work every clinician would ideally do but doesn’t have time for: it consults thousands of clinical studies to find potential diagnoses relevant to the symptoms and history being reported, and ranks these diagnoses by probability. Both of those activities hijack a bit of the clinician’s human judgment, but they do not actually offer recommendations.

So there are clear and present justifications for demanding that CDS vendors demonstrate its reliability. We don’t really know what goes into CDS and how it works. Meanwhile, doctors are getting sick and tired of bearing the liability for all the tools they use, and the burden of their malpractice insurance is becoming a factor in doctors leaving the field. The doctors deserve some transparency and auditing, and so do the patients who ultimately incorporate the benefits and risks of CDS into their bodies.

CDS, like other aspects of the electronic health records into which it is embedded, has never been regulated or subjected to public safety tests and audits. The argument trotted out by EHR vendors–like every industry–when opposing regulation is that it will slow down innovation. But economic arguments have fuzzy boundaries–one can always find another consideration that can reverse the argument. In an industry that people can’t trust, regulation can provide a firm floor on which a new market can be built, and the assurance that CDS is working properly can open up the space for companies to do more of it and charge for it.

Still, there seems to be a pendulum swing away from regulation at present. The FDA has never regulated electronic health records as it has other medical software, and has been carving out classes of medical devices that require little oversight. When it took up EHR safety last year, the FDA asked merely for vendors to participate voluntarily in a “safety center.”

The prerequisite for gauging CDS’s reliability is transparency. Specifically, two aspects should be open:

  • The vendor must specify which studies, or analytics and data sets, went into the recommendation process.

  • The code carrying out the recommendation process must be openly published.

These fundamentals are just the start of of the medical industry’s responsibilities. Independent researchers must evaluate the sources revealed in the first step and determine whether they are the best available choices. Programmers must check the code in the second step for accuracy. These grueling activities should be funded by the clinical institutions that ultimately use the CDS, so that they are on a firm financial basis and free from bias.

The requirement for transparent studies raises the question of open access to medical journals, which is still rare. But that is a complex issue in the fields of research and publishing that I can’t cover here.

Finally, an independent service has to collect reports of CDS failures and make them public, like the FDA Adverse Event Reporting System (FAERS) for drugs, and the FDA’s Manufacturer and User Facility Device Experience (MAUDE) for medical devices.

These requirements are reasonably light-weight, although instituting them will seem like a major upheaval to industries accustomed to working in the dark. What the requirements can do, though, is put CDS on the scientific basis it never has had, and push forward the industry more than any “big data” can do.

Consumers Are Still Held Back From Making Rational Health Decisions

Posted on November 25, 2014 I Written By

Andy Oram is an editor at O'Reilly Media, a highly respected book publisher and technology information provider. An employee of the company since 1992, Andy currently specializes in open source, software engineering, and health IT, but his editorial output has ranged from a legal guide covering intellectual property to a graphic novel about teenage hackers. His articles have appeared often on EMR & EHR and other blogs in the health IT space. Andy also writes often for O'Reilly's Radar site (http://oreilly.com/) and other publications on policy issues related to the Internet and on trends affecting technical innovation and its effects on society. Print publications where his work has appeared include The Economist, Communications of the ACM, Copyright World, the Journal of Information Technology & Politics, Vanguardia Dossier, and Internet Law and Business. Conferences where he has presented talks include O'Reilly's Open Source Convention, FISL (Brazil), FOSDEM, and DebConf.

Price and quality of care–those are what we’d like to know when we need a medical procedure. But a perusal of a recent report from the Government Accountability Office reminded me that both price and quality information are hard to get nowadays.

This has to make us all a little leery about trends in health reform. Governments, insurers, and employers want us to get choosy about where we have our procedures. They justify rises in copays and deductibles by saying, “You patients should start to take responsibility for the costs of your own health care.”

Yeah, as responsible as a person looking for his car keys in the dark. Let’s start with prices, which in many countries are uniform and are posted on the clinic wall.

Sites such as Clear Health Costs and Castlight Health prove what we long knew anecdotally: charges in the US vary vertiginously among different institutions. Anyone who had missed that fact would have been enlightened by Steven Brill’s 2013 Time Magazine article.

But aspirations become difficult when we get down to the issue at hand–choosing a provider. That’s because US insurance and reimbursement systems are also convoluted. We don’t know whether a hospital will charge our insurer their official price, or how much the insurer will cover. It might feel righteous to punish a provider with high posted prices (or prices reported by other consumers), but most patients have a different goal: to keep as much of their own money as they can.

We can gauge the depth of the cost problem from one narrow suggestion made in the GAO report that yet could help a lot of health consumers: the suggestion that Centers for Medicare & Medicaid Services (CMS) publish out-of-pocket expenditures for Medicare recipients as well as raw costs of procedures (page 31). Even this is far from simple. HHS pointed out that 90% of Medicare patients have supplemental overage that reduces their out-of-pocket expenditures (page 43). Tracking all the ancillary fees is also a formidable job.

Castlight Health is out in front when it comes to measuring the real impact of charges on consumer. They achieve great precision by hooking up with employers. Thus, they know the insurer and the precise employer plan that covers each individual visiting their site, and can take deductibles, exclusions, and caps into account when calculating the cost of a procedure. A recent study found that Castlight users enjoyed lower costs, especially for labs and imaging. Some nationwide system built around standards for reporting these things could unpack the cost conumdrum for all patients.

Let’s turn to quality. As one might expect, it’s always a slippery concept. The GAO report pointed out that quality may be measured in different ways by different providers (page 26). A recently begun program releases Medicare data on mortality and readmissions, but it hasn’t been turned into usable consumer information yet (pages 27-28). Two more observations from the report:

  • “…with the exception of Hospital Compare, none of CMS’s transparency tools currently provide information on patient-reported outcomes, which have been shown to be particularly relevant to consumers considering common elective medical procedures, including hip and knee replacements.” (Page 21)

  • “CMS’s consumer testing has focused on assessing the ability of consumers to interpret measures developed for use by clinicians, rather than to develop or select measures that specifically address consumer needs.” (Page 25)

Some price-check sites simply don’t try to measure quality. A highly publicized crowdsourcing effort by California radio station KQED, based on the Clear Health Costs service, admitted that quality measures were not available but excused themselves by citing the well-known lack of correlation between price and quality.

Price and quality may not be related, but that doesn’t relieve consumers of concerns over quality. Can you really exchange Mount Sinai Hospital in New York for Daddy-o’s Fix-You-Up Clinic based on price alone? Without robust and reliable quality data, people will continue choosing the historically respected hospitals with the best marketing and PR departments–and the highest prices.

A recent series on health care costs concludes by admonishing consumers to “get in the game and start to push back.” The article laments the passivity of consumers in seeking low-cost treatment, but fails to cite the towering barriers that stand in the way.

The impasse we’ve reached on consumer choice, driven by lack of data, reflects similar problems with analytics throughout the health care field. For instance, I recently reported on how hard a time researchers have obtaining and making use of patient data. Luckily, the GAO report cites several HHS efforts to enhance their current data on price and quality. Ultimately, of course, what we need is a more rational reimbursement system, not a gleaming set of computerized tools to make the current system more transparent. Let’s start by being honest about what we’re asking health consumers to achieve.

Are Limited Networks Necessary to Reduce Health Care Costs?

Posted on September 10, 2014 I Written By

Andy Oram is an editor at O'Reilly Media, a highly respected book publisher and technology information provider. An employee of the company since 1992, Andy currently specializes in open source, software engineering, and health IT, but his editorial output has ranged from a legal guide covering intellectual property to a graphic novel about teenage hackers. His articles have appeared often on EMR & EHR and other blogs in the health IT space. Andy also writes often for O'Reilly's Radar site (http://oreilly.com/) and other publications on policy issues related to the Internet and on trends affecting technical innovation and its effects on society. Print publications where his work has appeared include The Economist, Communications of the ACM, Copyright World, the Journal of Information Technology & Politics, Vanguardia Dossier, and Internet Law and Business. Conferences where he has presented talks include O'Reilly's Open Source Convention, FISL (Brazil), FOSDEM, and DebConf.

Among the dirty words most hated by health care consumers–such as “capitation” and “insufficient medical necessity”–a special anxiety infuses the term “out-of-network.” Everybody harbors the fear that the world-famous specialist who can provide a miracle cure for a rare disease he or she may unexpectedly suffer from will be unavailable due to insurance limitations. So it’s worth asking whether limited networks save money, and whether they improve or degrade health care.
Read more..

A Great EMR Survey from AAFP

Posted on August 11, 2009 I Written By

Some of the best and most objective information about EMRs comes from the Center for Health IT at the American Academy of Family Practice. Real doctors who have purchased EMRs rate their EMR in 5 different categories: Quality, Value, Usability, Productivity and Support.

This report is ONLY available to members of the AAFP. I think if the AAFP really wanted to do all of us a big favor, they would release this report to anyone who is interested in seeing it. I don’t understand why they are keeping it secret.

It is going to be very difficult for doctors to find a good EMR because there are so many EMRs and so many “bad” EMRs (hard to use, reduce productivity, expensive). Starting with this survey can help doctors start their EMR search on the right foot.

Contact the AAFP and ask them if you can get a copy of their report.

Center for Health IT

Hopefully they will have our great Healthcare System’s best interest at heart. By making this report available to all doctors, they can help us all get “good” EMRs that are usable and high in quality.

If you are a doctor looking for an EMR, start your search with a few EMRs that get good ratings in this survey.