Free EMR Newsletter Want to receive the latest news on EMR, Meaningful Use, ARRA and Healthcare IT sent straight to your email? Join thousands of healthcare pros who subscribe to EMR and EHR for FREE!

Correlations and Research Results: Do They Match Up? (Part 2 of 2)

Posted on May 27, 2016 I Written By

Andy Oram is an editor at O'Reilly Media, a highly respected book publisher and technology information provider. An employee of the company since 1992, Andy currently specializes in open source, software engineering, and health IT, but his editorial output has ranged from a legal guide covering intellectual property to a graphic novel about teenage hackers. His articles have appeared often on EMR & EHR and other blogs in the health IT space. Andy also writes often for O'Reilly's Radar site (http://radar.oreilly.com/) and other publications on policy issues related to the Internet and on trends affecting technical innovation and its effects on society. Print publications where his work has appeared include The Economist, Communications of the ACM, Copyright World, the Journal of Information Technology & Politics, Vanguardia Dossier, and Internet Law and Business. Conferences where he has presented talks include O'Reilly's Open Source Convention, FISL (Brazil), FOSDEM, and DebConf.

The previous part of this article described the benefits of big data analysis, along with some of the formal, inherent risks of using it. We’ll go even more into the problems of real-life use now.

More hidden bias

Jeffrey Skopek pointed out that correlations can perpetuate bias as much as they undermine it. Everything in data analysis is affected by bias, ranging from what we choose to examine and what data we collect to who participates, what tests we run, and how we interpret results.

The potential for seemingly objective data analysis to create (or at least perpetuate) discrimination on the basis of race and other criteria was highlighted recently by a Bloomberg article on Amazon Price deliveries. Nobody thinks that any Amazon.com manager anywhere said, “Let’s not deliver Amazon Prime packages to black neighborhoods.” But that was the natural outcome of depending on data about purchases, incomes, or whatever other data was crunched by the company to produce decisions about deliveries. (Amazon.com quickly promised to eliminate the disparity.)

At the conference, Sarah Malanga went over the comparable disparities and harms that big data can cause in health care. Think of all the ways modern researchers interact with potential subjects over mobile devices, and how much data is collected from such devices for data analytics. Such data is used to recruit subjects, to design studies, to check compliance with treatment, and for epidemiology and the new Precision Medicine movement.

In all the same ways that the old, the young, the poor, the rural, ethnic minorities, and women can be left out of commerce, they can be left out of health data as well–with even worse impacts on their lives. Malanga reeled out some statistics:

  • 20% of Americans don’t go on the Internet at all.

  • 57% of African-Americans don’t have Internet connections at home.

  • 70% of Americans over 65 don’t have a smart phone.

Those are just examples of ways that collecting data may miss important populations. Often, those populations are sicker than the people we reach with big data, so they need more help while receiving less.

The use of electronic health records, too, is still limited to certain populations in certain regions. Thus, some patients may take a lot of medications but not have “medication histories” available to research. Ameet Sarpatwari said that the exclusion of some populations from research make post-approval research even more important; there we can find correlations that were missed during trials.

A crucial source of well-balanced health data is the All Payer Claims Databases that 18 states have set up to collect data across the board. But a glitch in employment law, highlighted by Carmel Shachar, releases self-funding employers from sending their health data to the databases. This will most likely take a fix from Congress. Unless they do so, researchers and public health will lack the comprehensive data they need to improve health outcomes, and the 12 states that have started their own APCD projects may abandon them.

Other rectifications cited by Malanga include an NIH requirement for studies funded by it to include women and minorities–a requirement Malanga would like other funders to adopt–and the FCC’s Lifeline program, which helps more low-income people get phone and Internet connections.

A recent article at the popular TechCrunch technology site suggests that the inscrutability of big data analytics is intrinsic to artificial intelligence. We must understand where computers outstrip our intuitive ability to understand correlations.

Correlations and Research Results: Do They Match Up? (Part 1 of 2)

Posted on May 26, 2016 I Written By

Andy Oram is an editor at O'Reilly Media, a highly respected book publisher and technology information provider. An employee of the company since 1992, Andy currently specializes in open source, software engineering, and health IT, but his editorial output has ranged from a legal guide covering intellectual property to a graphic novel about teenage hackers. His articles have appeared often on EMR & EHR and other blogs in the health IT space. Andy also writes often for O'Reilly's Radar site (http://radar.oreilly.com/) and other publications on policy issues related to the Internet and on trends affecting technical innovation and its effects on society. Print publications where his work has appeared include The Economist, Communications of the ACM, Copyright World, the Journal of Information Technology & Politics, Vanguardia Dossier, and Internet Law and Business. Conferences where he has presented talks include O'Reilly's Open Source Convention, FISL (Brazil), FOSDEM, and DebConf.

Eight years ago, a widely discussed issue of WIRED Magazine proclaimed cockily that current methods of scientific inquiry, dating back to Galileo, were becoming obsolete in the age of big data. Running controlled experiments on limited samples just have too many limitations and take too long. Instead, we will take any data we have conveniently at hand–purchasing habits for consumers, cell phone records for everybody, Internet-of-Things data generated in the natural world–and run statistical methods over them to find correlations.

Correlations were spotlighted at the annual conference of the Petrie-Flom Center for Health Law Policy, Biotechnology, and Bioethics at Harvard Law School. Although the speakers expressed a healthy respect for big data techniques, they pinpointed their limitations and affirmed the need for human intelligence in choosing what to research, as well as how to use the results.

Petrie-Flom annual 2016 conference

Petrie-Flom annual 2016 conference

A word from our administration

A new White House report also warns that “it is a mistake to assume [that big data techniques] are objective simply because they are data-driven.” The report highlights the risks of inherent discrimination in the use of big data, including:

  • Incomplete and incorrect data (particularly common in credit rating scores)

  • “Unintentional perpetuation and promotion of historical biases,”

  • Poorly designed algorithmic matches

  • “Personaliziaton and recommendation services that narrow instead of expand user options”

  • Assuming that correlation means causation

The report recommends “bias mitigation” (page 10) and “algorithmic systems accountability” (page 23) to overcome some of these distortions, and refers to a larger FTC report that lays out the legal terrain.

Like the WIRED articles mentioned earlier, this gives us some background for discussions of big data in health care.

Putting the promise of analytical research under the microscope

Conference speaker Tal Zarsky offered both fulsome praise and specific cautions regarding correlations. As the WIRED Magazine issue suggested, modern big data analysis can find new correlations between genetics, disease, cures, and side effects. The analysis can find them much cheaper and faster than randomized clinical trials. This can lead to more cures, and has the other salutory effect of opening a way for small, minimally funded start-up companies to enter health care. Jeffrey Senger even suggested that, if analytics such as those used by IBM Watson are good enough, doing diagnoses without them may constitute malpractice.

W. Nicholson Price, II focused on the danger of the FDA placing too many strict limits on the use of big data for developing drugs and other treatments. Instead of making data analysts back up everything with expensive, time-consuming clinical trials, he suggested that the FDA could set up models for the proper use of analytics and check that tools and practices meet requirements.

One of exciting impacts of correlations is that they bypass our assumptions and can uncover associations we never would have expected. The poster child for this effect is the notorious beer-and-diapers connection found by one retailer. This story has many nuances that tend to get lost in the retelling, but perhaps the most important point to note is that a retailer can depend on a correlation without having to ascertain the cause. In health, we feel much more comfortable knowing the cause of the correlation. Price called this aspect of big data search “black box” medicine.” Saying that something works, without knowing why, raises a whole list of ethical concerns.

A correlation stomach pain and disease can’t tell us whether the stomach pain led to the disease, the disease caused the stomach pain, or both are symptoms of a third underlying condition. Causation can make a big difference in health care. It can warn us to avoid a treatment that works 90% of the time (we’d like to know who the other 10% of patients are before they get a treatment that fails). It can help uncover side effects and other long-term effects–and perhaps valuable off-label uses as well.

Zarsky laid out several reasons why a correlation might be wrong.

  • It may reflect errors in the collected data. Good statisticians control for error through techniques such as discarding outliers, but if the original data contains enough apples, the barrel will go rotten.

  • Even if the correlation is accurate for the collected data, it may not be accurate in the larger population. The correlation could be a fluke, or the statistical sample could be unrepresentative of the larger world.

Zarsky suggests using correlations as a starting point for research, but backing them up by further randomized trials or by mathematical proofs that the correlation is correct.

Isaac Kohane described, from the clinical side, some of the pros and cons of using big data. For instance, data collection helps us see that choosing a gender for intersex patients right after birth produces a huge amount of misery, because the doctor guesses wrong half the time. However, he also cited times when data collection can be confusing for the reasons listed by Zarsky and others.

Senger pointed out that after drugs and medical devices are released into the field, data collected on patients can teach developers more about risks and benefits. But this also runs into the classic risks of big data. For instance, if a patient dies, did the drug or device contribute to death? Or did he just succumb to other causes?

We already have enough to make us puzzle over whether we can use big data at all–but there’s still more, as the next part of this article will describe.

Making Health Data Patient-Friendly

Posted on May 6, 2016 I Written By

Anne Zieger is veteran healthcare consultant and analyst with 20 years of industry experience. Zieger formerly served as editor-in-chief of FierceHealthcare.com and her commentaries have appeared in dozens of international business publications, including Forbes, Business Week and Information Week. She has also contributed content to hundreds of healthcare and health IT organizations, including several Fortune 500 companies. Contact her at @ziegerhealth on Twitter or visit her site at Zieger Healthcare.

Most of the efforts designed to make healthcare processes more transparent hope to make patients better shoppers. The assumption is that better-informed patients make better decisions, and that ultimately, if enough patients have the right data they’ll take steps which improve outcomes and lower the cost of care. And while the evidence for this assumption is sparse, the information may increase patient engagement in their care — and hopefully, their overall health.

That’s all well and good, but I believe too little attention has been paid to another dimension of transparency. To wit, I’d argue that it’s more than time to present patients with clinical data on a real- or near-real-time basis. Yes, shopping for the right doctor is good, but isn’t it even more important for patients to see what results he or she actually gets in their particular medical case?

Patients rarely get a well-developed look at their clinical data. Patient portals may offer access to test and imaging results from today through 10 years ago — my health system does — but offer no tools to put this data in context. If a patient wants to take a good look at their health history, and particularly, how test results correlate with their behavior, they’ll have to map the data out themselves. And that’s never going to work for your average patient.

Of course, there are obstacles to making this happen:

  • Physicians aren’t thrilled with the idea of giving patients broad healthcare data access. In fact, more than one doctor I’ve seen wouldn’t let me see test results until he or she had “approved” them.
  • Even if you set out to create some kind of clinical data dashboard, doing so isn’t trivial, at least if you want to see patients actually use it. Significant user testing would be a must to make this approach a success.
  • To my knowledge, no EMR vendor currently supports a patient dashboard or any other tools to help patients navigate their own data. So to create such an offering, providers would need to wait until their vendor produces such a tool or undertake a custom development project.

To some extent, the healthcare IT industry is already headed in this direction. For example, I’ve encountered mobile apps that attempt to provide some context for the data which they collect. But virtually all healthcare apps focus on just a few key indicators, such as, say calorie intake, exercise or medication compliance. For a patient to get a broad look at their health via app, they would have to bring together several sets of data, which simply isn’t practical.

Instead, why not give patients a broad look at their health status as seen through the rich data contained in an EMR? The final result could include not only data points, but also annotations from doctors as to the significance of trends and access to educational materials in context. That way, the patient could observe, say, the link between blood pressure levels, exercise, weight and med compliance, read comments from both their cardiologist and PCP on what has been working, and jump to research and education on cardiovascular health.

Ultimately, I’d argue, the chief obstacle to creating such an offering isn’t technical. Rather, it’s a cultural issue. Understandably, clinicians are concerned about the disruption such approaches might pose to their routine, as well as their ability to manage cases.

But if we are to make patients healthier, putting the right tools in their hands is absolutely necessary. And hey, after paying so much for EMRs, why not get more value for your money?

P.S. After writing this I discovered a description of a “digital health advisor” which parallels much of what I’m proposing. It’s worth a read!

Why We Store Data in an EHR

Posted on April 27, 2016 I Written By

John Lynn is the Founder of the HealthcareScene.com blog network which currently consists of 10 blogs containing over 8000 articles with John having written over 4000 of the articles himself. These EMR and Healthcare IT related articles have been viewed over 16 million times. John also manages Healthcare IT Central and Healthcare IT Today, the leading career Health IT job board and blog. John is co-founder of InfluentialNetworks.com and Physia.com. John is highly involved in social media, and in addition to his blogs can also be found on Twitter: @techguy and @ehrandhit and LinkedIn.

Shereese Maynard offered this interesting stat about the data inside an EHR and how that data is used.


I then made up this statistic which isn’t validated, but I believe is directionally accurate:


Colin Hung then validated my tweet with his comment:

It’s a tricky world we live in, but the above discussion is not surprising. EHRs were created to make an office more efficient (many have largely failed at that goal) and to help a practice bill at the highest level. In the US, you get paid based on how you document. It’s safe to say that EHR software has made it easier to document at a higher level and get paid more.

Notice that the goals of EHR software weren’t to improve health outcomes or patient care. Those goals might have been desired by many, but it wasn’t the bill of goods sold to the practice. Now we’re trying to back all this EHR data into health outcomes and improved patient care. Is it any wonder it’s a challenge for us to accomplish these goals?

When was the last time a doctor chose an EHR based on how it could improve patient care? I think most were fine purchasing an EHR that they believed wouldn’t hurt patient care. Sadly, I can’t remember ever seeing a section of a RFP that talks about an EHRs ability to improve patient care and clinical outcomes.

No, we store data in an EHR so we can improve our billing. We store data in the EHR to avoid liability. We store data in the EHR because we need appropriate documentation of the visit. Can and should that data be used to improve health outcomes and improve the quality of care provided? Yes, and most are heading that way. Although, it’s trailing since customers never demanded it. Plus, customers don’t really see an improvement in their business by focusing on it (we’ll see if that changes in a value based and high deductible plan world).

In my previous post about medical practice innovation, Dr. Nieder commented on the need for doctors to have “margin in their lives” which allows them to explore innovation. Medical billing documentation is one of the things that sucks the margins out of a doctor’s life. We need to simplify the billing requirements. That would provide doctors more margins to innovate and explore ways EHR and other technology can improve patient care and clinical outcomes.

In response to yesterday’s post about Virtual ACO’s, Randall Oates, MD and Founder of SOAPware (and a few other companies), commented “Additional complexity will not solve healthcare crises in spite of intents.” He, like I, fear that all of this value based reimbursement and ACO movement is just adding more billing complexity as opposed to simplifying things so that doctors have more margin in their lives to improve healthcare. More complexity is not the answer. More room to innovate is the answer.

Another Quality Initiative Ahead of Its Time, From California

Posted on March 21, 2016 I Written By

Andy Oram is an editor at O'Reilly Media, a highly respected book publisher and technology information provider. An employee of the company since 1992, Andy currently specializes in open source, software engineering, and health IT, but his editorial output has ranged from a legal guide covering intellectual property to a graphic novel about teenage hackers. His articles have appeared often on EMR & EHR and other blogs in the health IT space. Andy also writes often for O'Reilly's Radar site (http://radar.oreilly.com/) and other publications on policy issues related to the Internet and on trends affecting technical innovation and its effects on society. Print publications where his work has appeared include The Economist, Communications of the ACM, Copyright World, the Journal of Information Technology & Politics, Vanguardia Dossier, and Internet Law and Business. Conferences where he has presented talks include O'Reilly's Open Source Convention, FISL (Brazil), FOSDEM, and DebConf.

When people go to get health care–or any other activity–we evaluate it for both cost and quality. But health care regulators have to recognize when the ingredients for quality assessment are missing. Otherwise, assessing quality becomes like the drunk who famously looked for his key under the lamplight instead of where the key actually lay. And sadly, as I read a March 4 draft of a California initiative to rate health care insurance, I find that once again the foundations for assessing quality are not in place, and we are chasing lamplights rather than the keys that will unlock better care.

The initiative I’ll discuss in this article comes out of Covered California, one of the Unites States’ 13 state-based marketplaces for health insurance mandated by the ACA. (All the other states use a federal marketplace or some hybrid solution.) As the country’s biggest state–and one known for progressive experiments–California is worth following to see how adept they are at promoting the universally acknowledged Triple Aim of health care.

An overview of health care quality

There’s no dearth of quality measurement efforts in health care–I gave a partial overview in another article. The Covered California draft cites many of these efforts and advises insurers to hook up with them.

Alas–there are problems with all the quality control efforts:

  • Problems with gathering accurate data (and as we’ll see in California’s case, problems with the overhead and bureaucracy created by this gathering)

  • Problems finding measures that reflect actual improvements in outcomes

  • Problems separating things doctors can control (such as follow-up phone calls) with things they can’t (lack of social supports or means of getting treatment)

  • Problems turning insights into programs that improve care.

But the biggest problem in health care quality, I believe, is the intractable variety of patients. How can you say that a particular patient with a particular combination of congestive heart failure, high blood pressure, and diabetes should improve by a certain amount over a certain period of time? How can you guess how many office visits it will take to achieve a change, how many pills, how many hospitalizations? How much should an insurer pay for this treatment?

The more sophisticated payers stratify patients, classifying them by the seriousness of their conditions. And of course, doctors have learned how to game that system. A cleverly designed study by the prestigious National Bureau of Economic Research has uncovered upcoding in the U.S.’s largest quality-based reimbursement program, Medicare Advantage. They demonstrate that doctors are gaming the system in two ways. First, as the use of Medicare Advantage goes up, so do the diagnosed risk levels of patients. Second, patients who transition from private insurance into Medicare Advantage show higher risk not seen in fee-for-service Medicare.

I don’t see any fixes in the Covered California draft to the problem of upcoding. Probably, like most government reimbursement programs, California will slap on some weighting factor that rewards hospitals with higher numbers of poor and underprivileged patients. But this is a crude measure and is often suspected of underestimating the extra costs these patients bring.

A look at the Covered California draft

Covered California certainly understands what the health care field needs, and one has to be impressed with the sheer reach and comprehensiveness of their quality plan. Among other things, they take on:

  • Patient involvement and access to records (how the providers hated that in the federal Meaningful Use requirements!)

  • Racial, ethnic, and gender disparities

  • Electronic record interoperability

  • Preventive health and wellness services

  • Mental and behavioral health

  • Pharmaceutical costs

  • Telemedicine

If there are any pet initiatives of healthcare reformers that didn’t make it into the Covered California plan, I certainly am having trouble finding them.

Being so extensive, the plan suffers from two more burdens. First, the reporting requirements are enormous–I would imagine that insurers and providers would balk simply at that. The requirements are burdensome partly because Covered California doesn’t seem to trust that the major thrust of health reform–paying for outcomes instead of for individual services–will provide an incentive for providers to do other good things. They haven’t forgotten value-based reimbursement (it’s in section 8.02, page 33), but they also insist on detailed reporting about patient engagement, identifying high-risk patients, and reducing overuse through choosing treatments wisely. All those things should happen on their own if insurers and clinicians adopt payments for outcomes.

Second, many of the mandates are vague. It’s not always clear what Covered California is looking for–let alone how the reporting requirements will contribute to positive change. For instance, how will insurers be evaluated in their use of behavioral health, and how will that use be mapped to meeting the goals of the Triple Aim?

Is rescue on the horizon?

According to a news report, the Covered California plan is “drawing heavy fire from medical providers and insurers.” I’m not surprised, given all the weaknesses I found, but I’m disappointed that their objections (as stated in the article) come from the worst possible motivation: they don’t like its call for transparent pricing. Hiding the padding of costs by major hospitals, the cozy payer/provider deals, and the widespread disparities unrelated to quality doesn’t put providers and insurers on the moral high ground.

To me, the true problem is that the health care field has not learned yet how to measure quality and cost effectiveness. There’s hope, though, with the Precision Medicine initiative that recently celebrated its first anniversary. Although analytical firms seem to be focusing on processing genomic information from patients–a high-tech and lucrative undertaking, but one that offers small gains–the real benefit would come if we “correlate activity, physiological measures and environmental exposures with health outcomes.” Those sources of patient variation account for most of the variability in care and in outcomes. Capture that, and quality will be measurable.

Randomized Clinical Trial Validates BaseHealth’s Predictive Analytics

Posted on March 11, 2016 I Written By

Andy Oram is an editor at O'Reilly Media, a highly respected book publisher and technology information provider. An employee of the company since 1992, Andy currently specializes in open source, software engineering, and health IT, but his editorial output has ranged from a legal guide covering intellectual property to a graphic novel about teenage hackers. His articles have appeared often on EMR & EHR and other blogs in the health IT space. Andy also writes often for O'Reilly's Radar site (http://radar.oreilly.com/) and other publications on policy issues related to the Internet and on trends affecting technical innovation and its effects on society. Print publications where his work has appeared include The Economist, Communications of the ACM, Copyright World, the Journal of Information Technology & Politics, Vanguardia Dossier, and Internet Law and Business. Conferences where he has presented talks include O'Reilly's Open Source Convention, FISL (Brazil), FOSDEM, and DebConf.

One of the pressing concerns in health care is the validity of medical and health apps. Because health is a 24-hour-a-day, 365-day-a-year concern, people can theoretically overcome many of their health problems by employing apps that track, measure, report, and encourage them in good behavior. But which ones work? Doctors are understandably reluctant to recommend apps–and insurers to cover them–without validation.

So I’ve been looking at the scattered app developers who have managed to find the time and money for randomized clinical studies. One recent article covered two studies showing the value of a platform that provided the basis for Twine Health. Today I’ll look at BaseHealth, whose service and API I covered last year.

BaseHealth’s risk assessment platform is used by doctors and health coaches to create customized patient health plans. According to CEO Prakash Menon, “Five to seven people out of 1,000, for instance, will develop Type II diabetes each year. Our service allows a provider to focus on those five to seven.” The study that forms the basis for my article describes BaseHealth’s service as “based on an individual’s comprehensive information, including lifestyle, personal information, and family history; genetic information (genotyping or full genome sequencing data), if provided, is included for cumulative assessment.” (p. 1) BaseHealth has trouble integrating EHR data, because transport protocols have been standardized but semantics (what field is used to record each bit of information) have not.

BaseHealth analytics are based on clinical studies whose validity seems secure: they check, for instance, whether the studies are reproducible, whether their sample sizes are adequate, whether the proper statistical techniques were used, etc. To determine each patient’s risk, BaseHealth takes into account factors that the patient can’t control (such as family history) as well as factors that he can. These are all familiar: cholesterol, BMI, smoking, physical activity, etc.

Let’s turn to the study that I read for this article. The basic question the study tries to answer is, “How well does BaseHealth predict that a particular patient might develop a particular health condition?” This is not really feasible for a study, however, because the risk factors leading to diabetes or lung cancer can take decades to develop. So instead, the study’s authors took a shortcut: they asked interviewers to take family histories and other data that the authors called “life information” without telling the interviewers what conditions the patients had. Then they ran the BaseHealth analytics and compared results to the patients actual, current conditions based on their medical histories. They examined the success of risk assignment for three conditions: coronary artery disease (CAD), Type 2 diabetes (T2), and hypertension (HTN).

The patients chosen for the study had high degrees of illness: “43% of the patients had an established diagnosis of CAD, 22% with a diagnosis of T2D and 70% with a diagnosis of HTN.” BaseHealth identified even more patients as being at risk: 74.6% for CAD, 66.7% for T2D, and 77% for HTN. It makes sense that the BaseHealth predictions were greater than actual incidence of the diseases, because BaseHealth is warning of potential future disease as well.

BaseHealth assigned each patient to a percentile chance of getting the disease. For instance, some patients were considered 50-75% likely to develop CAD.

The study used 99 patients, 12 of whom had to be dropped from the study. Although a larger sample would be better, results were still impressive.

The study found a “robust correlation” between BaseHealth’s predictions and the patients’ medical histories. The higher the risk, the more BaseHealth was likely to match the actual medical history. Most important, BaseHealth had no false negatives. If it said a patient’s risk of developing a disease was less than 5%, the patient didn’t have the disease. This is important because you don’t want a filter to leave out any at-risk patients.

I have a number of questions about the article: how patients break down by age, race, and other demographics, for instance. There was also an intervention phase in the study: some patients took successful measures to reduce their risk factors. But the relationship of this intervention to BaseHealth, however, was not explored in the study.

Although not as good as a longitudinal study with a large patient base, the BaseHealth study should be useful to doctors and insurers. It shows that clinical research of apps is feasible. Menon says that a second study is underway with a larger group of subjects, looking at risk of stroke, breast cancer, colorectal cancer, and gout, in addition to the three diseases from the first study. A comparison of the two studies will be interesting.

What is Quality in Health Care? (Part 2 of 2)

Posted on February 10, 2016 I Written By

Andy Oram is an editor at O'Reilly Media, a highly respected book publisher and technology information provider. An employee of the company since 1992, Andy currently specializes in open source, software engineering, and health IT, but his editorial output has ranged from a legal guide covering intellectual property to a graphic novel about teenage hackers. His articles have appeared often on EMR & EHR and other blogs in the health IT space. Andy also writes often for O'Reilly's Radar site (http://radar.oreilly.com/) and other publications on policy issues related to the Internet and on trends affecting technical innovation and its effects on society. Print publications where his work has appeared include The Economist, Communications of the ACM, Copyright World, the Journal of Information Technology & Politics, Vanguardia Dossier, and Internet Law and Business. Conferences where he has presented talks include O'Reilly's Open Source Convention, FISL (Brazil), FOSDEM, and DebConf.

The first part of this article described different approaches to quality–and in fact to different qualities. In this part, I’ll look at the problems with quality measures, and at emerging solutions.

Difficulties of assessing quality

The Methods chapter of a book from the National Center for Biotechnology Information at NIH lays out many of the hurdles that researchers and providers face when judging the quality of clinical care. I’ll summarize a few of the points from the Methods chapter here, but the chapter is well worth a read. The review showed how hard it is to measure accurately many of the things we’d like to know about.

For instance, if variations within a hospital approach (or exceed) the variations between hospitals, there is little benefit to comparing hospitals using that measure. If the same physician gets wildly different scores from year to year, the validity of the measure is suspect. When care is given by multiple doctors and care teams, it is unjust to ascribe the outcome to patient’s principal caretaker. If random variations outweigh everything, the measure is of no use at all. One must also keep in mind practical considerations, such as making sure the process of collecting data would not cost too much.

Many measures apply to a narrow range of patients (for instance, those with pneumonia) and therefore may be skewed for doctors with a relatively small sample of those patients. And a severe winter could elevate mortality from pneumonia, particularly if patients have trouble getting adequate shelter and heat. In general, “For most outcomes, the impacts of random variation and patient factors beyond providers’ control often overwhelm differences attributable to provider quality.” ACMQ quality measures “most likely cannot definitively distinguish poor quality providers from high quality providers, but rather may illuminate potential quality problems for consideration of further investigation.”

The chapter helps explain why many researchers fall back on standard of care. Providers don’t trust outcome-based measures because of random variations and factors beyond their control, including poverty and other demographics. It’s hard even to know what contributed to a death, because in the final months it may not have been feasible to complete the diagnoses of a patient. Thus, doctors prefer “process measures.”

Among the criteria for evaluating quality indicators we see, “Does the indicator capture an aspect of quality that is widely regarded as important?” and more subtly, “subject to provider or public health system control?” The latter criterion heed physicians who say, “We don’t want to be blamed for bad habits or other reasons for noncompliance on the part of our patients, or for environmental factors such as poverty that resist quick fixes.”

The book’s authors are certainly aware of the bias created by gaming the reimbursement system: “systematic biases in documentation and coding practices introduced by awareness that risk-adjustment and reimbursement are related to the presence of particular complications.” The paper points out that diagnosis data is more trustworthy when it is informed by clinical information, not just billing information.

One of the most sensitive–and important–factors in quality assessment is risk adjustment, which means recognizing which patients have extra problems making their care more difficult and their recovery less certain. I have heard elsewhere the claim that CMS doesn’t cut physicians enough slack when they take on more risky patients. Although CMS tries to take poverty into account, hospital administrators suspect that institutions serving low-income populations–and safety-net hospitals in particular–are penalized for doing so.

Risk adjustment criteria are sometimes unpublished. But the most perverse distortion in the quality system comes when hospitals fail to distinguish iatrogenic complications (those introduced by medical intervention, such as infections incurred in the hospital) from the original diseases that the patient brought. CMS recognizes this risk in efforts such as penalties for hospital-acquired conditions. Unless these are flagged correctly, hospitals can end up being rewarded for treating sicker patients–patients that they themselves made sicker.

Distinguishing layers of quality

Theresa Cullen,associate director of the Regenstrief Institute’s Global Health Informatics Program, suggests that we think of quality measures as a stack, like those offered by software platforms:

  1. The bottom of the stack might simply measure whether a patient receive the proper treatment for a diagnosed condition. For instance, is the hemoglobin A1C of each diabetic patient taken regularly?

  2. The next step up is to measure the progress of the first measure. How many patients’ A1C was under control for their stage of the disease?

  3. Next we can move to measuring outcomes: improvements in diabetic status, for instance, or prevention of complications from diabetes

  4. Finally, we can look at the quality of the patient’s life–quality-adjusted life years.

Ultimately, to judge whether a quality measure is valid, one has to compare it to some other quality measure that is supposedly trustworthy. We are still searching for measures that we can rely on to prove quality–and as I have already indicated, there may be too many different “qualities” to find ironclad measures. McCallum offers the optimistic view that the US is just beginning to collect the outcomes data that will hopefully give us robust quality measures, Patient ratings serve as a proxy in the interim.

When organizations claim to use quality measures for accountable care, ratings, or other purposes, they should have their eyes open about the validity of the validation measures, and how applicable they are. Better data collection and analysis over time should allow more refined and useful quality measures. We can celebrate each advance in the choices we have for measures and their meanings.

What is Quality in Health Care? (Part 1 of 2)

Posted on February 9, 2016 I Written By

Andy Oram is an editor at O'Reilly Media, a highly respected book publisher and technology information provider. An employee of the company since 1992, Andy currently specializes in open source, software engineering, and health IT, but his editorial output has ranged from a legal guide covering intellectual property to a graphic novel about teenage hackers. His articles have appeared often on EMR & EHR and other blogs in the health IT space. Andy also writes often for O'Reilly's Radar site (http://radar.oreilly.com/) and other publications on policy issues related to the Internet and on trends affecting technical innovation and its effects on society. Print publications where his work has appeared include The Economist, Communications of the ACM, Copyright World, the Journal of Information Technology & Politics, Vanguardia Dossier, and Internet Law and Business. Conferences where he has presented talks include O'Reilly's Open Source Convention, FISL (Brazil), FOSDEM, and DebConf.

Assessing the quality of medical care is one of the biggest analytical challenges in health today. Every patient expects–and deserves–treatment that meets the highest standards. Moreover, it is hard to find an aspect of health care reform that does not depend on accurate quality measurement. Without a firm basis for assessing quality, how can the government pay Accountable Care Organizations properly? How can consumer choice (the great hope of many reformers) become viable? How can hospitals and larger bodies of researchers become “learning health systems” and implement continuous improvement?

Ensuring quality, of course, is crucial in a fee-for-value system to ensure that physicians don’t cut costs just by withholding necessary care. But a lot of people worry that quality-based reimbursement plans won’t work. As this article will show, determining what works and who is performing well are daunting tasks.

A recent op-ed claims that quality measures are adding unacceptable stress to doctors, that the metrics don’t make a difference to ultimate outcomes, that the variability of individual patients can’t be reflected in the measures, that the assessments don’t take external factors adequately into account, and that the essential element of quality is unmeasurable.

Precision medicine may eventually allow us to tailor treatments to individual patients with unique genetic prints. But in the meantime, we’re guessing a lot of the time we prescribe drugs.

The term quality originally just distinguished things of different kinds, like the Latin word qualis from which it is derived. So there are innumerable different qualities (as in “The quality of mercy is not strained”). It took a while for quality to be seen as a single continuum, as in an NIH book I’ll cite later, which reduces all quality measures to a single number by weighting different measures and combining them. Given the lack of precision in individual measures and the subjective definitions of quality, it may be a fool’s quest to seek a single definition of quality in health care.

Many qualities in play
Some of the ways to measure quality and outcomes include:

  • Longitudinal research: this tracks a group of patients over many years, like the famous Framingham Heart Study that changed medical care. Modern “big data” research carries on this tradition, using data about patients in the field to supplement or validate conventional clinical research. In theory, direct measurement is the most reliable source of data about what works in public health and treatment. Obvious drawbacks include:

    • the time such studies take to produce reliable results

    • the large numbers of participants needed (although technology makes it more feasible to contact and monitor subjects)

    • the risk that unknown variations in populations will produce invalid results

    • inaccuracies introduced by the devices used to gather patient information

  • Standard of care: this is rooted in clinical research, which in turn tries to ensure rigor through double-blind randomized trials. Clinical trials, although the gold standard in research, are hampered by numerous problems of their own, which I have explored in another article. Reproducibility is currently being challenged in health care, as in many other areas of science.

  • Patient ratings: these are among the least meaningful quality indicators, as I recently explored. Patients can offer valuable insights into doctor/patient interactions and other subjective elements of their experience moving through the health care system–insights to which I paid homage in another article–but they can’t dissect the elements of quality care that went into producing their particular outcome, which in any case may require months or years to find out. Although the patient’s experience determines her perception of quality, it does not necessarily reflect the overall quality of care. The most dangerous aspect of patient ratings, as Health IT business consultant Janice McCallum points out, comes when patients’ views of quality depart from best practices. Many patients are looking for a quick fix, whether through pain-killers, antibiotics, or psychotropic medications, when other interventions are called for on the basis of both cost and outcome. So the popularity of ratings among patients just underscores how little we actually know about clinical quality.

Quality measures by organizations such as the American College of Medical Quality (ACMQ) and National Committee for Quality Assurance (NCQA) depend on a combination of the factors just listed. I’ll look more closely at these in the next part of this article.

Meaningful Use Holdover Could Be Good News For Healthcare

Posted on January 25, 2016 I Written By

Anne Zieger is veteran healthcare consultant and analyst with 20 years of industry experience. Zieger formerly served as editor-in-chief of FierceHealthcare.com and her commentaries have appeared in dozens of international business publications, including Forbes, Business Week and Information Week. She has also contributed content to hundreds of healthcare and health IT organizations, including several Fortune 500 companies. Contact her at @ziegerhealth on Twitter or visit her site at Zieger Healthcare.

I know all of us are a flutter about the pending regulatory changes which will phase out Meaningful Use as we know it. And yes, without a doubt, the changes underway will have an impact that extends well beyond the HIT world. But while big shifts are underway in federal incentives programs, it’s worth noting that it could be a while before these changes actually fall into place.

As readers may know, the healthcare industry will be transitioning to working under value-based payment under the Medicare Access and CHIP Reauthorization Act, which passed last year. But as ONC’s Karen DeSalvo noted last week, the transition could take a while In fact, proposed draft regulations for MACRA rollout will be released this spring for public comment. When you toss in the time needed for those comments to be submitted, and for the feds to digest those comments and respond, my guess is that MACRA regs won’t go live until late this year at the earliest.

The truth is, this is probably a very good thing. While I don’t have to tell you folks that everyone and their cousin has a Meaningful Use gripe, the truth is that the industry has largely adapted to the MU mindset. Maybe Meaningful Use Stage 3 wouldn’t have provided a lot of jollies, but on the whole, arguably, most providers have come to terms with the level of process documentation required — and have bought their big-bucks EMRs, committing once and for all to the use of digital health records.

Value-based payment, on the other hand, is another thing entirely. From what I’ve read and researched to date, few health organizations have really sunk their teeth into VBP, though many are dabbling. When MACRA regs finally combine the Physician Quality Reporting System, the Value-based Payment Modifier and the Medicare EHR incentive program into a single entity, providers will face some serious new challenges.

Sure, on the surface the idea of providers being paid for the quality and efficiency they deliver sounds good. Rather than using a strict set of performance measures as proxies for quality, the new MACRA-based programs will focus on a mix of quality, resource use and clinical practice use measures, along with measuring meaningful use of certified EHR technology. Under these terms, health systems could conceivably enjoy both greater freedom and better payoffs.

However, given health systems’ experiences to date, particularly with ACOs, I’m skeptical that they’ll be able to pick up the ball and run with the new incentives off the bat. For example, health systems have been abandoning CMS’s value-based Pioneer ACO model at a brisk clip, after finding it financially unworkable. One recent case comes from Dartmouth-Hitchcock Medical Center, which dropped out of the program in October of last year after losing more than $3 million over the previous two years.

I’m not suggesting that health systems can afford to ignore VBP models, or that sticking to MU incentives as previously structured would make sense. But if the process of implementing MACRA gives the industry a chance to do more preparing for value-based payment, it’s probably a good thing.

Significant Articles in the Health IT Community in 2015

Posted on December 15, 2015 I Written By

Andy Oram is an editor at O'Reilly Media, a highly respected book publisher and technology information provider. An employee of the company since 1992, Andy currently specializes in open source, software engineering, and health IT, but his editorial output has ranged from a legal guide covering intellectual property to a graphic novel about teenage hackers. His articles have appeared often on EMR & EHR and other blogs in the health IT space. Andy also writes often for O'Reilly's Radar site (http://radar.oreilly.com/) and other publications on policy issues related to the Internet and on trends affecting technical innovation and its effects on society. Print publications where his work has appeared include The Economist, Communications of the ACM, Copyright World, the Journal of Information Technology & Politics, Vanguardia Dossier, and Internet Law and Business. Conferences where he has presented talks include O'Reilly's Open Source Convention, FISL (Brazil), FOSDEM, and DebConf.

Have you kept current with changes in device connectivity, Meaningful Use, analytics in healthcare, and other health IT topics during 2015? Here are some of the articles I find significant that came out over the past year.

The year kicked off with an ominous poll about Stage 2 Meaningful Use, with implications that came to a head later with the release of Stage 3 requirements. Out of 1800 physicians polled around the beginning of the year, more than half were throwing in the towel–they were not even going to try to qualify for Stage 2 payments. Negotiations over Stage 3 of Meaningful Use were intense and fierce. A January 2015 letter from medical associations to ONC asked for more certainty around testing and certification, and mentioned the need for better data exchange (which the health field likes to call interoperability) in the C-CDA, the most popular document exchange format.

A number of expert panels asked ONC to cut back on some requirements, including public health measures and patient view-download-transmit. One major industry group asked for a delay of Stage 3 till 2019, essentially tolerating a lack of communication among EHRs. The final rules, absurdly described as a simplification, backed down on nothing from patient data access to quality measure reporting. Beth Israel CIO John Halamka–who has shuttled back and forth between his Massachusetts home and Washington, DC to advise ONC on how to achieve health IT reform–took aim at Meaningful Use and several other federal initiatives.

Another harbinger of emerging issues in health IT came in January with a speech about privacy risks in connected devices by the head of the Federal Trade Commission (not an organization we hear from often in the health IT space). The FTC is concerned about the security of recent trends in what industry analysts like to call the Internet of Things, and medical devices rank high in these risks. The speech was a lead-up to a major report issued by the FTC on protecting devices in the Internet of Things. Articles in WIRED and Bloomberg described serious security flaws. In August, John Halamka wrote own warning about medical devices, which have not yet started taking security really seriously. Smart watches are just as vulnerable as other devices.

Because so much medical innovation is happening in fast-moving software, and low-budget developers are hankering for quick and cheap ways to release their applications, in February, the FDA started to chip away at its bureaucratic gamut by releasing guidelines releasing developers from FDA regulation medical apps without impacts on treatment and apps used just to transfer data or do similarly non-transformative operations. They also released a rule for unique IDs on medical devices, a long-overdue measure that helps hospitals and researchers integrate devices into monitoring systems. Without clear and unambiguous IDs, one cannot trace which safety problems are associated with which devices. Other forms of automation may also now become possible. In September, the FDA announced a public advisory committee on devices.

Another FDA decision with a potential long-range impact was allowing 23andMe to market its genetic testing to consumers.

The Department of Health and Human Services has taken on exceedingly ambitious goals during 2015. In addition to the daunting Stage 3 of Meaningful Use, they announced a substantial increase in the use of fee-for-value, although they would still leave half of providers on the old system of doling out individual payments for individual procedures. In December, National Coordinator Karen DeSalvo announced that Health Information Exchanges (which limit themselves only to a small geographic area, or sometimes one state) would be able to exchange data throughout the country within one year. Observers immediately pointed out that the state of interoperability is not ready for this transition (and they could well have added the need for better analytics as well). HHS’s five-year plan includes the use of patient-generated and non-clinical data.

The poor state of interoperability was highlighted in an article about fees charged by EHR vendors just for setting up a connection and for each data transfer.

In the perennial search for why doctors are not exchanging patient information, attention has turned to rumors of deliberate information blocking. It’s a difficult accusation to pin down. Is information blocked by health care providers or by vendors? Does charging a fee, refusing to support a particular form of information exchange, or using a unique data format constitute information blocking? On the positive side, unnecessary imaging procedures can be reduced through information exchange.

Accountable Care Organizations are also having trouble, both because they are information-poor and because the CMS version of fee-for-value is too timid, along with other financial blows and perhaps an inability to retain patients. An August article analyzed the positives and negatives in a CMS announcement. On a large scale, fee-for-value may work. But a key component of improvement in chronic conditions is behavioral health which EHRs are also unsuited for.

Pricing and consumer choice have become a major battleground in the current health insurance business. The steep rise in health insurance deductibles and copays has been justified (somewhat retroactively) by claiming that patients should have more responsibility to control health care costs. But the reality of health care shopping points in the other direction. A report card on state price transparency laws found the situation “bleak.” Another article shows that efforts to list prices are hampered by interoperability and other problems. One personal account of a billing disaster shows the state of price transparency today, and may be dangerous to read because it could trigger traumatic memories of your own interactions with health providers and insurers. Narrow and confusing insurance networks as well as fragmented delivery of services hamper doctor shopping. You may go to a doctor who your insurance plan assures you is in their network, only to be charged outrageous out-of-network costs. Tools are often out of date overly simplistic.

In regard to the quality ratings that are supposed to allow intelligent choices to patients, A study found that four hospital rating sites have very different ratings for the same hospitals. The criteria used to rate them is inconsistent. Quality measures provided by government databases are marred by incorrect data. The American Medical Association, always disturbed by public ratings of doctors for obvious reasons, recently complained of incorrect numbers from the Centers for Medicare & Medicaid Services. In July, the ProPublica site offered a search service called the Surgeon Scorecard. One article summarized the many positive and negative reactions. The New England Journal of Medicine has called ratings of surgeons unreliable.

2015 was the year of the intensely watched Department of Defense upgrade to its health care system. One long article offered an in-depth examination of DoD options and their implications for the evolution of health care. Another article promoted the advantages of open-source VistA, an argument that was not persuasive enough for the DoD. Still, openness was one of the criteria sought by the DoD.

The remote delivery of information, monitoring, and treatment (which goes by the quaint term “telemedicine”) has been the subject of much discussion. Those concerned with this development can follow the links in a summary article to see the various positions of major industry players. One advocate of patient empowerment interviewed doctors to find that, contrary to common fears, they can offer email access to patients without becoming overwhelmed. In fact, they think it leads to better outcomes. (However, it still isn’t reimbursed.)

Laws permitting reimbursement for telemedicine continued to spread among the states. But a major battle shaped up around a ruling in Texas that doctors have a pre-existing face-to-face meeting with any patient whom they want to treat remotely. The spread of telemedicine depends also on reform of state licensing laws to permit practices across state lines.

Much wailing and tears welled up over the required transition from ICD-9 to ICD-10. The AMA, with some good arguments, suggested just waiting for ICD-11. But the transition cost much less than anticipated, making ICD-10 much less of a hot button, although it may be harmful to diagnosis.

Formal studies of EHR strengths and weaknesses are rare, so I’ll mention this survey finding that EHRs aid with public health but are ungainly for the sophisticated uses required for long-term, accountable patient care. Meanwhile, half of hospitals surveyed are unhappy with their EHRs’ usability and functionality and doctors are increasingly frustrated with EHRs. Nurses complained about technologies’s time demands and the eternal lack of interoperability. A HIMSS survey turned up somewhat more postive feelings.

EHRs are also expensive enough to hurt hospital balance sheets and force them to forgo other important expenditures.

Electronic health records also took a hit from ONC’s Sentinel Events program. To err, it seems, is not only human but now computer-aided. A Sentinel Event Alert indicated that more errors in health IT products should be reported, claiming that many go unreported because patient harm was avoided. The FDA started checking self-reported problems on PatientsLikeMe for adverse drug events.

The ONC reported gains in patient ability to view, download, and transmit their health information online, but found patient portals still limited. Although one article praised patient portals by Epic, Allscripts, and NextGen, an overview of studies found that patient portals are disappointing, partly because elderly patients have trouble with them. A literature review highlighted where patient portals fall short. In contrast, giving patients full access to doctors’ notes increases compliance and reduces errors. HHS’s Office of Civil Rights released rules underlining patients’ rights to access their data.

While we’re wallowing in downers, review a study questioning the value of patient-centered medical homes.

Reuters published a warning about employee wellness programs, which are nowhere near as fair or accurate as they claim to be. They are turning into just another expression of unequal power between employer and employee, with tendencies to punish sick people.

An interesting article questioned the industry narrative about the medical device tax in the Affordable Care Act, saying that the industry is expanding robustly in the face of the tax. However, this tax is still a hot political issue.

Does anyone remember that Republican congressmen published an alternative health care reform plan to replace the ACA? An analysis finds both good and bad points in its approach to mandates, malpractice, and insurance coverage.

Early reports on use of Apple’s open ResearchKit suggested problems with selection bias and diversity.

An in-depth look at the use of devices to enhance mental activity examined where they might be useful or harmful.

A major genetic data mining effort by pharma companies and Britain’s National Health Service was announced. The FDA announced a site called precisionFDA for sharing resources related to genetic testing. A recent site invites people to upload health and fitness data to support research.

As data becomes more liquid and is collected by more entities, patient privacy suffers. An analysis of web sites turned up shocking practices in , even at supposedly reputable sites like WebMD. Lax security in health care networks was addressed in a Forbes article.

Of minor interest to health IT workers, but eagerly awaited by doctors, was Congress’s “doc fix” to Medicare’s sustainable growth rate formula. The bill did contain additional clauses that were called significant by a number of observers, including former National Coordinator Farzad Mostashari no less, for opening up new initiatives in interoperability, telehealth, patient monitoring, and especially fee-for-value.

Connected health took a step forward when CMS issued reimbursement guidelines for patient monitoring in the community.

A wonky but important dispute concerned whether self-insured employers should be required to report public health measures, because public health by definition needs to draw information from as wide a population as possible.

Data breaches always make lurid news, sometimes under surprising circumstances, and not always caused by health care providers. The 2015 security news was dominated by a massive breach at the Anthem health insurer.

Along with great fanfare in Scientific American for “precision medicine,” another Scientific American article covered its privacy risks.

A blog posting promoted early and intensive interactions with end users during app design.

A study found that HIT implementations hamper clinicians, but could not identify the reasons.

Natural language processing was praised for its potential for simplifying data entry, and to discover useful side effects and treatment issues.

CVS’s refusal to stock tobacco products was called “a major sea-change for public health” and part of a general trend of pharmacies toward whole care of the patient.

A long interview with FHIR leader Grahame Grieve described the progress of the project, and its the need for clinicians to take data exchange seriously. A quiet milestone was reached in October with a a production version from Cerner.

Given the frequent invocation of Uber (even more than the Cheesecake Factory) as a model for health IT innovation, it’s worth seeing the reasons that model is inapplicable.

A number of hot new sensors and devices were announced, including a tiny sensor from Intel, a device from Google to measure blood sugar and another for multiple vital signs, enhancements to Microsoft products, a temperature monitor for babies, a headset for detecting epilepsy, cheap cameras from New Zealand and MIT for doing retinal scans, a smart phone app for recognizing respiratory illnesses, a smart-phone connected device for detecting brain injuries and one for detecting cancer, a sleep-tracking ring, bed sensors, ultrasound-guided needle placement, a device for detecting pneumonia, and a pill that can track heartbeats.

The medical field isn’t making extensive use yet of data collection and analysis–or uses analytics for financial gain rather than patient care–the potential is demonstrated by many isolated success stories, including one from Johns Hopkins study using 25 patient measures to study sepsis and another from an Ontario hospital. In an intriguing peek at our possible future, IBM Watson has started to integrate patient data with its base of clinical research studies.

Frustrated enough with 2015? To end on an upbeat note, envision a future made bright by predictive analytics.