Free EMR Newsletter Want to receive the latest news on EMR, Meaningful Use, ARRA and Healthcare IT sent straight to your email? Join thousands of healthcare pros who subscribe to EMR and EHR for FREE!

Randomized Controlled Trials and Longitudinal Analysis for Health Apps at Twine Health (Part 2 of 2)

Posted on February 18, 2016 I Written By

Andy Oram is an editor at O'Reilly Media, a highly respected book publisher and technology information provider. An employee of the company since 1992, Andy currently specializes in open source, software engineering, and health IT, but his editorial output has ranged from a legal guide covering intellectual property to a graphic novel about teenage hackers. His articles have appeared often on EMR & EHR and other blogs in the health IT space. Andy also writes often for O'Reilly's Radar site (http://oreilly.com/) and other publications on policy issues related to the Internet and on trends affecting technical innovation and its effects on society. Print publications where his work has appeared include The Economist, Communications of the ACM, Copyright World, the Journal of Information Technology & Politics, Vanguardia Dossier, and Internet Law and Business. Conferences where he has presented talks include O'Reilly's Open Source Convention, FISL (Brazil), FOSDEM, and DebConf.

The previous section of this article described the efforts of Dr. John Moore of Twine Health to rigorously demonstrate the effectiveness of a digital health treatment platform. As Moore puts it, Twine Health sought out two of the most effective treatment programs in the country–both Harvard’s diabetes treatment and MGH’s hypertension treatment are much more effective than the standard care found around the country–and then used their most effective programs for the control group of patients. The control group used face-to-face visits, phone calls, and text messages to keep in touch with their coaches and discuss their care plans.

The CollaboRhythm treatment worked markedly better than these exemplary programs. In the diabetes trial, they achieved a 3.2% reduction in diabetic patients’ A1C levels over three months (the control group achieved 2.0%). In the hypertension trial, 100% of patients reached a controlled blood pressure of less than 140/90 and the average reduction in blood pressure was 26mmHg (the control group had an average 16mmHg reduction and fewer than one-third of the patients went down less than 140/90).

What clinical studies can and cannot ensure

I see a few limitations with these clinical studies:

  • The digital program being tested combines several different intervention, as described before: reminders, messaging, virtual interactions, reports, and so on. Experiments show that all these things work together. But one can’t help wondering: what if you took out some time-consuming interaction? Could the platform be just as successful? But testing all the options would lead to a combinatorial explosion of tests.

    It’s important that interventions by coaches started out daily but decreased over the course of the study as the patient became more familiar and comfortable with the behavior called for in the care plans. The decrease in support required from the human coach suggests that the benefits are sustainable, because the subjects are demonstrating they can do more and more for themselves.

  • Outcomes were measured over short time frames. This is a perennial problem with clinical studies, and was noted as a problem in the papers. The researchers will contact subjects in about a year to see whether the benefits found in the studies were sustained. Even one year, although a good period to watch to see whether people bounce back to old behaviors, isn’t long enough to really tell the course of chronic illness. On the other hand, so many other life events intrude over time that it’s unfair to blame one intervention for what happens after a year.

  • Despite the short time frame for outcomes, the studies took years to set up, complete, and publish. This is another property of research practice that adds to its costs and slows down the dissemination of best practices through the medical field. The time frames involved explain why the researchers’ original Media Lab app was used for studies, even though they are now running a company on a totally different platform built on the same principles.

  • These studies also harbor all the well-known questions of external validity faced by all studies on human subjects. What if the populations at these Boston hospitals are unrepresentative of other areas? What if an element of self-selection skewed the results?

Bonnie Feldman, DDS, MBA, who went from dentistry to Wall Street and then to consulting in digital health, comments, “Creating an evidence base requires a delicate balancing act, as you describe, when technology is changing rapidly. Right now, chronic disease, especially autoimmune disease is affecting more young adults than ever before. These patients are in desperate need of new tools to support their self-care efforts. Twine’s early studies validate these important advances.”

Later research at Twine Health

Dr. Moore and his colleagues took stock of the tech landscape since the development of CollaboRhythm–for instance, the iPhone and its imitators had come out in the meantime–and developed a whole new platform on the principles of CollaboRhythm. Twine Health, of which Moore is co-founder and CEO, offers a platform based on these principles to more than 1,000 patients. The company expects to expand this number ten-fold in 2016. In addition to diabetes and hypertension, Twine Health’s platform is used for a wide range of conditions, such as depression, cholesterol control, fitness, and diet.

With a large cohort of patients to draw on, Twine Health can do more of the “big data” analysis that’s popular in the health care field. They don’t sponsor randomized trials like the two studies cited early, but they can compare patients’ progress to what they were doing before using Twine Health, as well as to patients who don’t use Twine Health. Moore says that results are positive and lasting, and that costs for treatment drop one-half to two-thirds.

Clinical studies bring the best scientific methods we know to validating health care apps. They are being found among a small but growing number of app developers. We still don’t know what the relation will be between randomized trials and the longitudinal analysis currently conducted by Twine Health; both seem of vital importance and they will probably complement each other. This is the path that developers have to take if they are to make a difference in health care.

Randomized Controlled Trials and Longitudinal Analysis for Health Apps at Twine Health (Part 1 of 2)

Posted on February 17, 2016 I Written By

Andy Oram is an editor at O'Reilly Media, a highly respected book publisher and technology information provider. An employee of the company since 1992, Andy currently specializes in open source, software engineering, and health IT, but his editorial output has ranged from a legal guide covering intellectual property to a graphic novel about teenage hackers. His articles have appeared often on EMR & EHR and other blogs in the health IT space. Andy also writes often for O'Reilly's Radar site (http://oreilly.com/) and other publications on policy issues related to the Internet and on trends affecting technical innovation and its effects on society. Print publications where his work has appeared include The Economist, Communications of the ACM, Copyright World, the Journal of Information Technology & Politics, Vanguardia Dossier, and Internet Law and Business. Conferences where he has presented talks include O'Reilly's Open Source Convention, FISL (Brazil), FOSDEM, and DebConf.

Walking into a restaurant or a bus is enough to see that any experience delivered through a mobile device is likely to have an enthusiastic uptake. In health care, the challenge is to find experiences that make a positive difference in people’s lives–and proving it.

Of course, science has a time-tested method for demonstrating the truth of a proposition: randomized tests. Reproducibility is a big problem, admittedly, and science has been shaken by the string of errors and outright frauds perpetrated in scientific journals. Still, knowledge advances bit by bit through this process, and the goal of every responsible app developer in the health care space is the blessing offered by a successful test.

Consumer apps versus clinical apps

Most of the 165,000 health apps will probably always be labeled “consumer” apps and be sold without the expense of testing. They occupy the same place in the health care field as the thousands of untested dietary supplements and stem cell injection therapies whose promise is purely anecdotal. Consumer anger over ill-considered claims have led to lawsuits against the Fitbit device manufacturer and Lumosity mental fitness app, leading to questions about the suitability of digital fitness apps for medical care plans.

The impenetrability of consumer apps to objective judgment comes through in a recent study from the Journal of Medical Internet Research (JMIR) that asked mHealth experts to review a number of apps. The authors found very little agreement about what makes a good app, thus suggesting that quality cannot be judged reliably, a theme in another recent article of mine. One might easily anticipate that subjective measures would produce wide variations in judgment. But in fact, many subjective measures produced more agreement (although not really strong agreement) than more “objective” measures such as effectiveness. If I am reading the data right, one of the measures found to be most unreliable was one of the most “objective”: whether an app has been tested for effectiveness.

Designing studies for these apps is an uncertain art. Sometimes a study may show that you don’t know what to measure or aren’t running the study long enough. These possible explanations–gentler than the obvious concern that maybe fitness devices don’t achieve their goals–swirl about the failure of the Scripps “Wired for Health” study.

The Twine Health randomized controlled trials

I won’t talk any more about consumer apps here, though–instead I’ll concentrate on apps meant for serious clinical use. What can randomized testing do for these?

Twine Health and MIT’s Media Lab took the leap into rigorous testing with two leading Boston-area partners in the health care field: a diabetes case study with the Joslin Diabetes Center and a hypertension case study with Massachusetts General Hospital. Both studies compared a digital platform for monitoring and guiding patients with pre-existing tools such as face-to-face visits and email. Both demonstrated better results through the digital platform–but certain built-in limitations of randomized studies leave open questions.

When Dr. John Moore decided to switch fields and concentrate on the user experience, he obtained a PhD at the Media Lab and helped develop an app called CollaboRhythm. He then used it for the two studies described in the papers, while founding and becoming CEO of Twine Health. CollaboRhythm is a pretty comprehensive platform, offering:

  • The ability to store a care plan and make it clear to the user through visualizations.

  • Patient self-tracking to report taking medications and resulting changes in vital signs, such as glycemic levels.

  • Visualizations showing the patient her medication adherence.

  • Reminders when to take medication and do other aspects of treatment, such as checking blood pressure.

  • Inferences about diet and exercise patterns based on reported data, shown to the patient.

  • Support from a human coach through secure text messages and virtual visits using audio, video, and shared screen control.

  • Decision support based on reported vital statistics and behaviors. For instance, when diabetic patients reported following their regimen but their glycemic levels were getting out of control, the app could suggest medication changes to the care team.

The collection of tools is not haphazard, but closely follows the modern model of digital health laid out by the head of Partners Connected Health, Joseph Kvedar, in his book The Internet of Healthy Things (which I reviewed at length). As in Kvedar’s model, the CollaboRhythm interventions rested on convenient digital technologies, put patients’ care into their own hands, and offered positive encouragement backed up by clinical staff.

As an example of the patient empowerment, the app designers deliberately chose not to send the patient an alarm if she forgets her medication. Instead, the patient is expected to learn and adopt responsibility over time by seeing the results of her actions in the visualizations. In exit interviews, some patients expressed appreciation for being asked to take responsibility for their own health.

The papers talk of situated learning, a classic education philosophy that teaches behavior in the context where the person has to practice the behavior, instead of an artificial classroom or lab setting. Technology can bring learning into the home, making it stick.

There is also some complex talk of the relative costs and time commitments between the digital interventions and the traditional ones. One important finding is that app users expressed significantly better feelings about the digital intervention. They became more conscious of their health and appreciated being able to be part of decisions such as changing insulin levels.

So how well does this treatment work? I’ll explore that tomorrow in the next section of this article, along with strengths and weaknesses of the studies.