Randomized Controlled Trials and Longitudinal Analysis for Health Apps at Twine Health (Part 1 of 2)

Posted on February 17, 2016 I Written By

Andy Oram is an editor at O'Reilly Media, a highly respected book publisher and technology information provider. An employee of the company since 1992, Andy currently specializes in open source, software engineering, and health IT, but his editorial output has ranged from a legal guide covering intellectual property to a graphic novel about teenage hackers. His articles have appeared often on EMR & EHR and other blogs in the health IT space. Andy also writes often for O'Reilly's Radar site (http://oreilly.com/) and other publications on policy issues related to the Internet and on trends affecting technical innovation and its effects on society. Print publications where his work has appeared include The Economist, Communications of the ACM, Copyright World, the Journal of Information Technology & Politics, Vanguardia Dossier, and Internet Law and Business. Conferences where he has presented talks include O'Reilly's Open Source Convention, FISL (Brazil), FOSDEM, and DebConf.

Walking into a restaurant or a bus is enough to see that any experience delivered through a mobile device is likely to have an enthusiastic uptake. In health care, the challenge is to find experiences that make a positive difference in people’s lives–and proving it.

Of course, science has a time-tested method for demonstrating the truth of a proposition: randomized tests. Reproducibility is a big problem, admittedly, and science has been shaken by the string of errors and outright frauds perpetrated in scientific journals. Still, knowledge advances bit by bit through this process, and the goal of every responsible app developer in the health care space is the blessing offered by a successful test.

Consumer apps versus clinical apps

Most of the 165,000 health apps will probably always be labeled “consumer” apps and be sold without the expense of testing. They occupy the same place in the health care field as the thousands of untested dietary supplements and stem cell injection therapies whose promise is purely anecdotal. Consumer anger over ill-considered claims have led to lawsuits against the Fitbit device manufacturer and Lumosity mental fitness app, leading to questions about the suitability of digital fitness apps for medical care plans.

The impenetrability of consumer apps to objective judgment comes through in a recent study from the Journal of Medical Internet Research (JMIR) that asked mHealth experts to review a number of apps. The authors found very little agreement about what makes a good app, thus suggesting that quality cannot be judged reliably, a theme in another recent article of mine. One might easily anticipate that subjective measures would produce wide variations in judgment. But in fact, many subjective measures produced more agreement (although not really strong agreement) than more “objective” measures such as effectiveness. If I am reading the data right, one of the measures found to be most unreliable was one of the most “objective”: whether an app has been tested for effectiveness.

Designing studies for these apps is an uncertain art. Sometimes a study may show that you don’t know what to measure or aren’t running the study long enough. These possible explanations–gentler than the obvious concern that maybe fitness devices don’t achieve their goals–swirl about the failure of the Scripps “Wired for Health” study.

The Twine Health randomized controlled trials

I won’t talk any more about consumer apps here, though–instead I’ll concentrate on apps meant for serious clinical use. What can randomized testing do for these?

Twine Health and MIT’s Media Lab took the leap into rigorous testing with two leading Boston-area partners in the health care field: a diabetes case study with the Joslin Diabetes Center and a hypertension case study with Massachusetts General Hospital. Both studies compared a digital platform for monitoring and guiding patients with pre-existing tools such as face-to-face visits and email. Both demonstrated better results through the digital platform–but certain built-in limitations of randomized studies leave open questions.

When Dr. John Moore decided to switch fields and concentrate on the user experience, he obtained a PhD at the Media Lab and helped develop an app called CollaboRhythm. He then used it for the two studies described in the papers, while founding and becoming CEO of Twine Health. CollaboRhythm is a pretty comprehensive platform, offering:

  • The ability to store a care plan and make it clear to the user through visualizations.

  • Patient self-tracking to report taking medications and resulting changes in vital signs, such as glycemic levels.

  • Visualizations showing the patient her medication adherence.

  • Reminders when to take medication and do other aspects of treatment, such as checking blood pressure.

  • Inferences about diet and exercise patterns based on reported data, shown to the patient.

  • Support from a human coach through secure text messages and virtual visits using audio, video, and shared screen control.

  • Decision support based on reported vital statistics and behaviors. For instance, when diabetic patients reported following their regimen but their glycemic levels were getting out of control, the app could suggest medication changes to the care team.

The collection of tools is not haphazard, but closely follows the modern model of digital health laid out by the head of Partners Connected Health, Joseph Kvedar, in his book The Internet of Healthy Things (which I reviewed at length). As in Kvedar’s model, the CollaboRhythm interventions rested on convenient digital technologies, put patients’ care into their own hands, and offered positive encouragement backed up by clinical staff.

As an example of the patient empowerment, the app designers deliberately chose not to send the patient an alarm if she forgets her medication. Instead, the patient is expected to learn and adopt responsibility over time by seeing the results of her actions in the visualizations. In exit interviews, some patients expressed appreciation for being asked to take responsibility for their own health.

The papers talk of situated learning, a classic education philosophy that teaches behavior in the context where the person has to practice the behavior, instead of an artificial classroom or lab setting. Technology can bring learning into the home, making it stick.

There is also some complex talk of the relative costs and time commitments between the digital interventions and the traditional ones. One important finding is that app users expressed significantly better feelings about the digital intervention. They became more conscious of their health and appreciated being able to be part of decisions such as changing insulin levels.

So how well does this treatment work? I’ll explore that tomorrow in the next section of this article, along with strengths and weaknesses of the studies.