Reply to Dr. Jacob Reider on NIST Dissects Workflow: Is Anyone Biting?

Posted on March 31, 2014 I Written By

When Carl Bergman isn't rooting for the Washington Nationals or searching for a Steeler bar, he’s Managing Partner of EHRSelector.com, a free service for matching users and EHRs. For the last dozen years, he’s concentrated on EHR consulting and writing. He spent the 80s and 90s as an itinerant project manger doing his small part for the dot com bubble. Prior to that, Bergman served a ten year stretch in the District of Columbia government as a policy and fiscal analyst.

One comment on my latest post, NIST Dissects Workflow: Is Anyone Biting?, deserves a more than casual reply.

Here’s the comment from Jacob Reider (Note: Dr. Reider is ONC’s Acting Principle Deputy National Coordinator and Chief Medical Officer. He has made major contributions to the HIT field and is one of its significant advocates.)

Carl, ONC’s UCD requirement references ISO 9241–11, ISO 13407, ISO 16982, NISTIR 7741, ISO/IEC 62366 and ISO 9241–210 as appropriate UCD processes.

We also require summative testing as defined in NISTIR 7742.

Might “Refuses to incorporate NIST recommendations” be a bit of an overstatement?

We solicited public comment in our proposed rule for 2015 certification and would welcome specific suggestions for how we can/should improve user experience of health IT products for efficiency and safety.

Dr. Reider, thank you for your comment – it certainly falls into the category of you never know who’s reading.

Let’stake a look at your last comment first, “Might ‘Refuses to incorporate NIST recommendations’ be a bit of an overstatement?”

Obviously, I don’t think so, but I am not alone.

I based my comment on ONC’s statement in its rule making that refers to NIST’s usability protocols. It says:

While valid and reliable usability measurements exist, including those specified in NISTIR 7804 “Technical Evaluation, Testing and Validation of the Usability of Electronic Health Records,” (21) we are concerned that it would be inappropriate at this juncture for ONC to seek to measure EHR technology in this way.

Sounds like a rejection to me, however, don’t take my word. Here’s the AMA’s response to this decision. First, they demur and quote ONC:

We disagree with ONC’s assertion in the Version 2014 final rule that, “[w]hile valid and reliable usability measurements exist, including those specified in NISTIR 7804 ‘‘Technical Evaluation, Testing and Validation of the Usability of Electronic Health Records,’’ we expressed that it would be inappropriate for ONC to seek to measure EHR technology in this way.”

It then says:

To the contrary, we believe that it is incumbent upon ONC to include more robust usability criteria in the certification process.  The incentive program has certainly spurred aggressive EHR uptake but has done so through an artificial and non-traditional marketplace.  As a consumer, the physician’s choice of products is limited not only by those EHRs that are certified but also by the constraint that all of these products are driven by federal criteria.  The AMA made several detailed recommendations for improving Version 2014 certification in our Stage 2 comment letter, which were not adopted, but still hold true, and we recommend ONC consider them for the next version.  Testimony of AMA’s Health IT Policy Committee’s Workgroups on Certification/Adoption and Implementation, July 23, 2013, pp. 5-6

I recognize that ONC says that it may consider the protocols in the future. Nevertheless, I think the plain English term rejected fits.

In the first part of his statement, Dr. Reider cites several ISO standards. With the exception of the Summative Testing, all of these have been referred to, but none have been adopted. Reference to a standard is not sufficient for its inclusion under the operation of the federal Administrative Procedure Act, which governs all federal agency rulemaking. In other words, these standards are important, but ONC simply calls them out for attention, nothing more.

I think two factors are at work in ONC’s reluctance to include the NIST usability protocols. The first is that the vendors are adamantly opposed to having them mandated. However, I believe there is a way around that objection.

As I have argued before, ONC could tell vendors that their products will be subject to a TURF based review of their product for compliance and that the results would be made public. That would give users a way to judge a product for suitability to their purpose on a uniform basis. Thus, users looking at the results could determine for themselves whether or not one or more non compliance was important to them, but at least they would have a common way to look at candidate EHRs, something they cannot do now , nor under ONC’s proposed approach.

The other factor is more complex and goes to the nature of ONC’s mission. ONC is both the advocate and the standards maker for HIT. In that, it is similar to the FAA, which is vested with both promoting and regulating US aviation.

It’s well established that the FAA’s dual role is a major problem. It’s hard to be a cheerleader for an industry and make it toe the line.

With the FAA, its dual mandate is exacerbated when the highly respected NTSB investigates an incident and makes recommendations. The FAA, acting as industry friend, often defers NTSB’s authoritative and reasonable recommendations to the public’s determent.

I believe that something similar is going on with ONC. NIST’s relationship to ONC is roughly analogous to that of the NTSB’s to the FAA.

NIST is not an investigative agency, but it is the federal government’s standards and operations authority. It isn’t infallible. However, ONC dismissing NIST’s usability protocols, in one word, inappropriate. It did this without explanation or analysis (at least none that they’ve shared). In my view, that’s really inappropriate.

ONC has a problem. It’s operating the way it was intended, but that’s not what patients and practioners need. To continue the aviation analogy, ONC needs to straighten up and fly right.