Watson, M.D.: How IBM And Cleveland Clinic Are Working To Reboot Health Care With Supercomputing

IBM is working with Cleveland Clinic to bring Watson, the supercomputer “Jeopardy!” winner, to health care to change the way clinical decisions are made.

IBM’s Watson — the celebrated computer system that beat a pair of trivia champions at “Jeopardy!” — is now in medical school, learning how to become a doctor so it can help solve some of the cognitive problems in medicine, representatives from IBM and Cleveland Clinic revealed last month.

Eric Brown, a senior manager with IBM research, presented the challenge the computer system is attempting to solve this way: “Given some rich, natural language question, [can you] have a computer system that can analyze a broad domain of knowledge, and can give precise answers with accurate confidences with … justifications over a fast response time?” Brown spoke during an Oct. 15 panel at the Cleveland Clinic’s recent annual Innovation Summit.

The value of such analysis is particularly acute in today’s health system, Martin Harris, Cleveland Clinic’s chief information officer, said while speaking on a panel Oct. 16, noting that without analytical processes providing “early triage,” the system would be overwhelmed by the torrent of patients entering “300 million weights four times a day.”

“We see … organizations [that] are completely overwhelmed by [data], and feel intimidated and at risk by that,” Sean Hogan, IBM’s VP of health care delivery, concurred.

The IBM-Cleveland Clinical Watson collaboration is attacking smaller, more bite-sized situations, starting with enhancing electronic health records and a basic diagnosis system, before driving at its grander ambitions.

Other summit participants noted that medical knowledge is expanding rapidly with respect to clinical trials, with approximately 20,000 studies conducted yearly.

“Physicians of my generation tried to memorize the entire body of knowledge,” Joe Cunningham, director of venture capital fund Santé Ventures, said Oct. 16. “The training of physicians [is] now more like lawyers’ — [it’s] training how to access information rather than just memorizing it all.”

Figuring out how to analyze the data is a top priority, and Cleveland Clinic physicians believe that a supercomputer like Watson can play an important role. “We are creating this pile of data in which there are questions we don’t know to ask. That is the next big function of what we’re calling analytics today, and it’s the idea that a computer can actually suggest the question,” Harris said, rather than simply answering direct queries from a person.

Watson is a cognitive computer system that combines natural language processing, machine learning, and hypothesis generation and evaluation to analyze information, according to IBM. The system gained fame in 2011 when it appeared and succeed on “Jeopardy!” and since then the company has been building it as a tool for transforming multiple industries, with health care as a primary target.

But while participants at the summit characterized the potential applications of Watson as a “game changer,” the IBM-Cleveland Clinic collaboration is attacking smaller, more bite-sized challenges first: It’s mostly focused on medical education, and it’s starting with enhancing electronic health records and a basic diagnosis system, before driving at its grander ambitions.

EMRA: Toward Decision Support

The two tools IBM is currently testing with Cleveland Clinic are called EMRA and WatsonPaths.

The first tool — which stands for “Electronic Medical Record Analysis” (or “Application”; Brown joked that IBM hadn’t decided which and hoped that the simple acronym would catch on) — works by summarizing available information in a one-page document for doctors.

It’s differentiated from the average electronic health record in two ways. First, it’s able to parse “unstructured data” — that is, textual data or natural language, which is more ambiguous than numerical data. The summary means that doctors have to do less digging through the patient’s record, a common frustration with current EHR systems. Second, IBM and Cleveland Clinic are trying to enhance the data to make it more helpful for clinical decision-making.

Eric Jelovsek, a doctor with the Cleveland Clinic’s Education Institute, explained in an interview, “It’s not just, ‘Does it make life easy — but is Watson able to identify elements in the record that either would’ve taken the provider a long time to go through and eventually find out, or does it highlight items that the provider didn’t look at?’” Ultimately, the aim is for EMRA to directly suggest courses of action to a doctor.

Currently, EMRA is programmed to automatically create a “problem list” based on the unstructured data in a patient’s record. IBM’s Brown, in his presentation, brought up a demo patient record. Based on clinical notes and medications, the EMRA was able to determine the “patient” was a diabetic; with a click of a button, the doctor could easily look at lab results, glucose levels and the like.

Brown focused on the ease of the problem list feature — “Typically [problem lists are] manually updated, but might become out-of-date,” he said, noting that automation might improve the process significantly. A future version of the program might take the process one step further, and automatically suggest next steps to the doctor based on the EMRA’s read of the patient record.

There are also efforts to get EMRA to populate graphs of patient results with “sentinel events,” explained Neil Mehta — an associate director of medicine technology at the Clinic’s affiliated medical school, the Lerner College of Medicine.

These are graphical annotations that “might explain why a value is higher than it should be,” Mehta said. “Say, a radiological test with a contrast dye” might influence some values, he suggested.

WatsonPaths: Making A Diagnosis, In Style

While the EMRA mostly seeks to summarize rather than analyze information, WatsonPaths seeks to actually present diagnoses.

It’s “trying to find the connection between input factors and diagnoses,” Brown said. Starting with inputs from the patient’s electronic medical record, it tries to suggest diagnoses, with varying levels of confidence – with the height of the bars on the graph showing the level of confidence the machine has in each potential diagnosis.

The demonstrations in the conference showed off a slick user interface, more polished than the typical EMR system. “That’s on purpose,” Mike Barborak — the manager of IBM’s natural language engineering and frameworks project for Watson — said.The firm has created a design lab to help aid in configuring the interface and works closely with Cleveland Clinic doctors to ensure a friendly user experience.

“That’s been a tremendous boon to us, to be able to do things that are a little more refined, a little more polished, a little more usable,” he said.

The focus on design helped create EMRA’s one-page experience, “which is [a] dramatically different kind of view than what you would’ve normally gotten in a traditional [EHR] system.”

Jelovsek, with the Cleveland Clinic’s Education Institute, praised the focus on design, saying, “You have to make sure the technology is simple to understand for the average person who’s not geeky like some of us on the team [at Cleveland Clinic], but is actually working in a clinical situation.” That way, providers “can get access to the data very quickly and make decisions and move forward,” he noted.

The IBM team will focus on dividing information more cleanly in the future. “There needs to be more work in dividing information up amongst specialties, for example. There needs to be more work in grouping the information that you see, depending on which dimension the doctor’s interested in looking at,” Barborak continued; he also noted the challenge of displaying the patient’s history over time.

What also interests Barborak is making more analysis based on patient information: “It’s one thing to take the [EHR] and summarize it, but then we can really begin to use our WatsonPaths technology to ask questions about it and do inferences and see if we find evidence in the [EHR] that supports what we’re looking at, which would allow us to bring more interesting and deeply-hidden information to the doctor,” he said.

Testing At Cleveland Clinic

The tools are currently being tested and refined at Cleveland Clinic, mostly in an educational setting.

Medical students at the hospital system’s Lerner College of Medicine are using WatsonPaths to inform their own decision-making process and to teach them how to think like doctors. Since the tool explains its own reasoning, drawing on evidence from a patient’s record and the literature, students are able to correct Watson’s errors and develop the software as well as deepen their education.

“By having the student work in collaboration with Watson, the student can be giving information back that will be useful in improving this system,” IBM’s Barborak said, noting how students can rate Watson’s use of evidence and information in coming to a conclusion, powering machine learning.

One type of error Watson makes is that it suggests irrelevant factors. “The technology currently will identify very relevant factors,” Jelovsek said. “But it also identifies some factors that aren’t relevant. We have to find out why did the system identify those as relevant when they’re not?” That means drilling down into the algorithms of the system.

“My analogy is … the old maps app that Apple released that [had] incorrect routes. If you put that out there in a medical condition, the danger here is much greater than going the wrong way” – Eric Jelovsek, Cleveland Clinic

Separate from the medical school’s efforts, the Cleveland Clinic hospital is currently conducting an early-stage simulation of Watson’s tools that is slated to conclude in November. In the simulation, doctors will see a given patient as they normally would; then, afterwards, with a student, walk through the visit with the aid of Watson and ask themselves, “What would we have done differently if we had this tool?”

That testing is crucial, because offering an unrefined system to the public would be dangerous. “You simply cannot provide technology that’s inaccurate,” Jelovsek said.

“My analogy is … the old maps app that Apple released that [had] incorrect routes. If you put that out there in a medical condition, the danger here is much greater than going the wrong way,” he said.

“We have to make sure that the accuracy is at least as good as mid-level practitioners or even first-line type of practitioners,” he concluded, noting that the critical test will be when the system is moved from a more controlled situation to simulated scripts and then finally to a real-life situation, “where the pace and the action are less controlled [and] there’s more variability.”

The Clinic is also uploading 6 million patient records into Watson to help with its machine-learning process. Barborak said that the data should be a “treasure trove” for the system, “just because we see more examples of how language is used, we find more ways of being able to match the way people describe things, paraphrasing and such.”

“In the machine-learning process, we need to see lots of data in order to train our models appropriately,” he said. “So having 6 million patient records allows us to do this in a very deep and broad … way, so that we can actually begin finding those corner cases or those less-common instances where we want to find a relation between some drug and some problem, [which] we wouldn’t necessarily see in a smaller dataset.”

Broadening The Workforce, Minding The Pitfalls

While porting the system broadly into the clinical space is still a ways away — Barborak said the timeline is the “next few years” — Cleveland Clinic officials couldn’t help but speculate at the summit last month about the ways it would transform doctors’ work.

For example, Jelovsek said, the system can come up with counterintuitive but valuable insights. “That’s one of the coolest things about the technology,” he proclaimed. “Experienced clinicians are in an automated pathway,” he explained. “Our clinical experience will tell us this is the most likely diagnosis. So we immediately jump there. I can imagine in a setting where … I’m not 100 percent sure about an answer … This is going to be very relevant because it’s going to say, ‘Did you consider these things?’”

Moreover, Harris argued, it will broaden the range of tasks performed by clinicians. “We think this will shift the workforce,” he said.

“Physicians will shift to [being] knowledge workers,” he continued. “What we’re going to be able to do with these tools is we will leverage the alliance of health professionals, so they can practice at the top of their license.”

Referencing predictions in the digital health business community that 80 percent of doctors’ tasks can be replaced by machines, Harris concluded, “[It’s] not going to replace 80 percent of physicians, but 80 percent of physician tasks. There will be new tasks that are invented” (Also see "Machines Can Replace 80% Of What Doctors Do – VC Vinod Khosla" - Medtech Insight, 10 Dec, 2012.).

Jim Young, the Clinic’s chair of endocrinology and metabolism, speculated that Watson will help “amplify physicians’ presence,” particularly in light of the doctor shortage in many areas of the country.

“Even today, ICUs are being manned remotely. You could imagine an instrument like Watson being deployed remotely,” he said. Harris offered similar thoughts, saying that Watson could perhaps adjust care in real-time, in response to data from remote patient monitoring.

But the cognitive help Watson could provide could lead providers astray, Jelovsek warned. It’s possible that physicians or other health care workers could be excessively reliant upon the system and assume that its answer is the right one.

“People do some crazy things,” Jelovsek said. “So I can actually see cases where physicians may use less judgment than they should, and they go forward with a diagnosis because either a device or a technology told them, ‘That’s the right thing to do.’ We see this all the time now with other medical devices.”

“Where I think it could be more problematic is in people who are less experienced, [or] when you get into mid-level or first-responding type levels of providers,” he continued.

“For example, if you were to use Watson to judge a call-in type of service where … a child’s complaining of an aching ear, they’re just going to follow the algorithms. And if the algorithms are coming from an incorrect or inaccurate technology, then you’d better believe that’s going to create some issues,” he concluded.

That worry has preoccupied some on the regulatory side, with a user’s “substantial dependence” on the quirks of the system forming the basis of risk stratification in the Clinical Decision Support Coalition’s proposal on how the U.S. government should regulate the sector. (See (Also see "Coalition Proposes FDA Regulatory Framework For Clinical-Decision Support" - Medtech Insight, 29 Jul, 2013.).)

Future Of Watson

Despite the worries, officials at IBM and Cleveland Clinic are pushing forward with the technology. And, they say, there are two other potential sources of data that are being eyed for incorporation with Watson-style analytics: genomics and medical images.

The insights Watson is minting now will have some effect on IBM’s other current ventures with the technology. The firm has partnered with several health care companies, including insurer WellPoint Inc. and hospitals Memorial Sloan-Kettering Cancer Center of New York and The University of Texas MD Anderson Cancer Center of Houston. The firm’s data analysis expertise is helping with oncology tools that the hospitals have launched.

IBM executives can’t help but think about the future, even if it’s some time away. Brown joked that while the “new computer overlords” are far in the distance, he’s trying to bring them to medicine.

Jelovsek simply said Watson’s education is still focusing on the basics. “It’s early in medical school. It’s like first year of medical school.”

[Editor’s note: This story was contributed by “The Gray Sheet,” which provides in-depth coverage of medical device and diagnostics developments.]