DeepMind Health, the division of the Google-owned AI company that’s applying machine learning to medical data in the hopes of profiting from diagnostic gain, has inked another services agreement with the U.K.’s National Health Service — expanding the deployment of an alerts, messaging and task management app, Streams, to a hospital in Taunton & Somerset.
This expansion comes despite ongoing controversy over the company’s first NHS data-sharing agreement. The sharing of 1.6 million patients’ medical records with DeepMind by the Royal Free NHS Trust during the development of Streams remains under investigation by the U.K.’s data protection watchdog, the ICO.
Patients were not informed nor their consent sought. Yet the Streams app has since been actively deployed in the Royal Free’s hospitals. And DeepMind is now forging ahead further, by inking a commercial agreement with a third NHS Trust to deploy the task management app, following an agreement with Imperial College Healthcare NHS Trust last December.
Announcing the latest Streams app agreement on its blog, DeepMind makes no mention of the ongoing data-sharing consent controversy attached to the app’s development.
“At Musgrove Park Hospital, part of Taunton and Somerset NHS Foundation Trust, [Streams’] features will alert doctors and nurses to a potential deterioration in their patients’ vital signs that could indicate a serious problem,” it writes. “We believe that by making it as quick and easy as possible for clinicians to intervene if something is wrong, we’ll be able to improve patient safety across the hospital.”
DeepMind would not comment when asked about the ethics of expanding a commercial rollout when doubt has been cast over the legal basis under which patients’ data was obtained and used during the development phase of the app.
DeepMind and the Royal Free previously maintained there was no need for them to obtain patient consent for the sharing of medical records as the Streams app would be used for so-called “direct patient care.” However, this May a review of the arrangement by the U.K.’s patient data safety advisory body, the National Data Guardian (NDG), resulted in Dame Fiona Caldicott taking a very different view.
“My considered opinion therefore remains that it would not have been within the reasonable expectation of patients that their records would have been shared for this purpose,” wrote Caldicott in a letter sent to DeepMind and the Trust in question on February 20.
The NDG has been liaising with the ICO as part of its investigation into the data sharing.
Asked whether it has had any specific concerns relating to the controversy surrounding Streams, the Trust’s deputy chief executive Peter Lewis told us: “Clearly we are taking the information governance issues around this very seriously. And we’ve looked at exactly what we are doing with that app, and will be doing with DeepMind. Clearly we’ve taken the appropriate advice on that. So we’re quite clear in terms of what we’ve done and that it’s legal, what we’re doing.”
There is not yet a firm date for launch of Streams within Musgrove Park Hospital, whose website states it treats more than 450,000 patients annually. “We’re starting on the planning phase just now,” Andrew Forrest, CIO, told us. “We’ll be quite open about when we’re going to start using it as a pilot once we know.”
The Trust also said it does not yet know how many patients’ medical records will be shared under the arrangement — because work has not yet commenced and no data has started flowing.
In a statement about the forthcoming deployment of the Streams app at Musgrove Park Hospital, the Trust says the app “will be available at the bedside to alert doctors and nurses to any patients needing immediate assessment, and help them rapidly determine whether the patient has other serious conditions such as acute kidney injury.”
“What we’re doing with DeepMind is not inconsistent around the roadmap we had with digital anyway,” added Lewis, in an interview with TechCrunch. “In terms of thinking about how do we need to run a hospital if we can really take advantage of what we can do digitally — what will make a big difference?”
Data streams under review
DeepMind co-developed Streams between fall 2015 and 2016 with the Royal Free NHS Trust. The app uses an existing NHS algorithm to push alerts to care staff when a patient might be at risk of a condition called Acute Kidney Injury.
However, after it emerged how much patient-identifiable data was being shared under the original arrangement, and how regulators had not been pro-actively informed of the project, the ICO initiated a probe.
DeepMind and the Royal Free subsequently went on to reboot the project via a new contract in fall last year. Yet underlying data governance questions remain.
The new services agreement, with Taunton & Somerset NHS Trust, appears, at first glance, to be more a robustly defined entity than the original information sharing agreement (ISA) inked with the Royal Free NHS Trust.
A redacted version of the services agreement between DeepMind and Somerset has been uploaded to DeepMind’s website. Several portions have been entirely redacted — such as a section listing subcontractors on the project, and a diagram showing a data flow map.
Notably, though, there’s a narrowing of the types of data that are being shared — to specify the Somerset Trust “shall not provide DeepMind with access to the personal data of patients who are not active patients.”
This contrasts favorably with the first Royal Free ISA, which included sharing historical medical records from patients dating back five years, for example, many of whom may no longer be actively undergoing treatment at the hospital or ever likely to return.
There also is a limit on DeepMind’s processing of Trust patients’ personal data so it cannot be used to develop software outside the remit of the agreement.
And there are what look like terms to purpose-limit how DeepMind can exploit any IP that emerges as a result of the collaboration — i.e. insights that can be deduced from the data, and/or derived as a result of the company being in a position to see how NHS detection moves through to treatment, for example.
Again, IPR limits were missing from the first Royal Free ISA.
Here’s the relevant paragraph from the Somerset agreement:
The Trust’s Lewis told us: “Two things are really important that we have been very clear about: one is that we’re not using any patient data for testing. We will use dummy data for testing the app here. And we will then, as we roll it out, we will be using live data and it will only be our clinical staff who are accessing that live data to treat the patients when we do it.
“The second important consideration was the amount of data that is transferred — so that we’re only transferring data that relates to active patients who are under our care… So it’s very clear therefore what patients are active and only transferring data relating to them.”
He added that “active patients” means patient who are on an “open treatment pathway with us,” and patients admitted in emergencies in some circumstances — given there may also be a risk of A&E admissions developing AKI.
“What we’ve also done is written to the National Data Guardian outlining this approach — not just to seek advice if we are taking the wrong approach but to confirm that we are doing work which is safe and which is legal,” added Forrest. “We haven’t started this work yet — this is our planned approach.”
Giving an early take on the latest Streams agreement, professor Eerke Boiten, who conducts research in data privacy and ethics, suggested that “lots of lessons” have been learned by DeepMind after its Royal Free collaboration ran into trouble — noting, for example, that the Somerset agreement includes an “explicit statement that data will not be linked.”
Concerns have previously been raised about what else the Google-owned company might do with Royal Free patients’ data as a result of a lack of clearly defined limits in the contract.
Digitizing health data and delivering apps
Streams has never utilized any of DeepMind’s AI expertise or its own machine learning algorithms. Although a memorandum of understanding between the company and the Royal Free set out DeepMind’s hope to “gain data for machine learning research” as part of the wider, multi-year collaboration between the pair.
There is no similar MoU with the Taunton & Somerset NHS, according to Lewis. Nor is there any mention of applying AI or DeepMind machine learning algorithms to its patients’ data — the focus is on text messaging and clinical task management.
“This is specifically about the Streams application. We don’t have any other agreement — an MoU or a formal contract — with DeepMind. Just this one,” he confirmed, adding that the Trust has no interest in applying AI to patient data at this time.
The five-year services agreement with the Taunton & Somerset NHS Trust, which was signed on May 10 this year, covers both the delivery of the Streams app and an underlying patient data API infrastructure — what’s known as an FHIR API — which DeepMind fleshed out in more detail last year.
While the company began its health tech push by working on a single app (Streams), it has since bolted that to this underlying app delivery infrastructure — thereby potentially positioning itself to be a broker for other app developers wanting to push their own apps to NHS Trusts in the future.
On this possibility the Trust sounded upbeat.
“The whole point of the FHIR development is it’s open and standardized so that there are options which become available. I think the FHIR standard will become the defacto standard across the NHS completely,” said Forrest.
While the services agreement is tied to Streams, the Trust also is evidently hoping to be able to apply a similar task management app approach to additional conditions in the future — with the agreement talking about additional “modules” that might be developed for the app, i.e. to expand out from just acting as an earlier detection warning system for AKI.
Lewis explained the aim is to digitize other existing paper-based process — though it’s not yet clear exactly which additional “condition alerts” might be bolted on to its deployment of Streams in the future.
“This is really based on the patient, so that what we’ve got in there is data about the patient and about what’s happening to the patient while they’re here with us as an in-patient that can help us identify when there are potential problems,” said Lewis.
“So whether it’s about AKI or whether it’s about sepsis, for instance, we will know when things are happening with that patient that we need to react to. And all of that happens at the moment — the problem is a lot of it is recorded manually on paper, and staff have to do specific calculations to calculate what’s called things like ‘early warning scores,’ which means we need to react. So whilst that works well on paper you’ll appreciate it’s much better to digitize that and have it done much more in real time, and then the alerts and so on are automated.”
“That’s how we really will drive safer patient care through doing this,” he added.
Commercial question marks
Lewis confirmed to TechCrunch that the Trust did not put out a tender for the development of what it bills as a “patient-focused mobile app.”
Rather, the agreement with DeepMind came out of a series of conversations with the company. He noted that the Trust was last year one of a handful named as a “global digital exemplar,” which led it to “various conversations” with suppliers — aimed at delivering on a digital roadmap.
“We were having a conversation with various suppliers, including DeepMind, and this agreement has come out of that,” he told us. “I can’t remember exactly when the initial contact happened but there was quite an informal contact between us and DeepMind and this just grew out of a conversation that was happening… It wasn’t something that we deliberately went and said this is what we need to do.”
What we haven’t done, and haven’t needed to do, because we’ve got legal advice for this, is gone out to a full competitive tender process. Peter Lewis
Is that standard practice for the Trust’s procurement of services? “It depends on exactly what we are doing and how we are testing those options. So we didn’t just say ‘DeepMind’s the only option and that’s what we’re going to do’ — we’ve been looking at alternatives. What we haven’t done, and haven’t needed to do, because we’ve got legal advice for this, is gone out to a full competitive tender process.”
He also confirmed there are no wider agreements with the company — such as the Royal Free’s MoU. But he declined to disclose anything about the commercial arrangement with DeepMind.
“We’re not at liberty to discuss the commercial detail of the contract,” said Lewis.
Commercial terms have been redacted from the ISA uploaded to DeepMind’s website:
On this, a DeepMind spokeswoman told us: “The contract is minimally redacted… I believe it would be up to the Trust if they wish to disclose any further commercial information.
“Regarding the business model, under our agreement the Trust will pay DeepMind a limited service fee if DeepMind incurs support costs when providing the services. As with our other agreements, we don’t believe in charging the NHS for yet-to-be-proven solutions,” she added.
“Only when we can prove that we have improved outcomes will we be paid accordingly within IT supplier market rates. We’re not driven by a desire to maximize profit, but rather to create a mutually sustainable business model so that we can continue to grow our team, work with more hospitals and, ultimately, help more patients.”
The ongoing lack of any commercial information for the pricing of the Streams service is significant given that this is a service being sold to the public sector. And if NHS Trusts are inking deals with a tech giant without putting out open tenders, there could be legal issues to consider — such as compliance with procurement rules, and even concerns relating to possible anticompetitive behavior.
Although, evidently, Lewis is confident that the Trust’s own legal advice on the procurement point is sound — i.e. that it didn’t need to run an open, competitive procurement in this case.
Update: The Trust has now given TechCrunch the following statement regarding the financial terms of the five-year agreement: “We are not paying DeepMind for the duration of this contract, unless the service support exceeds £15,000 a month.”
Another consideration: The BBC reported earlier that NHS patients are not being offered an opt-out for their medical records to be shared with the Google-owned company.
Asked about this, Lewis said their aim is to engage with patients so they are aware of the app and its implications.
“We need to work this through. Clearly there’s a balance here — we’re trying to be very open about what we’re doing and therefore we have a process in place from here on about how we’re engaging with patients to work through exactly how we do that. But this is built on the premise that we are using the information about these patients in this way to improve their care and make them more safe so that particularly we’re escalating deteriorating patients much earlier and we can therefore react to that,” he said.
“That’s the whole premise on which we’re doing this but we are going to go through that engagement process with patients — one so they can understand exactly what we’re doing, and two so that we can understand those concerns and look at how we can mitigate them.”
The Trust will be holding workshops, displays and open day events with staff and the public in the coming “days and weeks,” so that people can “see how the app works, what it will mean for patients and how it might be developed in future,” as it puts it.
The first of these events will take place on July 17, with information slated to be displayed in the public concourse of the hospital.
The Trust also has indicated it intends to publish its Privacy Impact Assessment (PIA) of the agreement with DeepMind — though at the time of writing, this document has not yet surfaced. [Update: it’s now been put online here.]
In the (partially redacted) document the Trust describes its intention for DeepMind to help it deliver what it describes as a “truly integrated Electronic Patient Record (EPR)”, with the Google-owned company acting as the data processor and “delivering that EPR via a mobile application”.
“Neither the mobile application nor Trust data will use DeepMind’s machine learning, Artificial Intelligence or unauthorised algorithm/processing,” it further states. “And will solely be used for the purposes of direct patient care. The Trust, as data controller, will define any decision tree based algorithms to be applied to defined datasets or patient cohorts in line with NHS clinical guidance and will instructed [sic] DeepMind on all future developments.”
The PIA also confirms the Trust will not explicitly seek patient consent for the use of their data — arguing it would be too burdensome to collect consent from all patients and to run two systems in parallel.
“The Trust has received the necessary assurances that DeepMind will only act as a data processor as defined within the service agreement, and it will ensure that the agreement is complied with,” it adds. “The Trust has been mindful of the sensitive nature of the personal data which will be processed. It is satisfied that, for the reasons summarised in this PIA, its processing is justified and proportionate.”
Commenting on the services agreement, Phil Booth, coordinator at health data privacy group medConfidential — which raised early concerns over the data-sharing between DeepMind and the Royal Free — told us: “Will the copy of data that Google will keep be regulated as any other life critical EPR [electronic patient records] system? Or do they still believe that the rules don’t apply to them?
“The reason you have backups, is so that if there’s a problem, you can use them. Is this PIA claiming an unregulated and unaudited system will be used by the Trust?”
In a statement about its ongoing investigation into the original data-sharing deal between DeepMind and the Royal Free, an ICO spokesperson also told us: “The ICO continues to investigate the sharing of 1.6 million patient details by the Royal Free NHS Foundation Trust with Google DeepMind in support of the testing and development of an alert, diagnosis and detection system for acute kidney injury. We hope to conclude that work shortly.
“We will also speak with Taunton and Somerset NHS Foundation Trust to offer our advice on the law,” the spokesperson added.
Correction: This post originally stated that DeepMind’s Somerset services agreement is the second agreement it has inked with an NHS Trust to roll out the Streams app; in fact it is the third. The earlier agreements are with the Royal Free NHS Trust and Imperial College Healthcare NHS Trust.
Source link