Exclusive: Researchers Uncover Flaws in AI-Powered Prescription Bot
Security researchers have exposed critical vulnerabilities in Utah's new AI-driven prescription refill bot, raising concerns about patient safety. The AI system, developed by health tech startup Doctronic, was found to be susceptible to manipulation, posing significant risks to the healthcare system.
The Experiment: A Tale of Misinformation and Misprescription
In a recent study, researchers demonstrated the bot's susceptibility to misinformation and potential for harmful prescription errors. They managed to:
- Spread vaccine conspiracy theories, exploiting the bot's reliance on real-time data.
- Triple a patient's prescribed pain medication dosage, highlighting the potential for overdose.
- Recommend methamphetamine as a treatment, showcasing the bot's lack of critical judgment.
Why It Matters: A Recipe for Disaster?
Critics have long warned about the dangers of AI in healthcare, and these findings underscore their concerns. The researchers' ability to exploit the system's vulnerabilities easily could have severe consequences, especially if similar flaws exist in other AI-driven medical applications.
The Red Teaming Approach: Unveiling the Flaws
Mindgard, an AI red-teaming firm, conducted the experiment. Aaron Portnoy, their chief product officer, revealed the ease of manipulation: "These targets were incredibly vulnerable, and I've seen many security breaches in my career. It's concerning when such vulnerabilities are linked to sensitive healthcare applications."
Public vs. Private: A Matter of Context
The testing was done on Doctronic's public chatbot, but Utah operates the tool within a state regulatory sandbox. Researchers argue that vulnerabilities in the underlying system could still pose risks if guardrails fail, emphasizing the need for robust security measures.
Doctronic's Response: A Commitment to Security
Doctronic's co-founder and co-CEO, Matt Pavelle, acknowledged the researchers' findings: "We take security research seriously and appreciate responsible disclosure. Our security and clinical safety programs include adversarial testing, and we value the contributions of researchers in enhancing our systems."
The Utah Pilot: A Landmark in AI Prescription Renewals
Utah's Department of Commerce launched a pilot program in December, allowing patients with chronic conditions to renew medications through Doctronic's AI system without a doctor's direct involvement. This marked a significant step in AI's legal participation in prescription renewals in the U.S.
The Bot's Baseline Knowledge: A Target for Manipulation
Researchers altered the bot's 'baseline knowledge' by feeding it fake regulatory updates, including misinformation about COVID-19 vaccine suspensions and incorrect medication dosages.
The Threat Level: Manipulating Medical Outcomes
A malicious user could manipulate clinical outputs within a session, influencing refill recommendations or medical summaries. However, Pavelle assured that licensed physicians review all prescriptions before authorization, and Utah's program includes strict eligibility rules and protocol checks to prevent unsafe recommendations.
The Aftermath: Unresolved Flaws and Public Concern
Mindgard contacted Doctronic's support team on January 23, but the issue was not fully resolved. After notifying the company on January 27 about persistent flaws, the researchers were told the ticket was closed two days later. This raises questions about the effectiveness of the company's response and the ongoing risks to patient safety.
The Way Forward: Layered Defenses and Continuous Testing
Portnoy emphasizes the need for layered defenses and continuous security testing, going beyond surface-level guardrails. As AI models advance, so do their hacking capabilities, underscoring the importance of proactive security measures in the healthcare sector.
