California tries to thread the needle with new laws around AI clinical decision-making

California lawmakers have never been shy about using legislation to shape the debate on hot-button issues, and artificial intelligence is no exception.

With the federal government taking a largely hands-off approach to AI regulation, Governor Gavin Newsom and the California state legislature are making efforts to offer a state-level model of governance for the burgeoning AI ecosystem – especially in areas like healthcare, where the consequences of a laissez-faire approach could be significant.

The latest bills to land on Newsom’s desk show just how challenging it can be to develop guidance about technology that is still evolving at the speed of light while satisfying the wide range of opinions about the role of AI in clinical decision-making.

What are California’s new laws?

Out of the most recent crop of legislation to address the matter, two bills have been enacted into law, while two others have been vetoed.  

The first to gain the Governor’s signature is AB 1979, which requires healthcare provider entities to take “reasonable steps to ensure that a licensed health care provider…retains the ability to exercise independent professional judgment in their care of a patient whenever that care is informed by the output of a clinical decision support system.”

Providers may not deploy an AI tool or system that “independently perform[s] any clinical function that is required by law to be performed by a person with a professional license,” or use any system that instructs or supervises any unlicensed personnel in the performance of a clinical function that must be performed by someone with a professional license.

There are exemptions for training and study tools, as well as for systems that don’t involve the application of clinical judgement, such as those that automatically inform patients of updates to their records, generate reminders, or help patients find information they’re looking for.  

“Protecting patient safety, keeping our professional workforce engaged, and preserving the integrity of health care requires that we not rely exclusively on AI to do things quicker and cheaper,” said Assemblymember Mia Bonta, who sponsored the legislation. “AB 1979 does that by ensuring medical records stay protected when accessed through direct-to-consumer health chatbots and that licensed health care professionals retain the ability to exercise professional judgment when a clinical decision support system informs their care of a patient.”

The second law addresses bias mitigation. SB 503 requires healthcare providers to “make reasonable efforts to identify clinical decision support systems developed for use by deployers that are known or have a reasonably foreseeable risk for biased impacts resulting from deployment of the system in health programs or activities.” 

Developers will need to make statements about the intended use and foreseeable risks of their systems, and deployers of the technologies will have to regularly monitor their AI tools for bias and take steps to mitigate any issues that arise.

What didn’t make it past the pen?

Another two bills focused on AI clinical decision-making didn’t fare as well, due to concerns about the practicality of implementation.

SB 903 would have placed restrictions on the use of AI tools in psychotherapy, limiting AI systems to administrative and supplementary support.  Companies would have had to secure clearly informed consent from patients before AI was used to record or transcribe mental health sessions, and would have been barred from marketing chatbots as “therapists” or delivering “therapy” services. The bill would also have mandated that a licensed professional review and approve AI-generated therapeutic decisions, triage work, and treatment plans.

“Unfortunately, this bill is overly broad and would drastically limit a clinician’s use of tools that benefit the delivery of care today,” Newsom wrote in his veto message. “I encourage the author and stakeholders to revisit this issue next year to find a solution that ensures professional judgment is not replaced, while still allowing licensed professionals to use tools that enhance their delivery of quality care.”

The other bill, AB 2575, would have prevented employers from punishing clinicians who override AI clinical decision-making recommendations and disallowed vendors from blaming clinicians when a system causes harm.

The legislation received support from several labor unions, including the California Nurses Association, which expressed disappointment at the veto.

“The governor knows AI should never replace a nurse’s clinical judgment, but he vetoed the bill that would have protected us for using our judgment,” said Sandy Reding, RN and president of California Nurses Association. “That makes no sense to any nurse at the bedside. Today, a nurse who sees that the algorithm is wrong and acts to save her patient can still be disciplined or fired by her employer.”

What do these actions indicate about accountability in clinical practice?

The rejected and accepted proposals set up interesting questions about accountability of humans and AI systems in real-world clinical practice.

While California seems supportive of putting guardrails around what AI can and cannot do, the veto of the employee protection bill hints that the state is taking a more cautious approach around how individual clinicians and their employers actually operate within those boundaries.

When vetoing AB 2575, Governor Newsom did not necessarily dispute the need for a similar type of policy, but explained that the bill, as written, included problematic enforcement processes.

Nevertheless, the absence of an employee protection mechanism at the state level leaves open questions about what might happen when a clinician relies on their own judgement, especially if that decision is in conflict with an AI recommendation.  

How can state-level policies best promote and protect the critical role of human decision-making if they don’t also offer sufficient protections for humans when they exercise that judgement?

California doesn’t yet have an answer, and other states are facing the same questions about enacting laws that promote the utility of AI clinical decision-making tools while preserving the judgement, autonomy, and discretion of experienced human clinicians.

As state and federal policymakers continue to wrestle with the nuances of establishing boundaries around AI use, healthcare industry leaders will need to consider how their own policies clarify accountability, support clinicians when they use independent judgement, and ensure that AI can become a tool for enhancing patient care instead of replacing human expertise.

Jennifer Bresnick is a journalist and freelance content creator with a decade of experience in the health IT industry.  Her work has focused on leveraging innovative technology tools to create value, improve health equity, and achieve the promises of the learning health system.  She can be reached at [email protected].