PODCAST: Predetermined Change Control Plans (PCCPs) for AI-Enabled IVDs

Mitzi Rettinger: I'm Mitzi Rettinger and this is Expert Insights from DCN Diagnostics. Today we're talking about predetermined change control plans or PCCPs and what they mean if you're working on an AI-enabled IVD. Thanks to FDORA and a new final guidance from FDA, AI-enabled device software functions can now ship with an approved plan for certain future changes. That's a big shift for teams building readers or algorithms on top of lateral flow or other rapid tests. It affects how you design your studies, how you think about retraining, and what done looks like in a submission. To dig into this, I'm joined by two people you've heard on the show before, Dan Simpson, who leads Regulatory Affairs here at DCN, and Emily Friedland, our VP of clinical Research. Dan, can you kick us off with a clear definition? What is a PCCP in FDA's language, and how does that play out for an AI-enabled IVD?

Dan Simpson: Sure. So a PCCP obviously, by that last letter is a plan. And that plan is meant to identify changes that you think may happen with the device or that in this particular case in AI algorithm. Over the lifecycle of the product and address it early in the submission, so you don't have to do subsequent submissions when your device or algorithm changes later on.

Mitzi: What pushed FDA to formalize this? Why wasn't the old software change guidance enough once AI came into the picture?

Dan: By nature, AI changes over time, and so with the old system, when changes would happen to your device, when you notice your device was shifting or things like that, or you made a modification, you would have to follow the FDA guidance on modifications to your device. And a lot of the times you would have to do a new submission in order to address those changes. And so if an AI device is changing frequently over time, that means multiple submissions, most likely that you would have to do so. That's impractical. So FDA wanting to foster innovation in the AI space has come up with this plan to make it easier for AI products to get on the market.

Mitzi: So, Dan, based on that definition and just kind of in layman's terms, you know, someone who's not a regulatory expert, if I had a device and before this came into play, and let's say it does have AI on it, and it's learning over time and it's starting to shift within this diagnostic, what would we have had to do in the past? And then why would that be so impractical? And now that this is in place, how does it look different?

Dan: That's a great question. So in the past there's a guidance by FDA on how you address changes after you do a marketing submission with AI. Because the AI changes so much, you would have to do new submissions frequently due to just the normal operations of the AI, and that includes doing a risk assessment of the changes and really analyzing what that does. So now with the PCCP, you can do that ahead of time instead of having to do a new submission with FDA every time there's a major change to the product.

Mitzi: Define a major change.

Dan: A major change is one that substantially affects either the safety or efficacy of the device, or could potentially affect the safety and efficacy of the device.

Mitzi: If a company has to resubmit every time there is a modification. What does that entail versus what they can do now? Explain to me the difference between what why would it be impractical? What all goes into that versus today. They can, I guess. Are they predetermining per the definition? Here are the things that we're going to do so that we don't have to resubmit. Is that how that works?

Dan: Yes, 100%. You said it. So the problem with the old system is you would have to re potentially resubmit every time. Either that or you would have to go through a justification of why you don't have to resubmit. But all of that takes time, money, resources. And so with the new PCCP, you can do this ahead of time and you can validate it ahead of time. Those changes with FDA, in order to allow this, this playground to play in, let's call it this area of where you can change without having to document, justify, or submit because you've already done that justification and validation with FDA the first time.

Mitzi: Okay, so the AI PCCP guidance centers on three elements. Can either of you walk through each one of those in the context of an AI-enabled IVD? Maybe this will help me, you know, kind of better understand what you're describing.

Dan: Then I'll start off and Emily can add if she wants to. The three parts is a description of the modification, the modification protocol, and then an impact assessment. And to tie it back to the old system. This is kind of what you would do after your submission before. It's just now you're able to do it as a part of your first submission for changes that you think may happen. So the description of modification, you basically just say what it is, exactly what that change could be, and then you show how you are going to test those changes in your system, in the protocol. And then probably the most important thing, because it always comes down to risk right, with FDA is the impact assessment. So what could potentially go wrong with these changes and how could that affect, you know, in the end, the safety of the patient.

Mitzi: That sounds like that could be a lot for a startup or someone who just generally is trying to understand FDA guidance in general. Where do you see teams getting this wrong in early drafts?

Dan: I really think it comes from how well your design control program is and how well you define your requirements, your design requirements, what does the user need and what do you expect this product to do for AI? How do you expect it to adapt? So you should really have. When you start off with an AI algorithm, you should really have an idea of what that is going to do by your training sets and your validation sets of just testing AI to know how it's going to change over time. So it's really understanding that. And so I could see that some companies may get it wrong by not really doing that adequate, adequate testing of their AI algorithm upfront and really knowing how it's going to perform over time.

Mitzi: Emily, when you look at the modification protocol from a clinical and data perspective, what changes for you? I mean, how does this affect how you would design a study if you had an AI-enabled IVD?

Emily Friedland: It's a good question, really. It doesn't change much in your initial protocol study design. And the reason why I say that is that we have been working with devices that have software algorithms that may change over time for many, many years. What this allows us to do is to plan in advance for post-market, continued clinical research, clinical studies to gather data to ensure that as the algorithm changes, that as Dan suggested earlier, that the safety and efficacy of the product are changing along with that which would in fact then require us to submit an update to the FDA. So we want to continue planning for ongoing data collection in future after our primary study. We also may expand the data points that we collect in the primary study, beyond what's just required to validate the device. Usually we try and in clinical collect as much data as we can where it makes sense for validating your product against the primary objectives of your study. In this case, we may collect additional data. Which often scientific affairs medical affairs teams, marketing teams want anyway for other reasons beyond just the original clearance or approval. And so we may expand that data collection in the primary study so that as we move into post-market, we have a baseline data set to compare to.

Mitzi: For an AI-enabled IVD What are the signs that you should build a PCCP into that first submission? You know, you're talking about what we might want to do in addition from a clinical, but what are some of the other signs that would push someone to want to do this?

Dan: So really it's the nature of the AI itself. Obviously, if it's a machine learning type of thing, it's going to change a lot because it's learning over time. So you just really need to look at the potential for change of that algorithm and really see if you expect it to change a lot. And as Emily mentioned, software has been changing for a while now. It's just that now it's really gone to the forefront because our lives are now inundated with AI. That by nature will change over time as it learns. And then there's Specific ML machine learning applications that really do that. So you really should know how much you expect that to change. And is it really a feature of the device. And obviously if it's a feature of the device and the software, you're going to want to probably consider a PCCP.

Mitzi: It seems just like it would be smart to do that. I mean, again, I'm these are this is this is the regulatory naive speaking here. I feel like if I have an algorithm and I've incorporated AI, it would be silly not to have this upfront because if I don't do it then and there, there are changes over time. I'm going to have to resubmit and that's going to be costly. So it just almost seems like everybody should do it.

Dan: Yeah, I mean, I agree, I think at a minimum you should go through the exercise. Right. You should you should at least do it internally and see if you come up with valuable insights into how this is going to change. Like you said, it kind of starts with your design requirements. But if you go through this exercise of creating a PCCP, at least internally, you'll kind of know whether it's going to be beneficial. But what I tend to agree with you Mitzi think if you're in the AI space, it's almost an automatic that you do this.

Mitzi: So I've heard you say that once a PCCP is approved, it has to live in the QMS or and not just in the submission. So what does that mean in practice for an AI-enabled IVD manufacturer?

Dan: Right. And so, you know with everything there is there's good and bad to everything. So this could be considered, you know, by some people as a downside to a PCCP. But really you're just incorporating it into your quality system basically. So when you do a PCCP with FDA at the submission, you are basically in agreement with FDA that you're going to follow this PCCP throughout the lifecycle of the product. So anytime you make an agreement with FDA, you really need to rely on your quality system to make sure you're doing it. Or there could be problems down the road. You need to set it up in your quality system. You obviously will do validations as a part of your PCCP. Those live in your quality system and then validations will drive procedures to make sure you are when you're manufacturing and testing and doing all those things that you're staying within that PCCP and you're addressing the conditions of that PCCP.

Mitzi: That makes a lot of sense. Don't rely on us to just remember actually document it and do it. As we all know, if you don't document, it doesn't happen with FDA.

Dan: So that is right.

Mitzi: Emily, from a clinical perspective, our clinical operations standpoint, what needs to be in place when a PCCP covered change is ready?

Emily: Really it's those protocols, as I mentioned for post launch, right. So it may be a continuation of your primary study. So you might just continue enrolling under that original umbrella protocol and continue to monitor the performance of the data. And as those changes roll out, having planned analyses to look at the difference. Or potential difference between data collected post-change and data collected pre-change, and making sure that data stays within the predetermined requirements of the PCCP and does not now move your device into an area where you do need to do a post-market change to your application through somebody like Dan. So that's the continued data monitoring. Historically, we have not done a great job of post-market data in the IVD space because it hasn't been as much of a requirement. So generally speaking, you get your clearance and then post-market data is collected for marketing purposes or for maybe claim expansion, things like that. This to me is very similar to the MDR, which now requires a significant amount of post-market data. And having experienced that on the traditional device side, that data didn't exist. And so organizations either killed a bunch of products, thousands of products in some cases, or had to go and really figure out how to get real-world data. So the key for these PCCP related studies is to make sure that you are working through your design control process to plan for what that data collection will be continually after launch in some way, shape or form to ensure that you are meeting the requirement of the PCCP.

Mitzi: If you plan for that. I mean, you know, I could see some people saying, well, oh my gosh, that's just going to be an extra cost that's going to cost me more money. Do I really need to do this? I mean, is there a kind of that place where. I mean, how much is enough? You know where you could do that? And it won't cost as much as what you were doing in the study. Like, is there a balance there?

Emily: You'd be in the commercial space at that point. And so a lot of the time it would be working with even customers to say, like, would you be willing to provide data back to us that has been anonymized so that we can continue to monitor the performance of the product? So that could be done also under like the marketing umbrella, but with a scientific medical monitoring to make sure that everything is being done in a compliant way. So there are ways to get around it without doing a formal clinical study where you're continuing to pay sites or labs to provide service. This is really a surveillance of your post-market data and real-world evidence, potentially to make sure that your product continues to perform as expected and that there aren't any major safety or efficacy changes. Sensitivity and specificity being a significant part of that.

Mitzi: Listening to what you say, I'm sitting here remembering an email I got just like a week ago. And what I'm thinking about is IVDs that are at like at home or that people can do themselves, that you could potentially get this free because I have a wearable won't mention any names here, but they are adding to their algorithm and their device and they're saying, hey, could we put you as part of our trial? Yes, and it's all free for them. If I just recently opted into a research setting. With a wearable device.

Emily: As a research professional, I feel like I have to have offered the opportunity to participate in research and for all listeners. If you're offered the opportunity to participate in research and it's not too much of a hassle for you, please volunteer. But that is a good point. And that is the way a lot of, especially these wearable organizations are collecting data post-market or for market expansion or for new claims or new applications of their devices. So as these kind of wellness devices downloads, the wellness devices move into a more regulated space. And we've seen some real failures of compliance in the market for these types of devices. So the organizations that are being compliant, they do have this user base, who they have access to, who they can compliantly have them opt into ongoing observational research, which is really what this is. It's not as much of a planned clinical research study. It's a real-world gathering of data for which they're going to do analysis for potential claims.

Mitzi: Yeah, and that makes a lot of sense. And where I saw that comparison was even in like women's health. I mean, there's so many things that are at home now for people that does incorporate algorithms and different things to, you know, especially when it comes to hormones. And so that would be a good way for them to collect that without it being an extra expense. Oh absolutely. I like those thoughts. So let's talk about FDA's August 2024 draft guidance on PCCPs for all medical devices. How should teams think about that document?

Dan: Well, teams should really be excited about that, right? So I think what happened is FDA came up with this framework for AI and proposed it to the industry. And the industry responded extremely favorably and probably hinted to FDA that, well, this would be a great idea for all devices in all aspects of devices. Like I said, you know, there was a FDA guidance before for after submission. That basically, in the draft guidance are rolling all of those points forward into pre-submission. So basically any type of design change, that really isn't a change in the intended use of the device more. You know, if you're an IVD, you're changing concentrations of reagents, things like that. You really don't have to prove safety and efficacy again, can fall under a PCCP. So if anything that can be validated probably apart from a clinical study. So what this means is that things also like I could use maybe COVID or the flu as an example. How antigens change over time. You could potentially build that kind of thing. You know, how do you test new antigens, new things as they change each season? How do you validate that? So you don't have to do a new submission every year, basically.

Mitzi: That makes a lot of sense. And I was already, you know, thinking just what you're saying. Critical reagents, you know, anything that you've labeled as critical potentially could have a problem on the back end. And that if you're preparing ahead of time, then you wouldn't have to worry about that. So it sounds to me really what this says is to anyone who has medical device, spending that extra time upfront with regulatory as you're going through your development process and kind of outlining, here's all these things that potentially could cause you to have to resubmit. You know, pre-plan on it. Go ahead and incorporate it into your validation. And then that becomes a part of your PCCP to say, this is what we've done and this is what we'll do. If that happens, that can just allow people to get more new products to market versus having to resubmit the old ones over and over.

Dan: Absolutely. You're dead on there. And again, what I think, you know, companies can do to kind of, you know, it's kind of a new way of thinking now. And so really building into your design control plan, the area when you're planning the actual bullet point that is PCCP planning and go through all of the potential things that could change in, is a PCCP going to be a good thing for your company to do?

Mitzi: Thank you Dan. So for an IVD team that is starting an AI-enabled program right now, what would you tell them about PCCP that you wish everyone already understood upfront? And I'd love to hear from both of you if you both have a perspective or either of you.

Dan: So I think you kind of touched on it a little bit is the first impression may be, oh man, I'm just going to be inundated with procedures and things that, you know, that I don't want to do, but you need to consider that you're doing the work upfront, so you're saving a lot of time and money down the road. So I would really recommend it if it's a possibility, just because in the end, your product is going to be much more sustainable through the lifecycle with less hassle if you utilize this process.

Emily: And what I would add to that, or a couple things. One, just to point out that this is FDA keeping pace with and being a partner to industry. Which doesn't always happen on the regulated side. So we should look at this as a very favorable move from the FDA, and that this really shows their understanding of where technology is going and keeping pace with that, so that manufacturers can continue to bring solutions to patients and to the healthcare community so that investment there is to improve healthcare overall, which is what our goal is always right. So that's the kind of my first thought. And the other thought is it's a small investment of time to free up, as you guys both mentioned, and previously, perhaps your R&D personnel and all other personnel to continue making new product outside of these products. That might change. Also really always fun to see new data coming in and see how things change over time. And I do think manufacturers often kind of lose touch with how products perform after market. So this is also really good for lifecycle management, product and potential future innovation for new product.

Mitzi: Dan, Emily, thank you for walking through this. It's really been insightful for our listeners who want more detail. You can read the full whitepaper from DCN Diagnostics, Regulatory and Clinical team on PCCPs for AI-enabled IVD and future applications, it can be accessed on DCNDx.com. If you're considering a PCCP in an upcoming submission, please reach out to our IVD CRO group to talk through what makes sense for your program. Thank you for joining us for this episode of Expert Insights. Don't forget to subscribe and we'll see you next time.