Jake Stowe: Welcome again to everyone. Before we begin, we'd like to cover a few housekeeping items. First, the webinar is being recorded, and it will be sent out along with the slides following the conclusion. We'll also share other resources in the chat during the webinar. Second, go ahead and put questions in the Q&A at any time.
Our colleagues, Iskander and William, are on the line, and they'll be surfacing your questions from the chat to us throughout our presentation. So it looks like many of you on the line are already integrating AI into your product or have plans to, which is really exciting, and we can totally understand why you're here. Alex, from our customer base, is, I think, as expert as they come in the PCCP process, and I'm sure many of you are interested in how you can apply that to your own regulatory workflows. And so we're excited to have that conversation. We will move forward now.
And I think that is a good lead-in, Alex, for us to introduce ourselves. Would you mind introducing yourself to the audience?
Alexander Chan: Absolutely. Yeah. Nice to meet everyone.
I'm Alex Chan. I'm VP of analytics and machine learning at Beacon Biosignals, where we're really pushing forward using machine learning and AI in advancing neurobiomarkers for brain health and really helping accelerate therapies in that area. My background really is in computational neuroscience and medical engineering. I spent a lot of time really at the intersection between neuroscience and machine learning, applying machine learning to understanding the brain, and also really moving forward towards understanding how we can use machine learning to help people with various disorders of the central nervous system. I spent a lot of time in my past career at Apple as well, leading the health technologies algorithms team that delivered sleep staging for Apple Watch as well as sleep apnea detection.
But really, really happy to be here with you, Jake, to chat about PCCP and the things we're doing on the machine learning and AI side at Beacon.
Jake Stowe: Yeah. I'm super excited for the conversation, Alex. And to give you a brief introduction to who I am, my name is Jake Stowe. I am Ketryx's vice president of client operations.
I manage pretty much all aspects of customer experience for Ketryx, and that includes implementation, support, and customer success. And my team is very heavily involved in helping our customers achieve the compliance and the operational outcomes that they seek to achieve. I studied at MIT. I got a master's degree in business administration and a master's in systems engineering. While I was in that experience, I co-oped at Amazon, where I did process engineering for them in one of their fulfillment centers.
I then moved on to Amgen, where I served in various operational capacities. I was a major deviation investigator in one of the largest biopharmaceutical manufacturing plants in the world, and I also was a manufacturing manager in a drug substance plant before I moved on to do digital product management, I should say, for a team that built AI models for various aspects of Amgen's process development function. And now I'm at Ketryx doing this and helping to solve a problem that I saw as being critically important while I was at Amgen.
Just briefly going to take you through the agenda. We're going to do a brief introduction to Beacon Biosignals. We are also going to talk about what PCCPs are and why Beacon pursued them. We're also going to talk about how Beacon trains and validates its AI models, and we're also going to talk about combining PCCPs, agile methodologies, and the right tooling to release faster in a regulated environment.
And so to kick us off, I want to ask you, Alex, can you share a bit with us about Beacon's mission?
Alexander Chan: Yeah.
Jake Stowe: Can you share a bit with us about your mission?
Alexander Chan: Yeah. So at Beacon Biosignals, we're really focused on leveraging AI and machine learning to really accelerate therapy development for diseases that affect the brain. This really can include a number of things. Obviously, one big one is sleep disorders, neurological disease, psychiatric disease. And really, anything that touches the brain is what we at Beacon are striving to accelerate therapies for.
Our main focus right now really has been working with pharma companies who are developing therapies for the brain, to really help accelerate their drug development in the neuro space by helping them incorporate these quantitative neurobiomarkers that we really can, in many cases, uniquely extract using machine learning and AI types of techniques. Incorporate these neurobiomarkers as part of clinical trials, and really help them move these therapies along to patients much more quickly. And we're also starting to think about how we can take our machine learning, our medical devices, and use them more directly to help diagnose and manage disorders of sleep, neurology, psychiatric disease, those types of things.
And so that's really what we're focusing on, and AI and machine learning really take a front and center role in helping us fulfill that mission.
Jake Stowe: Yeah. That's very exciting. Alex, one thing I wanted to make sure I asked you as a part of this: I've worked with you now for a couple of years, and I know that you are a world-class engineer. And I wonder if you could tell all of us what brought you to your current role at Beacon. What brought you to this mission?
Alexander Chan: Yeah. Absolutely. I think a big part of this is that my background has been in neuroscience. And so understanding the brain and trying to help those with diseases that touch the brain has always been something that is really personally interesting to me.
And so being able to get back into this area after spending a lot of time at Apple, also in the health space, but not quite in neuroscience or in neurology, was very exciting. That was one big piece of it. The other piece is really coming back to a startup. I had spent some time at a small startup, and I really enjoyed that particular experience, because you were working with a small team that's incredibly passionate about its mission, and you're able to potentially really effect change in a really quick and rapid way, and iterate towards a solution that'll really be able to help people with unmet need. And so I think that combination of things, being able to work in the neuroscience space with machine learning in a small startup, was really appealing to me.
Jake Stowe: Yeah. It was like a perfect job for you, wasn't it?
Alexander Chan: Yeah. I think it was really great to be able to find this particular role, and it's just been incredibly fulfilling over the now three or more years that I've been at Beacon.
Jake Stowe: Yeah. That's awesome. And it's a great lead-in to our next question. Would you mind walking us through the Beacon platform and who it helps? And could you touch on why it's important to you to get your products to market faster?
Alexander Chan: Absolutely. Let's start with the latter part of this. I think a really important part of this is core to our mission. Right? In the end, we want to be able to help patients who have various diseases that touch the brain. To be able to help those patients, we need to be able to rapidly accelerate therapy development, as I mentioned, with our pharma partners, especially in clinical trials.
And being able to do that well means being able to iterate quickly on the products we provide them, and respond to their needs in a really quick and reactive way. So at Beacon, what are we building? We're building a platform that really allows us to acquire and analyze brain data. It's right now largely focused on sleep, and it really is built to power these clinical trials for our pharma partners who are developing these types of neuro drugs. And so on the left, it really starts with acquisition hardware that makes EEG, or essentially measuring brain signals, much more accessible.
And we'll talk a little bit about how the current status quo is quite difficult for actually obtaining these types of signals. And so we have this wearable hardware that we call Waveband that allows us to acquire these signals easily in a longitudinal fashion in patients' homes. This data comes to the Beacon Cloud, where it goes into the Beacon data store, which allows it to be organized in a really structured way. And on top of that, we have analytics, which really is the core of the machine learning and AI analysis that we run to analyze this data, since a human really is not well suited to analyzing large amounts of longitudinal data here.
Jake Stowe: Mhmm.
Alexander Chan: We take that, and we display it in what we call the Beacon HQ. It's a front end that allows our own data scientists and our partners to be able to explore their data and generate scientific insights from it. And so all of this really enables trialists, researchers, other stakeholders, and clinicians to draw insight and more easily leverage brain signal data. And so as part of this, we'll dive a little bit into two specific medical devices that really leverage and exemplify the AI/ML capabilities that we have.
Jake Stowe: Yeah. And it's a great lead into the next slide. Can you tell us, Alex, a bit more about these two products with approved PCCPs?
Alexander Chan: Yeah. So both of these are our two FDA 510(k)-cleared medical devices, both with PCCPs. These are really the first two sleep-related medical devices with PCCPs authorized at all.
The first one, starting on the left, is called SleepStageML. This is a software as a medical device, or SaMD, that really, at its core, is a machine learning algorithm that aims to accelerate the evaluation of traditional in-lab sleep studies.
Jake Stowe: Mhmm.
Alexander Chan: So to say a little bit more about this: when a patient comes in to get diagnosed with a particular sleep disorder, they usually have to go into a sleep lab and get hooked up with a bunch of sensors on their head and on their body to monitor their brain activity, their breathing, their movement, all kinds of different signals simultaneously, and they're asked to sleep. This requires a trained technologist to do.
And then after they sleep for that night, a human expert would have to review that entire recording, all six to eight hours of it, 30 seconds at a time, to determine whether the person was in a particular stage of sleep. Were they awake, in light sleep, deep sleep, REM? And it takes an enormous amount of time for human experts to be able to do this. SleepStageML essentially is a machine learning model that automates that analysis of the EEG, the brain signal data, to do sleep staging. And this is a service we provide for our pharma partners who are doing traditional in-lab studies.
The other one of our medical devices is the Waveband device. This one's a combination of hardware and software. The core and really important part of this is the headband, which allows a patient to put it on themselves and acquire a night of brain signals, the same kind of brain signals you get from a traditional sleep study, but acquired in the patient's own home, in their own bed. The machine learning part of this is on the software side, where we're again using a machine learning model to analyze the full night of data to estimate sleep stages 30 seconds at a time for that entire night. And so this really allows us to evaluate sleep in a much more naturalistic environment.
Jake Stowe: Awesome. Could you go into a bit more detail about your AI model for sleep staging? And could you talk to us a little bit about your approach for training and validation?
Alexander Chan: Yeah. Absolutely. So both SleepStageML as well as the Waveband medical device have at their core a machine learning model, and actually an algorithmic pipeline, that performs this sleep staging. It's very, very similar across the two devices. It starts with the input signals. In the SleepStageML case, this is brain signals from a traditional in-lab sleep study. In the Waveband case, this is the brain signals that we get off of our dry-electrode EEG headband.
This goes through a number of stages: first signal preprocessing, which does some digital signal processing, filtering, those types of things to the signals, getting them ready for the machine learning model itself, which performs that sleep staging piece, followed by a post-processing piece that takes those probabilities that are output from the machine learning model and converts them to discrete stages every 30 seconds of the night. And so really the core of this is that machine learning, that green machine learning box there.
And for both these cases, we train these models on thousands to tens of thousands of nights of recordings with ground truth labels. The models themselves are fixed, so they don't continuously adapt in the field, which is an important consideration on the regulatory side. And then in terms of how we go about validating these particular devices, it involves collecting a representative sample of data from the device itself or traditional sleep lab data, and demonstrating that the performance of these devices meets our particular requirements and specifications and is able to actually help clinicians in the review and assessment of sleep.
And so those three components are actually important later on in terms of how we ended up structuring parts of our PCCP, in terms of how we describe those modifications.
Jake Stowe: That's so interesting. Alex, I have a bit of a naive scientific question for you. What is the variance like in this data? How different are people's experiences of sleep?
Alexander Chan: Oh, yeah. It's really interesting. I think sleep is one of these things where everybody does it, and so everybody has an opinion. But it is also very variable over the course of nights. And so it's a well-known fact that measuring a single night of sleep is not a great diagnostic test for many things.
For example, in sleep apnea, breathing events and these types of things can change from night to night. Individuals' sleep architecture can also change quite a bit from night to night based on your prior day's experience and many other factors. And even the ground truth of sleep staging is highly variable. When you ask two humans to stage the sleep of a person over the night, or even ask a single person to stage the same recording separated by several months, there's huge variability there. And I think this is one of the big promises of using machine learning: to allow us to get to more consistent states, where we're able to more consistently stage and analyze these types of data without some of the subjectiveness that inevitably comes in when humans are looking at it.
Jake Stowe: Yeah. That's so cool. I mean, it's one of those things that's so mysterious, what sleep is about and why. It's a really, really interesting problem, Alex.
Alexander Chan: Yeah. It's just something that I think, from our perspective at Beacon, is so interesting, also because sleep is an insight. It's a view into the body that can reflect so many other things that are not just sleep.
Jake Stowe: Yeah.
Alexander Chan: And so it really is this really nice window into human physiology that we think is super important to study.
Jake Stowe: Yeah. So cool. We have a question on the line. Steven, if you don't mind, I'm going to delay your question until we get through the next question, and then we'll come back to it. But Alex, could you please explain to everyone what a predetermined change control plan is, and why is it valuable for AI-enabled devices?
Alexander Chan: Right. Yeah. This is great. So a predetermined change control plan really allows a manufacturer of a medical device to modify their medical device without a new premarket submission by essentially prespecifying the exact modifications that might be made, the methods, and how you go about validating them. And really, the purpose and the goal of it is to allow for the continued safe, effective, and efficient update of medical devices via these preapproved pathways.
And so this diagram on the right here has two separate sides. The left side really shows what updating, in this specific case, a machine learning-based or AI-based medical device might look like without a predetermined change control plan, where you'd go through and modify it. Maybe you're retraining the model itself with more data. You'd have to go through validation and verification of that model, and the software as a whole, and the medical device as a whole. Then you have to go through this orange box.
Right? This is interacting with the FDA to get approval or clearance for this premarket submission for the changed and updated medical device. Then you can go ahead and market and deploy that, and you'd cycle through that loop. The issue is every time you want to cycle through that loop, you have this place where you have to interact with the regulatory body. And so what a PCCP allows us to do is essentially front-load that effort, such that once you have undertaken the efforts to prespecify, and have gotten authorization for, the specific modifications you might make and how you go about validating and verifying those modifications...
Once you have approval, or authorization, for that plan, which is the PCCP, now with this orange box, you can continue to iterate without necessarily needing to interact with that regulatory body, as long as those modifications fit within that plan, which is a really powerful thing, especially for machine learning. For one, on the machine learning side and with AI/ML devices, it's really common for us in this field to want to continue to improve our models by incorporating more and more training data. It's something that's commonly done all across AI/ML, even in nonregulated spaces. And being able to do that efficiently is really important for us to be able to continue to improve our products. The other piece of this is that these machine learning models are trained based on data, and the data distributions that we're operating on in the real world can change for a number of reasons.
Sometimes the patient populations you're using your medical device on might change. Sometimes clinical standards of care might change, which changes the data coming in, and being able to address those types of changes quickly is really important. And so the PCCP framework, I think, is a really important tool to allow us to address those types of updates to machine learning models, which really will allow us to continue delivering effective and safe medical devices.
Jake Stowe: Yeah. So I think our next question should cover your question, Steven. So let me just advance. Oh, I'm sorry. That's not the slide I wanted. So let's pause and answer Steven's question here, Alex. Steven asks, what changes are allowed under the PCCP, and does the software undergo full V&V with each change?
Alexander Chan: Great question. Yeah. So I think we might touch on this a little bit later as well, but there are a number of modifications that you can make. In most cases, the PCCP can't change the intended use or indications for use of the device. But the modifications typically are one of three main types.
One is usually modifications whose purpose is to change or improve the performance of a machine learning, or AI/ML, model, especially if we're focusing on that type of device, an AI/ML-enabled device. Two, modifications that are related to the device's inputs or compatibility. So for example, if you want your machine learning model to accept inputs from maybe a different accessory or a different medical device that it's analyzing, having a PCCP that allows for that type of modification is another piece. And then one other one is that there are some modifications around the device's use and performance. For example, within specific subpopulations that are within your intended use or indications for use, you could continue to train your model to perform better on these particular subpopulations, or at particular sites, or things like that.
And so those are the types of purposes. In terms of the actual modifications that can be made, it really depends. I don't think there are any hard and fast rules about what those modifications can be, but we'll talk a little bit more about how to navigate some of that complexity in terms of the specific technical modifications you can make to your device.
Jake Stowe: Yeah. Awesome. So we're going to advance into a poll here. And while we're doing that poll, I have another question for you, Alex, which is from Jess on the line. How does the data that your products collect compare with the detailed sleep data my Galaxy Watch collects?
Alexander Chan: Yeah. It's great. So I think a bunch of the consumer-based sleep monitors are usually based on recording accelerometer data, so gross movements. You can actually see interesting things on the accelerometer from wrist-based wearables, things like breathing, some crude breathing metrics. Sometimes you can even see heart rate and those types of things via the optical sensors. And so with a lot of that, you can estimate sleep reasonably well using a watch-based sensor.
But the thing to keep in mind is sleep is defined as a brain state.
Jake Stowe: Mhmm.
Alexander Chan: And so by not measuring the brain itself, you leave things on the table, especially if you really care about disease populations, which is where Beacon is focused. You really care about these abnormal sleep architectures and patient populations, which it's really difficult for wrist-based sleep tracking to accurately measure in many cases, and there are particular substages of sleep that you can't measure with your watch. But I'm a big fan of these types of consumer wearables for sleep tracking and that type of thing.
But I think for our purposes, we do think monitoring the brain itself is really important.
Jake Stowe: Yeah. Definitely understand. There's only so much you can do with the device sitting on your wrist. Right?
Alexander Chan: Absolutely. Yeah.
Jake Stowe: Awesome. Thank you so much for the question, Jess.
And we are sharing the results of that poll with you. So it seems like the challenges that you all are facing are that there are unclear regulatory guidelines. You're still trying to determine whether the PCCP aligns with or requires a significant change assessment. And I see incorporating PCCPs into the significant change assessment process, and then there's some tailing here about planning and scheduling PCCP changes. So it's very interesting. Appreciate all of the information, everyone.
We will move forward with the questions now. So, Alex, could you tell us what are the key regulatory considerations we should be thinking about when preparing a PCCP?
Alexander Chan: Yeah. Absolutely. So I think this gets back to some of the previous questions. But right, PCCPs, I think, really are meant to support changes to a medical device that would usually require a new premarket authorization. Right? And so things that, if you go through the typical flowcharts and FDA guidance, would not require a new premarket submission are not really within the scope of putting into a PCCP. Those should continue to flow through existing pathways, in terms of submitting letters to file if these are things like small bug fixes, and through your typical quality system.
And so really, PCCPs are meant to address those larger changes that would otherwise trigger a new regulatory submission. And in most cases, many algorithm updates, if those algorithms are core to the particular products and have a particular risk profile, often would trigger this. Not in all cases, but in many cases, which is why it's well suited for AI/ML types of medical devices. I had mentioned before also that PCCPs shouldn't change the intended use or, in most cases, the indications for use for medical devices. They can be part of most 510(k), De Novo, or PMA submissions.
Jake Stowe: Mhmm.
Alexander Chan: It really does include most of the types of submissions. It just can't include PCCPs in some of the submission types that don't require explicit FDA approval, or submissions that have a shorter review cycle. Those are the few limitations there. One thing that's important to note is the PCCP itself is considered a technological characteristic of the medical device itself. And so what that means is that updating the PCCP itself is essentially considered updating a technological characteristic of the device, and would typically require re-review by the FDA for the update to the PCCP.
And the other piece of this is that even in the case where you are adding a PCCP to an existing medical device, that addition of the PCCP, again because it's a technological characteristic, would require a full submission to get that PCCP authorized. And actually, in our case, we experienced this firsthand, and I'll talk a little bit about that later. And then the final thing to note here is that when you're using a device that has an authorized PCCP as a predicate, the device itself now can change without needing to go through additional premarket submissions. But when you're using a device with a PCCP as a predicate, you should be using the version of that device that was cleared before any PCCP-authorized changes were made to that device. Maybe we can go to the next slide...
Jake Stowe: Definitely.
Alexander Chan: ...and talk a little bit about what a PCCP looks like in terms of its structure. There are three main sections to a PCCP in terms of the document itself: a description of the modifications, the modification protocol, and then the impact analysis. The description of the modifications is really intended to list all of the really specific proposed modifications you're planning to make to the medical device, which essentially defines the scope of that PCCP. And so if any of those modifications are being made, then this PCCP would be the thing that kicks in to help ensure the guidelines for validation and verification and how you actually go about implementing it. Each of those modifications should be able to be V&V'd.
You need to specify things about the modifications regarding whether they are intended to be implemented manually or automatically. And I believe if you intend to have automatic modifications, like retraining of your machine learning, it will likely incur more scrutiny. It's not something that I believe is very common yet, but it is something that you could potentially explore with the PCCP. You do need to determine whether those modifications are intended to be rolled out globally to all medical devices or just some particular subsets, maybe certain sites, certain patients, and also describe the frequency of those modifications. Then the modification protocol, the second section, really describes how you go about implementing those modifications and validating and verifying them.
And so this really takes the place of the clinical validation protocol, as well as the statistical analysis plan, where you're describing how you're going to go about collecting the data to update the device or implement the modification, and collecting the data to test the device in terms of its performance, its algorithmic performance. You set and specify the practices by which you'll go about retraining your model and how you'll go about validating it. And so you will need very specific acceptance criteria for each of your modifications to understand whether a modification should be allowed to go into production. And really, you're linking the modifications themselves to each of the individual components of a protocol, if you have multiple modifications. The final piece of this is the impact analysis, where you're really analyzing, at a high level, the difference between your original device and a potential device with modifications implemented under this PCCP.
You need to discuss the risks and benefits of these modifications, as well as, and this is an important part, how different modifications might interact with each other, and whether there's any impact of multiple modifications being made in terms of the device's functioning and safety. And so these are really the main components of the PCCP that will have to be in every submission.
Jake Stowe: Thank you for that, Alex. That was awesome. Great introduction.
We have a couple of questions from a gentleman named Paul. I think we answered the first one, but I'm going to give you the second one. When in the FDA review process did you decide to do this? Was it pre-sub versus supplement review?
Alexander Chan: Oh, this is a great question. Yeah. Maybe we can go to the next slide and cover our history first, and then I'll...
Jake Stowe: Yeah. Yeah. For sure.
Alexander Chan: ...go from here. Yeah. So, first of all, I think before we get to that question: how did we go about deciding how to do our PCCPs for our two devices, SleepStageML and Waveband? Right? SleepStageML was a new 510(k) that we were planning to submit at the end of 2023. The guidance for PCCPs for AI/ML-enabled devices came early that year. And this was really well suited for us, because at Beacon we had a growing training dataset. We knew we wanted to continue to improve this particular model.
And so in this particular case, we proceeded with including that PCCP with the very first 510(k) submission for this new medical device, together, for a number of reasons. I think we have a particular set of reasoning for this, and it'll be different for many other manufacturers. But in our case, our device is relatively low risk and also demonstrated relatively high performance compared to a bunch of predicates as well as competitors in the space. And also, I think an important part is we had a fairly well-established validation pathway in terms of what the specific acceptance criteria are and what the methodologies are. We'd go about collecting data and validating these types of devices in a way that had been fairly well trodden in the past by these other devices that came before us. And so because of that, I think we felt confident to go forward with the PCCP here, with this original 510(k).
And to answer this question, in this case, we didn't go with a pre-sub. And part of the risk mitigation here is in this last bullet point. We made sure that the PCCP was decoupled from the rest of the submission, such that the rest of the submission didn't depend on any specific piece of the PCCP and could stand by itself. That way, if we got particular feedback from the FDA that made it risky that the entire device wouldn't be cleared with the PCCP, we could pull that PCCP at the last minute to allow the device itself to be cleared. So that was an important consideration for us.
Then on the Waveband side, this was originally cleared under the name Dreem 3S. The device itself was cleared without a PCCP in 2023. And here, after our success with SleepStageML, we decided to add a PCCP to this device in 2024. And in our communications with the FDA, it was very clear that even though the device itself hadn't changed, adding that new PCCP would require a new 510(k), as I mentioned, because it's a technological characteristic of the device. And so we did have to follow up with a full new 510(k).
The PCCP itself drew heavy inspiration from SleepStageML, mainly because it's a very similar device that just included a hardware component. The architecture was very similar. But we learned a lot from that first one that I'm happy to share more about here.
Jake Stowe: Awesome. We are getting a lot of questions, Alex. Why don't we see if we can knock a few more off? We have an anonymous attendee who asked, do you think the PCCP review and approval process would be different for premarket approval products versus 510(k) devices?
Alexander Chan: Yeah. I can't speak directly to this. This is speculation. I'm also not a regulatory professional, but my guess is yes. The risk profile of the device certainly plays into how much scrutiny a PCCP gets and the amount of controls that might need to be in place for the PCCP. So I definitely expect this to be very different for a PMA versus a 510(k).
I've heard in other conversations that there have been people who have been trying to do this on the PMA side and faced additional challenges. And so I think for us, with a relatively low-risk device, we've had to think very carefully, and we had to put in a lot of controls to really satisfy our reviewers that we're not, for example, proceeding down roads that would lead to overfitting and things like that. And so we've had to put in fairly rigorous controls, but I expect potentially even more would be necessary for a PMA.
Jake Stowe: Definitely. I can only echo that.
The next question I love. Erwin asks, do we have any awareness of PCCPs in non-AI/ML devices?
Alexander Chan: Good question. This is not something that I have firsthand knowledge of. I do know that the FDA fairly recently released the PCCP guidance for general medical devices, not just AI/ML. I don't actually, off the top of my head, know of devices that have leveraged this for other non-machine learning changes. I think there are some, but I don't think I could answer this question specifically regarding existing devices that might have done this.
Jake Stowe: I am also not aware of any devices that are non-AI/ML that have been developed with a PCCP. Awesome.
So we have another question here. You mentioned PCCPs for larger known changes that would require a new premarket submission. So would bug fixes not be part of a PCCP, or updates for cybersecurity vulnerabilities found in a PMA device?
Alexander Chan: It's a good question. Yeah. I think the main thing I have here is really: if the existing flows, when you go through the FDA flowcharts...
Jake Stowe: Mhmm.
Alexander Chan: ...do not require a premarket submission for that change, then the reading of the guidance really does suggest that it likely doesn't fit within a PCCP. I'm not a full expert on that, but my understanding is that even large changes that wouldn't require a premarket submission should follow the existing guidances. And it really is things that would otherwise trigger a submission. And so if it did, then it feels like the latest guidance on general PCCPs, or general medical device updates, could fit that, even if it's not an AI/ML update itself.
Jake Stowe: Awesome. Thanks, Alex. We have another very interesting question from another anonymous attendee. If our software is initially designed to address one infectious disease, is it possible to structure the PCCP in a way that it can later be extended to cover additional infectious diseases without requiring a full new submission? In other words, can a PCCP be written broadly enough to encompass a class like infectious diseases, rather than being limited to a single condition?
Yeah. I think that's going to be very dependent on the disease area. I think there's probably a technical component of the review, of how much the model needs to change in order to encompass the new disease state. But I think that's probably challenging. That would be my guess. I personally haven't heard of it, because I think the agency would probably look at that as changing the intended use of the device. That's my ten-second analysis of that situation. Alex, I don't know what you think.
Alexander Chan: I don't know if I have more than that specifically, but this is something that I think we at Beacon are also interested in. Could we leverage this?
And I think when you look at that restriction to not changing the intended use, or in most cases the indications for use, it makes it difficult. There might be ways to structure it, but from what I understand, going from general to specific is often challenging on the medical device side. If you have a general indication or intended use, going to say that you specifically treat, or are focused on, particular disease states within that feels tricky, but there may be ways to do it. I don't specifically know.
In the guidance, there's this discussion of improving performance on particular subpopulations within your intended use. And so, this is more speculation, but there may be ways to leverage that to accomplish parts of what you're trying to get at. But I think a lot of it comes down to how you plan on marketing and labeling the particular device, and so it feels like it likely crosses into a gray zone here.
Jake Stowe: So I know we have more questions. We do still have some content to get through, so I think I'm going to pause on the question answering right now so that Alex and I can move forward. But if we have time at the end, we will pick up the questions again.
So, Alex, what role do PCCPs play in helping Beacon achieve its goal to release updates faster? What challenges remain?
Alexander Chan: Yeah. That's a great question. So I think PCCPs really allowed us to retrain our AI/ML algorithms without needing to go through new 510(k) submissions, which made our iteration and release cycle for the AI/ML updates, the retraining, really more predictable, since we didn't have to necessarily interact with this external regulatory body for these types of improvements. But while they help us streamline that piece, PCCPs don't directly address the need to be able to iterate quickly with respect to general software development, in terms of following modern agile practices and those types of things. And so many changes that we make fall outside of PCCPs. Right?
They're these minor improvements or bug fixes that don't need a new submission, and making sure that we could effectively and quickly document those changes is not something that's directly addressed by PCCPs.
Jake Stowe: Awesome. Alex, for the room, can you just tell everyone how often you release?
Alexander Chan: Yeah. So it depends on our products, but for many of our products, we try to follow GCP and GxP for a bunch of our software products. We release them every other week, following design controls for them, leveraging Ketryx for our design controls and documentation. For our AI/ML pieces, I aspire to be able to train our models at such a rapid cadence. I'm not there yet, but we'll get there eventually.
Jake Stowe: And how instrumental would you say Ketryx has been in helping you achieve that release cycle?
Alexander Chan: Yeah. I think Ketryx has been really important in allowing us to do that. It's a combination of Ketryx as well as some of our testing practices. So really leaning in, and I think some of this is on our next slide too.
Jake Stowe: Yeah. Yeah.
Alexander Chan: I think really leaning in on the automated testing pieces has been really important. And then leveraging Ketryx's ability to track those test executions and generate automatically generated documentation that's compliant. That's been really important for us to be able to move more quickly and maintain that iteration cycle. I think that cycle has been really important for us to be able to continuously maintain within this regulated environment. So we were really looking for solutions that wouldn't necessarily slow us down, while maintaining high quality software in a way that is compliant with the regulations.
Jake Stowe: Yeah. And I think this is something that we see across our customer base: it's a combination of tooling, processes, and training to get to a really rapid release cadence. And it's something that we're seeing across our customer base, that people who are able to get those three things in harmony are really able to increase the speed and the velocity at which they release. Another thought here, Alex: my impression from working with our customers is that there's some concern in the industry that shipping faster will lead to some degradation in quality. And my perspective from working with our customers now is that every time you improve your release cadence, it actually improves the quality of your product. I wonder if you have any thoughts about that.
Alexander Chan: Yeah. I think so. I think there's something about reducing the burden that comes with making sure documentation for these changes is as low overhead as possible. Because otherwise, you have this potential disincentive to continue to improve your products, which I think is incredibly important. And for us at Beacon, it's really, really important to be able to continue to address the improvements that we want to continue making, and fix the bugs that we find.
All these things are really important for us to be able to do. And moving more quickly really allows us to continue to deliver these types of products. I think this is why agile development really is a thing and is important in the software industry: it allows us to more quickly get to high quality software, and I think that's no different in the medical space. And just finding that magic combination of processes, tooling, and training and practices is really important.
Jake Stowe: Alex, I talk to people about this a lot, and I've never heard it framed quite like that, which is that if you have a really burdensome validation process, it can almost disincentivize you from shipping improvements. And I think it's a brilliant framing. It's a very good way of thinking about it.
So let's talk about why utilizing the PCCP is so hard and, specifically, what advice you would offer other companies considering PCCPs and faster release cycles.
Alexander Chan: Yeah. And so from our perspective, we've gone through the PCCP cycle twice now, in terms of getting this in two of our medical devices. I think the things we've learned here are really to define modifications very carefully. I think the FDA is really looking for modifications that are as specific as possible and as well bounded as possible. And so to get a successful PCCP, I think the more well bounded these modifications are, the better.
Jake Stowe: Mhmm.
Alexander Chan: The other piece that I think has served us well is explicitly describing the differences between the PCCP acceptance criteria and the validation plan within the PCCP, compared with the original device's validation study, or whatever it ended up being.
And I think in our case, being able to point to implementing all of the software verification and validation that we did for the original device as part of any future PCCP update was really important for us. And actually, in this case, I think Ketryx also helped us here, because it makes it quite easy. The automation plus Ketryx makes it very easy to be able to say we're going to repeat everything that we did for the original medical device for any PCCP update. Then specifically for AI/ML, I think it's really important: our FDA review was really looking for strong controls over mitigating risks of overfitting, or overestimation of performance. And so strong technical controls on the test or final validation datasets are really important, to make sure that there is no potential leakage. Even if you don't actually use that data in training, deriving any type of human insight from it that could lead to that dataset being overfit to some degree was something that they cared strongly about.
And then again, more generally on the software development side, heavily investing in automated testing across all the stages of software development, and then coupling that with the right tooling in terms of minimizing the compliance burden for generating documentation, was really important for us. And so those are some of our learnings on the PCCP side that I think we took from our first one, SleepStageML, to Waveband, and got through successfully.
Jake Stowe: Yeah. And just to speak briefly about this interaction between automation and tooling, I think one of the main things that our customers communicate to us, and one of the reasons that they buy us, is that it's clear that we're building software and systems that are sophisticated enough now that you can't just use one monolithic tool to manage the validation or the building of those products.
And so the approach that we take is we connect different systems together and we automatically ingest information from them. And so what Alex and team have been able to achieve is that all of their design controls are managed in specific subsidiary systems, and their CI/CD pipeline reports evidence directly into our product. So that's just to give everyone a little bit of context about the way that our product is being used to support their PCCP.
And then I think we're going to get pushed to close here, but maybe we'll take one more question, Alex, before we do that. We have a question from Amy here. Are PCCP submissions automatic, or do they still require review?
Alexander Chan: Let's see. Yeah. So I think the question is about the original PCCP itself: the plan requires authorization by the FDA. Right? So that has to be part of a premarket submission, and that document needs to be authorized by the FDA as part of your medical device. And once that's approved, it's part of your medical device. And you can execute that plan, as long as you're following the guidelines of the plan that the FDA approved. Every time you go through that loop of executing a PCCP, introducing a modification and validating it, that doesn't need to go to the FDA anymore.
Right? And so that is the thing that really allows you to iterate more quickly: if you know what you want to do and you can get the FDA to authorize that, then all the subsequent changes of that type that you want to make, you can easily make.
Jake Stowe: Awesome. Alex, I have to say, such an informative hour, and I really, really appreciate your time and really appreciate you as a customer as well. If anyone in the audience is interested in learning more about Beacon and how they used Ketryx to support their PCCP, you can go to this case study on our website and learn more about it.
For those of you who have questions that we didn't get to, I see our team taking those right now, and we'll definitely follow up with you. And then I will just put in here that our next webinar is on Thursday, July 10th at 11 AM, and we are going to discuss navigating the FDA's total product lifecycle framework for generative AI devices. Really appreciate the time. And Alex, I will see you soon.
Alexander Chan: Absolutely. Thank you, Jake, and thank you, everybody.
Jake Stowe: Thank you all. Bye.