Erez Kaminski: So welcome, everybody, to our first webinar of 2025: Navigating the FDA's Total Product Lifecycle (TPLC) Framework for Generative AI Devices, and AI devices, I think, overall. There's not a huge difference in many of the things that are going to be talked about today. My name is Erez Kaminski. I'm the founder and CEO of Ketryx. With me is Jenn Dixon. I'll let her introduce herself.
Jenn Dixon: Hi, everyone. I'm Jen. I was previously a quality and regulatory professional myself. I worked on software medical devices, and I have experience in quality management applications and AI in general.
We placed the link for our next webinar in the chat below. Please go ahead and register for that next one. It'll also be focused on software as a medical device and all that good stuff, and we're really excited to have you join us for the next one. A question we always get when we move into this webinar phase is: when are we going to get the recording and the slides? Are we going to get them sent afterwards?
You will be getting the slides at the end of this webinar. So sit back, relax, and enjoy the show. We'll also be looking for your questions. We want to stay engaged and get a sense of what you're looking to get answered, and we want to tailor this webinar to your needs. And lastly, because we apply ISO 13485, we are always looking for feedback and trying to make these webinars a lot better for you. So there'll be a very short survey sent out after this webinar where we'll be looking for your feedback.
Of course, if you have any questions at any point throughout the webinar, please use your Zoom controls and submit your questions. We have folks on the call who are looking to answer those questions, and of course we'll try to take those questions live as well if they're questions that everyone wants to know the answers to. Alrighty.
So everyone should have access to a poll. What we'd like to know today is: what are your concerns about integrating GenAI into medical devices? I'm going to give everyone about a minute to answer this question. There could be lots of things that are bringing you here today. We really want to understand what are the things that keep you awake at night. What brings cause for concern when it comes to generative AI?
I know lots of people have been integrating generative AI into their workflow. Sometimes they've been integrating it into their products. They've been keeping up with the news. So we'd just like to know your thoughts on what your concerns are, particularly maybe in medical devices. What are your concerns about integrating GenAI into your products? And I'll just give it a...
Erez Kaminski: When I was trying to do this for the first time, moving from tech to biotech and medtech, trying to figure out how I was going to integrate AI and GenAI (at the time it was more called NLP, and then this kind of token-prediction type generation), I started to think, wow, this is going to be really, really hard to do. The complexity of the systems that were developed in medtech at the time, almost ten years ago, was much lower than the complexity of these types of systems.
And these systems just have so many moving parts, so many software items, that I thought it'd be almost impossible to make a single validated version. And then trying to change that version would make it not worth it. Right? Updating it would become completely impossible. So I'm pretty interested to see how people responded. Maybe we're ready to share the poll, Jen?
Jenn Dixon: Yes. So I'll go ahead and share the results for this poll. Talking about what you just said: you spend all this time building this perfect device, this perfect design history file, only to feel like you can never update it ever again.
So taking a look at the poll, hopefully everyone can see the results. It seems like risk management and change management, as you just mentioned, Erez, are definitely top of mind for some people. You build this wonderful device and you document it, and you find that you just can't maintain it very well, and you can't update the documentation. I see traceability is also a concern. Tracing your requirements, your risk management file, your testing, building that trace matrix can be quite the challenge for some people.
And of course, risk management can be applied throughout the process, and this can be very challenging as well. Any thoughts there, Erez? Is this what you feel is the general consensus when you're reaching out to other medical device manufacturers?
Erez Kaminski: Yeah. I think a lot of our partners, when we work with them, talk about the need for a better way to do change management. The level of complexity of their traceability, or risk management and end-to-end risk management, is something that comes up a lot. There's this traceability issue where traceability matrices now have hundreds of thousands of lines for some products.
And that's just going to get worse and worse with this, because even for unregulated products that do AI, because of certain risks, I think people already have hundreds of risk controls, thousands of risk controls being tested. Now if you take that to the end of a true medical AI system that can cause substantial harm to people, I think that would cause an explosion of risk and traceability items. And then you layer on top of that the fact that the AI engine will actually be used as part of a hardware system that is automated and performs more actions. I think there's a real need for something to change in the way we build these types of validated products.
Jenn Dixon: A hundred percent agree. And that's why a total product lifecycle process is probably the best way to do this, and that's one of the things we want to talk about today. You'll see that GenAI is coming into medical devices today, not just into our workplaces. Dexcom, a major public company, makes lots of medical devices every year. They're integrating GenAI into things like insights for your glucose levels. How can you change the way that you live day to day to improve your health outcomes?
Whether this is for a wellness device or it's an add-on to an existing Class I or Class II device, you're seeing GenAI creep into low-risk, high-impact functionality. But what you're also seeing is that this might be setting a precedent for higher-risk applications. The way that these companies are building out total product lifecycle management, the way that they're doing it, is going to set the precedent for how we submit our devices and how we get them cleared with the FDA. Erez, do you think that GenAI is really different from the previous AIs that have come before?
Erez Kaminski: I think it's not fundamentally different. It's a difference of scale. There is more uncertainty in these large language models because you don't really know what they were trained on. Right? They're trained on massive amounts of data, like the entire Internet. And as a result, there's a lot of stuff on the Internet that we wouldn't want interacting with the patient.
I think that's one challenge. You have less and less visibility into your supply chain. I think another challenge is that it will be harder to validate them, because they are going to do many more things. Take some classic machine learning device. I think the first device approved by FDA, in the nineties, is some kind of multilinear forecasting model. It forecasts some biomarker or some endpoint, and then it allows you to assess, given other things, drugs or some other intervention, where that biomarker is going to be.
I think that's a really different problem than if you need to answer questions about your medical conditions in general, or do things like that. There, the use case is much more narrow. With generative AI, the use case is very, very broad. And as a result, I think it's going to be pretty hard to validate them. I think there's going to be a lot of validation compared to gold standards, which is already done in many other fields of machine learning and other fields of medicine, and also a lot of risk management. I think in generative AI models, risk management will play an oversized portion of the work.
Jenn Dixon: For sure. And I think you've mentioned before that GenAI is going to be big. It's going to be really big, particularly when it comes to making sure that we serve enough patients out there, and it expands our scalability. Right?
Erez Kaminski: Yeah. It's funny, because we were talking about it the other day as we were planning for this. There is an issue coming where the population is growing so fast compared to our ability to train providers and to make things for those providers. So we need to find a way to give people, in rural areas or not in rural areas, access to more and more medicine. And I do think it's a great technology for that. We just need to make sure we're not telling them the wrong things.
We already know that even in computer games or just chatbot applications, there is substantial harm being caused. Recently, last year I think, there was a lawsuit about a young boy who killed himself. It's very, very sad and tragic for the family. And now you're thinking: what happens when that recommendation is coming from a medical device, not just a game or a chatbot, and what would be the result of that? Because at the end of the day, in some ways, these generative AI models are just really, really good liars. They make you feel like they said the right thing, and it's very hard to understand if they did or didn't.
And we need to make sure that people are doing that in the right way, using all the modern techniques like RAG and so on, to make sure that you understand what's going on. And that's part of the risk management. Are you actually referencing the right retrieval when you generate?
Jenn Dixon: For sure. And in the medical device world, we apply something a little bit different than the gaming world. We apply a total product lifecycle approach. And especially if you're working in software, you should be really familiar with IEC 62304. Is that right? And I think, Erez, you have been thinking about this standard a lot over the past couple of years now, and you have a really firm mental model of how it should be applied in a total product lifecycle approach.
Erez Kaminski: Yeah. I think it started from the time I moved into biotech, thinking about how to do this without really knowing the terms. And then I met a gentleman who now works with us called Paul Jones, who led this kind of work at FDA for about 25 years. He and I sat together and thought: what does this really mean, and how would this be applied widely to AI systems and other automated systems? So if we go to the next slide, I think we can start talking about it. Here is an example of what a product lifecycle would look like.
One of the points I always like to highlight is the supply chain. This is very different from how it's going to look for a hardware device, because there are so many more components that are used as part of software. Right? Most software systems today that are web-connected, apps, cloud systems, AI systems, have thousands of supply chain items, what are called dependencies, or SOUP in 62304, that are used here. What we're going to do today is give a really brief overview of the different steps and phases here, and then tie them as best we can to IEC 62304 and the new TPLC guidance.
So if you think of just the process of product design: you decide to launch a project, you set a bunch of requirements, use case requirements, and you start thinking about how you're going to fulfill them. That's IEC 62304, Section 5. Understand the requirements. And of course this is part of broader standards like IEC 82304 and 60601 that deal with PEMS and all the different parts. But 62304 is a pretty good place to start if you're building a software-only medical device.
And then for AI, we'll talk about model description and how you develop models. Right? The requirements, system requirements, and use case requirements eventually translate into specifications. That's a model specification, for example, or data specifications. You need to verify and validate them, not just the software now, but the data and the model.
So there are many, many more steps here. Then you need to show some kind of equivalency or go through a full clinical trial, get into the supply chain aspect, and bring all this information in from external suppliers into the specific variants, which are the purple line here. We had a webinar last month that talks in more detail about these things. And then we have manufacturing, services, and ops. So manufacturing, CI/CD pipelines, audits, as much as we need to launch the product.
And eventually you launch particular variants of the product. Right? That has to do with different regions or different things of that nature. And then you basically get into this post-market surveillance step. Of course, I clicked slightly early. I forgot the risk management that's here, as part of product design.
And then at the end here, now that the product's on the market, you have this need to understand what people are complaining about and how to modify it. Another form of modification will come from the fact that maybe you'll discover cybersecurity issues in those dependencies, in that supply chain. Right? You have a cybersecurity risk, a vulnerability. You now need to understand that, analyze it, maybe change it, and then make a new version. And that's Section 6 of 62304. In this guidance, it's talked about as device performance and monitoring. So there's a lot of work to do around that. And then I think... yeah.
Jenn Dixon: I've loved the way that you mapped it to 62304. 62304, I think, is a standard that people love to hate, but it really does provide a firm structure around how you can validate software in all the different ways that people build software, whether you're doing agile or waterfall. And I really love this diagram as it steps through all the different processes. And that loop back is a reminder that this is a total product lifecycle approach. There is feedback for us to improve the product over time.
This whole product lifecycle approach is something the FDA has always talked about. They talked about it recently in their executive summary. So if we just hop on to the next slide here. They've always advocated for a TPLC approach, and there's no reason why it doesn't work for GenAI as well. So they're thinking that your TPLC, the way that you create software, might be what you need to show evidence that this generative AI product you're developing is safe and effective.
You can cite those procedures. You can cite your practices and say that this is how we are planning to not just create a safe device, but maintain it over time. With the total product lifecycle approach, there are a lot of steps that are very specific to GenAI, if you want to step forward another slide. With AI and GenAI, there are a lot of different steps here, and I'm going to step through each of them and talk about each step.
Planning and design, obviously, is the first one. You should know what your project goals are, the scope of your product, your requirements, your algorithms. You should know the possible ethics of gathering certain information. Set that up front. You definitely don't want to think about ethics two years, three years down the line. You want to think about that straight away. Next step.
And then we hop into data management and collection, if you just click one more time. Gathering your data is going to be very important. Data is key. So you need to organize your data. You need to quality-control your data. What you want to avoid is a situation where you've developed something in R&D, you've gathered these online datasets, and then you renamed everything and lost sight of where all this data came from and when you used it. Did we use it during training, or did we use it during tuning? Let's avoid also using it for validation. Otherwise, we're going to run into overfitting issues. Next.
Of course, you need to develop and refine your AI model over time. You will likely do internal training, informal testing and training that happens before the V&V cycle. This is your developers checking to see whether you've even started building something that is effective, before you hand it over to your quality engineers to make sure it actually meets all the requirements of the FDA. So that's hyperparameter tuning and optimization. When the FDA is thinking about a total product lifecycle approach and you say that you have an AI device, these are the questions they're going to ask. They're going to expect these buckets of work to be happening when it comes to AI-specific technologies. Next.
And of course, the big one, the one I think we all worry about all the time, which is verification and validation. This is where we end up spending a lot of time, a lot of money, and a lot of expertise to assess whether the models we've created are actually accurate. Sometimes they're not super reliable, and we need to really prove that they are. So you need to have firm validation strategies for how you're going to do that. Next.
And we're actually going to go through model deployment and operation and maintenance, because I think we have a question here around this. Obviously, when you're deploying the model, you should know up front whether you're going to be changing the model over time. Is this adapting in production? Maybe it's a personalized algorithm, so it may need to learn over time. You want to build that into your design, and part of your design should include what the trigger points are. What are the alarms for whether this model is potentially falling out of conformance, no longer in specification? So as you're deploying, you want to keep that in mind.
You also want to make sure, if your model is susceptible to performance issues (maybe it hallucinates more because of poor performance, or maybe there's not enough memory on whatever it's running on), that you consider that in deployment, because it will impact performance. With operation and maintenance, the next step, we also want to track real-time performance.
So we have a question here: "Not all deployment options allow easy access to field data." That is true. Sometimes we have requirements saying that we can't retain some of the results. There can't be a memory. We don't want to use it for retraining. There are no logs. "Further, there may be contrary risks from a data privacy and security perspective to collecting field data."
Erez Kaminski: Right. And it could also be an embedded device where you just don't have access to this information. Right? It just does things at the edge.
Jenn Dixon: Exactly. "And how will the FDA navigate post-market requirements, since obtaining additional training data may be limited?" The FDA is very aware of this constraint, and they don't want to step into the realm of telling individual patients that their data has to be sent back to the manufacturer. They don't want to take that away from you. They want you to feel like you own your data. So the FDA has been pretty nonprescriptive about what needs to happen here.
Obviously, they want you to show evidence that you're mitigating risk as far as possible, which may include sending some data back to the manufacturer for retraining. But really, this is up to you to make that case. They're not going to start saying, "Hey, you can expect this from patients." So I'd say the FDA is navigating it very cautiously. For now, they are not stating a hard requirement that you have to do this, but they might come to the assessment, based on reviewing your submission, that you should. So, as always with regulatory, the answer is: it depends.
Erez Kaminski: And I think it'll also evolve over time, as the agency responds to that and helps people. But I do think there are also a lot of companies who are thinking: in order to really do effective post-market at scale, I do want to log this information, deal with it, and monitor it, to understand what I have on the market and also what kind of interaction the stuff I have on the market has with patients.
Jenn Dixon: Exactly.
Erez Kaminski: Exactly. And that's also a part of it. It's not just a requirement of safety and effectiveness. It's also a commercial desire, because the more information you have, the better your competitive advantage will be, because everybody is going to use, and is using, the same types of foundation models. So it's always about the data.
Jenn Dixon: For sure. And what I would love to see is those really large manufacturers really set the standard for what that looks like, really set the best practices for what that post-market real-time feedback will look like. And for our last step, real-time feedback: we want to continuously assess the effectiveness of our model through KPIs. You should have a firm sense of what these are, and typically these might get included in your submission. So compare against some sort of baseline. Maybe you have comparative testing as part of your design history file as well.
Alrighty. So today we're going to focus on just two of these, if you'll go to the next slide. Yes. This is where I think GenAI becomes most challenging to implement, particularly if you're a quality and regulatory professional. Data collection and management might be something you aren't used to doing a lot of, maybe just a little bit, but not as much as you would for an AI model. And lastly, verification and validation. So let's dig into these a little bit more. Next slide.
Alrighty. So if your design developers are coming to you and saying, "Hey, we're going to build an AI model," the first questions you need to ask are: How are you acquiring the data? Where is it coming from? Are you changing the data before you feed it to the model? What format or protocol might be used? In this case, with generative AI, it could be prompts, it could be long strings of data, it could be JSON. You really want to know that up front as you are collecting your data.
Where are you storing it? Is S3 the right place to store it? Should it be somewhere else? Do you have the rights to store it in that location? What prevents someone from accidentally coming in and adding or changing things? These are not questions you want to be asking years down the line, after you've used this data.
Traceability: is there a clear version history of this imaging data or this prompt data? Maybe you changed it a little bit. Maybe you preprocessed it and it's part of that model design. We want to be very sure. From raw data to what you're feeding the model, how has it been changed? How has it been cleaned? And are you using it for training, for tuning, so not V&V yet but still that early informal testing in the design and development process, or are you using it for V&V? Because one of the first questions the FDA is going to ask you is: have you reused your data at all? Have you made sure that the AI model has never seen this data before?
Erez Kaminski: Right. That's the number one false step that I've seen happen again and again and again. People have information and data from their training set in their test set. I think another thing about data traceability is: does this data actually trace to the claims you made about the device and its intended use population? Right? That's a big, big challenge, given that most devices are usually deployed in coastal regions: San Francisco, Boston, New York. Does that really represent the patient population you're trying to serve and that you've made the claims against in the label?
Jenn Dixon: Exactly. Which is one of the reasons why the FDA is really highlighting data traceability, making it public in your 510(k) summary or whatever submission pathway you're going through. They see data and information about the AI model as empowering. Lastly, labeling and annotation.
If you're using this for a medical application, you're probably going to have labelers for the data that goes in. Those labelers, whether a clinician, an MRI technician, whoever you're trying to replicate in terms of intelligence, their competencies will shape the way your model performs. So they are absolutely essential. Understanding their competencies is essential to understanding how effective your model will be. So the FDA will want to take a look at that as well.
Erez Kaminski: Yeah. And I think how the data is annotated matters too. I remember, I want to say it was eight or nine years ago, there was a really big problem in ImageNet, which is a big dataset of images that is used all around the world to train a lot of models. People found that in ImageNet there were categories, because it's open source, that were very racist and very wrong as well, completely just wrong. People had injected wrong information, kind of poisoned the data.
And there was a lot of effort from the open source community to clean that up. And for some of those things that I was aware of at the time and remember learning about, I'm very, very happy they're not making it into a medical device.
Jenn Dixon: Exactly. And you want to be able to show up to the FDA and say, "We are a hundred percent sure that none of that has ever happened with our data, and this will be a safe and effective device."
Erez Kaminski: And I think with foundation models that read the whole Internet, you'll never really know. But what you will know is that you've done the best you can and have created safeguards along the way.
Jenn Dixon: Exactly. Verification and validation. That's how we're going to address those foundation models, with our testing. One of the things the FDA will want to know is: how have you evaluated this model? Can you tie it to the intended use, as you were saying, Erez? Can we show evidence that this is really meeting the intended use, and that you've taken out as much as possible of any potential risks from the foundation model?
As you hand this model to your clinicians, are they able to interpret the results? Are patients able to interpret the results? Obviously, if a clinician finds the output of a GenAI prompt is completely hallucinatory, they're not going to rely on it. They're not going to want to use it. And it's really important that these models are not distracting. They're actually empowering, and they're going to elevate the amount of care and the care quality that you're going to receive.
I think we have a question from our Q&A section: "For medical use of GenAI, if the model is validated only for one or a few specific use cases, what would be the effective risk control measures that can be used to prevent the model from being used for other use cases once it's released into the market?"
So it sounds like what you're asking is how you make sure that you've validated this and protected against potential off-label use, or potential foreseeable misuse. This is something that you definitely want to embed into your risk management file and have somebody addressing that potential use. And it's really important that you build risk control measures around that. Now, of course, this is very contextual to your specific use case.
I would at least say that if you're going to have GenAI, you should try to implement quality control metrics. Some sort of stopgaps, or safety brakes, that would assess whether the GenAI is being used for the wrong purpose, and you can flag that to the end user, label it for them, and say, "Hey, you might be using this for a purpose that it's not been validated for. Please be careful," or maybe, "It's not recommended that you use this." Those are just some potential ideas of how you could do it. Of course, as always, it depends on how you want to embed it into your risk management file. Applying ISO 14971, of course, is the way to do this.
Erez Kaminski: And I actually see one question here that we do want to take right now: "Does FDA accept pre-trained models, for example ResNet based on the ImageNet dataset?" Of course. The FDA has accepted many devices that leverage existing datasets and existing pre-trained models, what are called foundation models today, as part of devices. You can't really develop most AI systems without having a lot of access to pre-trained models.
Jenn Dixon: Exactly. And they're not about adding an undue burden to the way that you design and develop devices. Using this foundation model, this pre-trained model, might actually make it go faster or better. And they cite that a lot in the premarket submission guidance. Alright. Coming back... Go ahead.
Erez Kaminski: I think one of the things to say about the verification and validation and all the traceability of that is that it would depend a lot on the particular case. For example, for this question about foreseeable misuse and off-label use, sometimes you will need to put in controls. Let's say the AI feels that you're asking it a question that's very far from what it's good at. It should say that: "I don't know how to answer that. That's the wrong use." I think there should be warnings in the user interface and things of that nature.
And overall, this is going to be an evolving field that is going to depend on the product, the company, the scale at which they operate, the level of scrutiny they operate under, and the risk of the product, of course. And I can tell you that even for low-risk or non-risk use cases, consumer applications like Anthropic and OpenAI, if you listen to a bunch of their talks and know some people who work there, you realize that they have massive traceability databases for risks, with hundreds and hundreds of tests, possibly thousands now. They're trying to address these different questions and make sure that they're not doing something that's totally off, including human testing, UAT, for prerelease candidates.
Jenn Dixon: I love all this talk about risk. Risk is super important to how we develop medical device products, and we'll touch upon that in a second. Just wrapping up this verification and validation piece over here: one thing the FDA has mentioned is that you potentially want to do both standalone validation of your model alone and validation of the human-device team. This is really key. This is where you're going to show, "Hey, when you give our device to a human, the combined performance is better." That's something you can definitely showcase. This is the benefit that you're introducing with your technology.
And of course, you want to document all of this. You never want to miss out on the opportunity to document your V&V. Every single detail counts, and it might be something that comes up more in the submission process. So make sure that you have a strong quality management system that makes sure you're meeting all the requirements around it, and of course, that you're connecting it to risk management. You should see risk management in your V&V cycle as a consideration as well. And Erez, I know you love to talk about risk management, I think in large part because you've been contributing to risk management, particularly for ML applications. Isn't that right?
Erez Kaminski: Yeah. Thanks for mentioning that, Jen. I was actually working on a technical report that supersedes the CR34971 the FDA mentioned in their guidance, called TIR34971, Application of ISO 14971 to Machine Learning in Artificial Intelligence. And I think it's just very challenging. Right? It's mostly that you're doing the same things. There are just many more risks that we're not as aware of, similar to the first time an engineer walks into a medical device or pharmaceutical environment, or any kind of high-risk environment, and tries to understand how to do all this risk management. Now there's this next step.
There are many, many risks that we are not used to. There are not great tools to manage that risk. And those risks will involve a substantial amount of testing, human review, and all kinds of trials to make sure that we are managing them in a way that's appropriate. So that's kind of why we wrote that technical report. I'd say that it's mostly a guide to many different ideas and a way to think about it. I think the previous one from 2022 that FDA mentioned, CR34971, is also a great one for that.
It's trying to be a database for professionals in the industry to start reading and understand: okay, here are the types of risks that people think of with these devices, and how do we now create procedures, tools, systems, and teams to address those things systematically? Because they're just going to come up over and over and over again.
Jenn Dixon: Yes. And I love how much information there is now about how to actually produce medical devices with ML and AI. I remember just a couple of years ago, there was not nearly this much information, or even concern from the FDA. If you look back in time, 2019 is when one of the first guidances came out about AI and ML for SaMDs. And a lot of that was very aspirational. It was like, yes, you should be doing all the things you were doing before, but just consider some specific things.
And now we're all the way to 2025. Just two weeks ago, the FDA put out the most prescriptive, descriptive way to actually submit these AI/ML devices. We've really come a long way. We're getting to the point now where there shouldn't be any questions anymore about what you should be doing and how you need to ensure the safety and effectiveness of this device. The FDA has done a really fantastic job engaging the community, bringing experts into the FDA, and consulting with industry to understand what the best practices are out there, and what they can tell medical device manufacturers, whether you're a startup or a large company, about what you need to do to get this out there.
So it's been really fantastic reading through all these guidances. Of course, the PCCP guidance was a big one. If you're a regulatory affairs professional or you're a designer and you haven't read the January guidance yet, we'll have a lot more information on the next slide, I think, about this. But it really is a fantastic guidance, because it really tells you what you need to submit and which section of your submission it needs to go in. So a couple of highlights. Surprise, surprise: implementing a robust QMS can be great evidence to show that you are building a device that is safe and effective. So implementing that...
Erez Kaminski: Love how they always position that, Jen. It's a great idea to do it. It is not a recommendation. It is in the statute, but they always recommend it again in the guidance.
Jenn Dixon: Yes. And you might be submitting a 510(k), where it might not be required to submit your QMS documents the way it is with a PMA. But go ahead and do it anyway, if it's what you need to really show evidence that you have a safe and effective process to update and maintain this model.
Documenting your data management and your model performance in your submissions: we just talked about how data management is super important. Everything from the raw data to how you preprocess it, those are all things you want to trace and keep track of, and that's something you'll want to include in your submission, obviously in the software section. And of course, you want to have a clear understanding of model performance, whether it's version one of the model, version five, or your production-equivalent model. They want to see all of that.
One thing they make extremely clear is that if you are using AI in your device, you should disclose it, and disclose it in a very public way. Previously, manufacturers didn't necessarily have to do that. If it wasn't very important or subject to the submission, it might not have been the most important thing to disclose in the public summary, but now you do. And this is good, because it means that we all know where the AI is. It's not a surprise, and we can keep that in mind as we use the products and see them performing in the field.
One somewhat surprising thing they put in this guidance is requiring more user interface details. We've always had to submit user interface details and labeling requirements in our submissions, but they really highlight it as a form of context gathering. They understand that AI has applications in many different ways, and one of the best ways for them to assess whether it's implemented in a good way, whether it aligns well with the intended use, is to get your UI described. And a picture is worth a thousand words. They really strongly recommend graphical representations of this. So if you have a really great UI for AI, they want to see it.
Lastly, and I think, Erez, you might have talked about this earlier: cyber threats. There are AI cyber threats. We haven't seen a lot of them or heard much about them, but when you have the intersection of a very adversarial field like cybersecurity with AI, which is a very unknown field, crazy things can happen. Right?
Erez Kaminski: Yeah. And I think there are all kinds of ways people hack AI systems to get them to tell you stuff that they shouldn't tell you. Right? That's one issue here when you're retraining. That's, I think, a big concern. Or all kinds of ways to try and hack, to get into the database behind the AI, or make it do all kinds of wrong things. So we're just in a modern generation of this, the same way it was with infrastructure. When infrastructure came up and became really, really widespread and robust, people found new ways to hack that. And now there are going to be new ways to hack these things. So the FDA is definitely asking about that.
Jenn Dixon: And of course, they want to see how you're going to maintain it over time. If you have PCCPs, make sure you implement them. Make sure you have post-market surveillance planned out. One of the key takeaways here, I think, in the executive summary for GenAI is that if you have GenAI, it may be a major point in your submission. So if you have an existing product and you're going to add GenAI to it, and I think a lot of people are considering this right now, it may require resubmission, and that's something you should definitely assess. And of course, a risk-based total product lifecycle approach is a good way to support a case of safety and effectiveness.
Erez Kaminski: Yeah. And I just want to say, before we go on, that there's also a blog post by you today, Jen, I think, about the new guidance and what it means for folks, a much more detailed overview in some ways than this webinar. So I welcome folks to go read it. Reach out to us if you want to talk more, and we're going to do a whole series just on that draft in a few weeks, to talk even more about this.
Jenn Dixon: Yes. I'm very excited to see what the latest submissions are going to look like, and very excited to engage on that. Lastly, as we wrap up GenAI: one of the key takeaways from this executive summary is that they're very well aware. Hallucinations are a major issue when it comes to generative AI, and they're going to introduce a lot of uncertainty into your device, largely because you're introducing foundation models. Right?
You don't technically have a clear history of how this model was developed, but you're going to use it for this new application for the benefit of patients. So these foundation models might warrant more focus from our regulators. And this all ties into your quality management system and how you balance that supply chain uncertainty. This unknown training data, this unknown design history file, you want to balance out with SOUP and OTS validation. Effectively, you're just going to make sure that you test it more if there is more uncertainty introduced into your supply chain.
So you want a very comprehensive V&V strategy that addresses all these potential risks, such as hallucinations. Of course, model drift is also something you need to consider. These are all things you should be embedding into your verification and validation process and into your V-model. And I think, Erez, you have a little bit of a demonstration for us, walking us through the V-model and how you can potentially do this for generative AI.
Erez Kaminski: Yeah. So we're just going to see how this is done. And thanks, Jen, for that wonderful overview. I think it's funny that a month ago, we were sitting and having a webinar talking about AI TPLC. Then two weeks later, this guidance came out. We wanted to do this webinar about GenAI in general, and it merged immediately with the previous webinar, very synergistically. So that's pretty fun.
So thank you for that wonderful overview of GenAI, GenAI use in medicine, the new guidance touching on some points, and the TPLC. And again, our last webinar is a much deeper dive into the TPLC, and it's going to be the beginning of many webinars about different aspects of the TPLC, as well as about Ketryx, which, as far as I know, is the world's first and only total product lifecycle management platform. We allow people to operate across different systems in their total product lifecycle, execute that, and generate evidence automatically as a result of work, with a lot of automation, a lot of AI, and a lot of other wonderful tools.
And again, it depends on the level of risk you want to take. You can turn those things on or off. You can have different grades of risk. You don't have to use our AI. We can custom-build your own AI for your company, and things of that nature. So today, I want to talk a little bit about how the TPLC looks, and then I want to do a demonstration and show you how to do that in the next ten minutes.
So it's a fast demo of how to actually do this in practice, and how we've helped companies, from Fortune 500s to startups, do this and develop and release software on a monthly, weekly, daily basis. And we're pretty excited about it. We just hit 20 million patients who are impacted by products we work on or our partners work on, and that we help them deliver to market. With this combination of the TPLC approach that we've been talking about for many, many years and trying to promote, and the PCCP, I think medical device manufacturers, and also people developing GxP software, finally have the toolset they need to go out and work as if they work in a tech company, and do this really, really fast and effectively, but still with the level of safety, risk management, and validation that we expect in our medical devices.
So going back to this diagram, which we talk about all the time now: how do you start thinking about the total product lifecycle and doing different sections of it? Right? There are many ways to do product design. Frankly, in an average machine learning tool chain, you probably have ten to twenty products, even in a small company, to do just this part. Then there are many other tools to do the other parts. What we want to do is sit across those tools and help you generate the evidence in the middle.
So if we think of just the pre- and post-market of the AI lifecycle, so not just generative AI but AI overall, though also generative AI, you can see this diagram on the left. You have some AI application. It's ready. It makes predictions, and we monitor it. And as it makes predictions, it generates new data. Right? So we now have new data, because it's making more and more predictions.
We then want to train on that data. We want to split it into a test set and a training set. We then use the training set to train a new model, every day, every week, every month, every quarter, whatever works for your company and your particular application in deployment mode. We do know that a lot of companies want to move faster and faster. And I think two weeks is becoming the standard for leaders in medical devices now.
And then we want to start testing it and seeing if it works well. So we test on the dataset, we see if that works, and then we evaluate to see if it's passed our metrics, which are really the PCCP metrics that people talk about often. After that evaluation, we need to ask: is this model really ready for the next step? If it does pass those metrics, we replace it in production, and then we go through this over and over and over again.
And there's all kinds of evidence that folks expect you to have. For example, premarket evidence for transparency. They want to see: do you know what your training datasets are? Do you understand your model architecture? They then want to see robust performance metrics. How are you going to measure this? Right? And those performance metrics, by the way, are the same metrics that you can use in your PCCP. Right? Those critical quality attributes of how you evaluate whether a PCCP change went well.
And then during post-market, after you launch the device, they want to see evidence of how you continuously evaluate that it works, that you don't have drift, that you understand what's going on and how patients are using it. And of course, you want to have strategies for managing regional biases. This is something that really pains me, because it is well known now that a model that works in one hospital doesn't necessarily work in another hospital, for many different reasons: population, types of devices used, the way physicians and other providers practice in that hospital. In the same way, it's known that a model used to do something in a drug manufacturing facility or medical device facility works for one line but doesn't necessarily work the same way for another line making the exact same product.
It's because they're so tied to what's going on and the specifics of their training. The FDA wants to see that you understand that as well: that when you're serving patients in Kenya, or in Japan, or in Europe, or in Chicago, you understand that challenge.
If we break down that lifecycle a little bit further for generative AI devices, you'll see that in this case, instead of just training data and training your own model, you have human-led generated data, reinforcement learning from human feedback data. You have a foundation model that you're consuming as part of your app. You still want to fine-tune that foundation model with your data and evaluate whether it works well or not.
And that's really, in many ways, a V-model. Right? I have some system requirements I want to meet, whether that's my patient population or my performance or the things I'm trying to have it do. I then have a dataset that has certain specifications, again tied to the system requirements, the population, what I'm trying to do with it, the disease area. Then I have a specific model I've created that has certain performance characteristics, and then I train the model, and of course verify that the model is correct.
And again, in machine learning it's confusing, because in machine learning they use the term validation for this. In medical devices, it would be called verification, but it's all basically the same. You test and you see that it's meeting a certain standard. It's not like design validation. Model validation doesn't mean testing it in the intended use environment. You also want to verify that the data is correct, of course, and then you want to validate that the entire system, in the intended use environment, conforms and can consistently meet your use cases.
And then you want to do all of that again and again and again, very, very fast. So you deploy it, you have an application, online prediction, and around and around we go. For reference, at most AI startups, I'd say this is something that's done probably every night, if not every week. In many large companies, like, I don't know, Netflix, this is something that's done every hour, for thousands of applications. All the time, they keep updating. CI/CD. We have a whole webinar about that from last year. That's great.
And now let's talk about the demonstration. What we're going to do today is see how this is actually done in practice, for Fortune 500 companies, startups, and growth-stage companies, by just making a device together today. We're going to have this Llama model. This is a Meta model, I believe, a generative AI model that people use. We're going to fine-tune that foundation model. We're not going to do everything, but we're going to do, I'd say, the QMS part of things.
And we're going to incorporate that within another device. We're going to show how it's part of a health insights platform. The health insights platform, the system of systems, is where you have the system requirements, the risks, test cases, and test executions for the top of the V-model. And then there will be specific risks, test cases, test executions, and design inputs for the bottom of the V-model, for this particular... you can think about it as a software item or a subsystem overall.
You're then going to be able to version the model independently. I know there was a question earlier about how you update. That's what we're going to show here. How do you version this independently, keep moving, but keep having a system that's updated and maybe deployed to different regions at different times? There will be different types of engineering controls applied to each model.
Engineering controls are ways to enforce compliance and assure quality through either physical or cyber-physical means. In a pharmaceutical factory, it might be the size of the tube that takes some drug product from one place to another, which limits the flow rate, the combination of that and the pump. In our case, in computer land, that would be all kinds of validated controls that prevent you from doing things. One of my favorite ones that we do is that we can connect into your CI/CD pipeline (people love this, especially at scaled companies) and prevent you from releasing software that does not have all of its risk controls mitigated or verified. And I think that's pretty exciting.
So let's jump in and see how this is done. We're going to go into Ketryx. You can try this as well at app.ketryx.com. Maybe someone can pop a link into the chat in case folks want to try this.
In here, you can see that in Ketryx you have projects. You can create different projects through this screen and try different things: create, connect to Jira, select the V-model, connect to other systems. Many other systems can now be connected to Ketryx. What we're going to start with is going into the subsystem. Just as a reminder, there's the subsystem, this medical Llama model, and then there's the top-level product, where the intended use is the actual product system, this health insights platform that is consuming the subsystem.
And you can see that producing v1.1 of this subsystem is in fact a milestone that is required to release version 2 of this particular system. So let's go into this and talk a little bit about what we see here. We see on the left all the different modules of Ketryx, from SBOM to risk management, risk control, traceability, graphing, and all items. In the All Items screen, we can see, per 62304, all the configuration items of the system. We can just go into one of them, like AIM-7, and we see that we're now going into Jira, the system of work where people actually work.
In Jira, Ketryx provides you with traceability, means of approval (there's a Part 11 compliant panel here), and full audit history, including a very nice traceability widget that can tell you what is traced to what and how things are connected. This particular requirement is a child of multiple other requirements. It's actually a risk control as well, for a risk. We can see that here. It is fulfilled by certain specifications and tested by others. Some of them are, again, in other systems or in Ketryx.
As you make modifications to this item, all of that is tracked in Ketryx. We can see here the entire version history and an ALCOA++ audit trail. We can change versions, update versions, lock versions, and then we can see all the different modifications that ever happened, all the different items that are generated based on this, and so on and so forth.
We can then look at many, many different items all at once. If we look at the All Items screen, we can see here that we have tons of different items, some of them coming from Git. We're going to explore that in a moment. Some of them integrate with Jira and come from Jira, like we just saw. And we can ask all kinds of questions, like: show me the difference between version 1 and version 2. Show me all new items, all removed items, all the same items. We see that there are not a lot of changes between version 1 and version 1.1. That's because we're just trying to show how this works. Show me all the risk controls. Show me all items that are missing tests. Well, all of them have tests.
Jenn Dixon: These are all lifecycle items. Right? This is all from IEC 62304?
Erez Kaminski: Yeah. Exactly. And it can accommodate your particular quality management system and lifecycle process, but you see you have requirements, software items, risks, test cases, and so on and so forth here. Then we can go into the traceability screen to see how the traceability works. You can see here that we have certain design inputs, in this case AIM-7, transparency and auditability. That leads to many different specifications, or design outputs. For example, we're using a base model called Llama 3 8B, eight billion parameters, and we can see and get all the different aspects of that.
And now we can test and make sure that this is the right one and that it does what we want. We can also use this to store information, so we know exactly where it's referenced and how it's all connected, to make this very uniform and cohesive, make it easy for people to understand what's going on in an audit, and move very, very fast. I think this is super exciting. This is the only way to really make massive AI systems and complex surgical robots, and then release them at the frequency that patients, providers, and our industry expect. And it works, which I think is a very exciting thing to see over the five years we've been building the system.
You can see here that we can go from here to see the different gaps in traceability. In this case, we don't have any traceability gaps. But if we had traceability gaps, even with a hundred thousand rows, it would still tell you what the gaps are. It would allow you to modify things and understand how things are connected, for example what design inputs have to do with the dataset specification, to continue to explore this. We also see that we have a lot of not just manual tests but automated tests that come from Git, or from other parts of my CI/CD, so I can run my tests. As they run, they create evidence of working, so you can run them every day as you release and keep working.
We can see here the different risks we have. In this case, we only wanted to make one risk, to keep it really simple. So we have one risk. That risk is about model hallucinations leading to inaccurate and harmful insights. We see that it's an unacceptable risk because of the way we configured our risk management matrix, which can all be configured to a particular organization's needs. We then have different risk control measures. Some of them are requirements, like we see here. Others are specifications, like we see here. Fine-tuning the model is a risk control measure here. And with that, the risk does become acceptable, so we're able to release. And if the risk is unacceptable, as an engineering control we can require you to have a benefit-risk analysis, which is exactly what 14971 requires.
We can then see here a list of all the risk control measures across my system. Again, it's a relatively short list, because this is a demo project. And then a really cool control, and someone asked about CI/CD, is that you can actually integrate the testing of this into whatever CI/CD tool you use: Jenkins, GitHub Actions, GitLab, CircleCI, and many others. You can read more about that in our documentation, under API. Maybe someone can share our documentation in the chat.
So we go through all of this, and then we want to understand how to release a project. Let's do that in the last minute we have. If you go into the health insights platform, you'll see that it is configured slightly differently. It's more of a top-level project that starts with user needs, system requirements, and subsystem requirements. We have really only one user need here, AHA-110. We have another system requirement that is actually a risk control. That's why it doesn't have a user need, and all the downstream effects of that.
We can also see it as a graph if we want. We can see all the different requirements from the system, like the one we just talked about, and how it interacts with the subsystem and the different aspects of the subsystem that we just saw in this traceability matrix.
And then we can go and release. We see here we have one version being actively monitored from a post-market surveillance standpoint. I can see version 2 and what needs to happen. I see that there's basically a bunch of milestones I need to work on, all kinds of controls and items that need to be done, and a release checklist that helps me understand what I need to do next and how people in my company can operate while they're operating in so many systems across the TPLC. And at the end of the day, I see that what I'm missing is an approval of the underlying project, that Llama project.
So if I go back there and just approve that project... let's go here. Let's jump into this version 1.1. I see it's ready to be approved. I reduced a lot of controls here, so the milestones are not required, just to make it easier to demo. I approve this. Part 11 compliant biometric approval, and soon, confetti. Confetti pops. We did a release. It's very exciting and wonderful. And now we can go back into this project and see that it's been updated.
As we can see, this has changed. It's now ready to be approved, because the underlying product version has been approved. We approve this, and then we can release our system. Now, all of these can block your release if you want. We're just trying to make the demo a little easier. And then we approve this version.
Before I approve it, I just want to show you that 100% of my design controls are done. 100% of my test executions have been run. I can see that I didn't modify anything, because again, it's a demo version. Changed vulnerabilities, change impact analysis, if I had one. Risk controls that are being tested and passed. All kinds of controls we can do here. I approve this, and we are good to go.
We just released a complex medical device system. We showed that we went from Jira to Git and back. And by the way, we generated all the needed documentation as part of that. For example, from this TPLC guidance, our model card, our bias analysis report, and many other things like a testing report, our problem report, and our system specification. I'll just show this as an example of how this could look. This is the typical Ketryx configuration. You're welcome to use it. Others can use their own templates, and that's kind of the whole thing. Thank you, everybody, for your time today.