Jenn Dixon: Hi, everyone. Welcome, welcome. Thank you so much for joining me today. I'm gonna go ahead and share my screen and get this party started. Hopefully all of y'all can see my screen now.
So welcome to today's session on how to use agentic AI to systematically prevent deviations. I'm Jenn Dixon, client operations manager here at Ketryx, and I'll be walking you through how agentic AI can help QA and RA teams maintain control as AI becomes part of everyday development. We'll talk about how AI agents can operate inside your existing tools, Jira, GitHub, Azure DevOps, enforce the processes automatically, and prevent deviations before they occur. Now, a little bit about me before we... maybe I'll jump into this later. I've done this role myself, very much manually, enforcing your QMS within Jira and GitHub, and it's definitely a time-consuming task. I'm really happy to present that we can do a lot of it now automatically.
This session will cover best practices, use real-life examples, and lessons learned from teams that have already embedded these tools, these agents, into regulated workflows. Alrighty. So let's go ahead and get started.
So before we dive into the content, I want to introduce Ketryx. You're probably wondering what Ketryx is. Ketryx is an AI-native compliance platform, purpose-built for life sciences teams to deliver products faster. We interoperate your tooling across various teams across your organization and apply your QMS rules across it. So that's something I definitely had to do manually across multiple systems.
With that data, we can also automatically generate the compliance evidence and documentation you need. So whether that's a 510(k) or an IND, we're here to document that and generate that deterministically or with agents. And with our AI agents, we can automatically trace your requirements, assess change impacts, validate test coverage, and also flag compliance risks in real time. For quality teams, this means less manual chasing of evidence and more confidence in audit or submission readiness. And for R&D teams, the benefit is more time spent coding or working in your developer-native tools, and not necessarily doing your manual documentation.
So I'll briefly introduce myself and my co-host Bailey, who I'm very excited to do this webinar with. So I come from Canada. I did my bachelor's in engineering, and I've also been a licensed engineer in Ontario for about a year now. I started my career at Synaptive Medical, where I was working in manufacturing quality, and that was a really great experience to understand the physical analog for a lot of the practices we do day to day. I then moved on to become a director of quality and regulatory at Omniscient Neurotechnology. Very different: it's a SaMD company, so that was very interesting, to apply all of your ISO 13485 principles but for a software-only environment.
And now I am a client operations manager here at Ketryx. Internally at Ketryx, I'm the quality and regulatory person for the entire client operations team. So whenever there's an issue around quality or regulatory, or new standards that come out, I'm here to be a voice within the company and make sure that we're following what the regulators expect of us and our clients. And Bailey, would you like to introduce yourself?
Bailey Canter: Absolutely. Thanks so much, Jen, and hey, everyone. So excited to be here today. I'm Bailey Canter, our director of solutions. My background was originally in physics at Chapel Hill, and then I went on to be a systems engineer at Raytheon in the defense space. After that, I was a back-end software developer as well as a scrum master in the healthcare space.
So my technical background, combining regulatory requirements written for defense with implementing them directly in the source code, showed me how challenging it is to be agile in a regulated industry. And so I'm so excited to talk with you all more today about how you can use agentic workflows as well and embed them directly into your SDLC. Thank you.
Jenn Dixon: Thanks, Bailey. One thing I love about working with Bailey, and we've worked together now for several months, is just how good she is at listening to all of our clients' problems. She can work with any client from any industry, and she understands exactly what they're going through because she really understands those pain points.
So I'm sure everyone at this point has heard a lot of the hype around AI agents. Let's quickly jump into what agentic AI is, how it works, and set the stage for why it matters. So across life sciences, AI agents are already reshaping how teams operate. Certainly, even outside of life sciences, we've definitely gotten prompts and outputs from various GPTs or whatnot, messaging apps and things like that. But in life sciences, there's even more disruption because of all the automation that needs to happen.
So analysts estimate that these systems could unlock up to $240 billion worth annually through faster time to market and automation. I'm sure if you've had drugs you're waiting for, or technologies you're waiting for, particularly in the life sciences industries, it tends to move quite slowly, and automation could unlock a lot of additional value just by getting the product to market faster. About 55% of today's workforce hours could be automated by AI agents, freeing up teams from those repetitive tasks to focus more on the critical tasks where you need a lot of accountability.
We're already seeing measurable results. So one major insurer achieved a 90% reduction in documentation time using AI agents. Imagine insurance, finance, even law firms are all using AI agents. There's a lot of documentation to go through. This 90% figure reflects what many of our customers, including Medtronic, HeartFlow, and DeepHealth, are seeing in our platform.
So imagine that same kind of efficiency applied to validation, traceability, or complaint processing. You know all that post-market data that's available out there? There's so much of it. Why don't we get the robots to do some of that work for us when it's really repetitive? So my question now is, how do we bring that potential into a regulated environment without losing control, right?
That's something that we're always concerned about. We want to demonstrate control. We want to show that we're meeting the regulators' expectations. And that's where AI agents can really solve this issue. So we just looked at how AI agents are reshaping life sciences as a whole, and this is what the transformation is inside the medical device environment.
Imagine AI helping teams generate design controls or code automatically. So think about the breakdown of a high-level idea into individual tasks. You might have someone who's a code specialist, a trace specialist. Well, now we can have agents who are instead that specialist, so it can surface relevant information instantly with natural language. That's something we know that agents are good at.
We can summarize dependencies and even monitor projects continuously for coverage gaps or redundant work. These are the types of real-life, practical use cases, and you'll see several of them on the screen right now, that are already changing how products are built. So faster iterations, fewer errors, and stronger traceability throughout. Now, Bailey, I don't know about you, but certainly in the earlier roles in my career, a lot of my work was around just reporting what had happened in the past week, right? Or summarizing things like, here's all the new requirements.
And though it was a good learning experience, I know that it was very hard for me to take on more critical work or more interesting work because I was kind of bogged down with all this repetitive stuff. Did you encounter that as well, Bailey?
Bailey Canter: Yeah, exactly. I remember filtering through hundreds of requirements. We actually used DOORS, which is another tool in itself, and having to Command-F to find a requirement, or go talk to an SME who was an expert. I had really no way to bring all of that relevant context to the forefront and centralize it for me.
And so some of the features you're showing on your screen are actually how you can build those agentic workflows in AI, not just in one tool, not just in GitHub Copilot, but throughout your entire SDLC, aiming for compliance and regulatory completeness.
Jenn Dixon: Yeah, for sure. And I know when I talk to folks about it, people are definitely worried: could it replace human judgment? I certainly think that there are ways to use it correctly, like any tool, where you're not offloading that accountability, right? You're actually just utilizing the power that it has. So again...
Bailey Canter: Exactly. Yeah. And it's keeping that human in the loop too, and that's key here. And we'll definitely show that throughout our time here today.
Jenn Dixon: Absolutely. So let's give teams the intelligence and visibility that they need. I certainly had to do that in the past. It was just part of my daily chores, and now I feel like it's part of my working persona, my professional experience, my professional benefit. So let's quickly talk about the agenda today.
AI agents are exactly what I'll explore. And Bailey's actually gonna demonstrate some of the agents that we have in house and how they're making our day-to-day workflows better. Again, we use Ketryx to build Ketryx, and it's certainly benefiting a lot of our customers. So first we'll look at the challenges that come up when AI operates inside a regulated workflow, where every output needs to be validated, traceable, and explainable. Then we'll cover strategies for maintaining control, including how to combine generative and deterministic AI, and I'll get into the details later about what that difference is: both creativity as well as reliability.
Then we'll go through a five-step framework for implementing AI agents in a risk-based, compliant way. You might be building your own tool internally, where you might have agents integrated into your workflow, so we'd recommend you pay attention to that part if you are. And finally, we'll close with real-world case studies and some short demonstrations of AI agents today. Folks, we'd like to engage the wider team here, the attendees.
So if there's anything you'd like to see, please let us know, because Bailey is ready to show you exactly what we have in store or what we can achieve with it. Alrighty. So let's start off with a poll here, a little pulse check. As your organization begins adopting AI, maintaining alignment with your quality management system can mean different things. So from design controls and validation to change control and oversight, this poll will help us understand where you expect the most effort to go as AI becomes part of your workflow.
So go ahead and select all the areas that apply. We'll take a look at the results in just a moment. So I see it's twelve minutes past the hour. I'm just gonna go ahead and say at thirteen we'll review the results. So where do you expect the most effort to maintain QMS alignment as your organization adopts AI?
Bailey, I'm not sure if I recall. Did you implement any tooling where you kind of had a series of butting heads with maybe quality or regulatory teams, where you really had this wonderful tool and then suddenly no one wanted it, in a way, but they wanted the results? Right? Did that ever happen to you?
Bailey Canter: Yeah, absolutely. I mean, especially just working with challenging different stakeholders, and I mean challenging in the sense that we all have our own goals and initiatives that we're trying to accomplish. In regulatory and quality, we're really focused on ensuring that we're following our QMS and our SOPs and approvals. Working with the software team intimately, it was about how we can develop this code as fast as possible. And we often were at odds when trying to achieve that. And I know, as an innovator myself, having worked on a previous software team, I want to use AI as much as possible, but we really need to be strategic about how we do that and how we use validated AI.
Jenn Dixon: Alrighty. So I see we have the results of the poll now. I think folks should be able to see it on their screen. So near the top of the list: validating AI tools to meet documentation, audit trail requirements, and output quality, and managing AI-generated content to avoid incomplete, inconsistent, or noncompliant artifacts. We'll certainly touch on that topic just after this.
So regulatory alignment's an interesting one. I was thinking that maybe most of the folks on this call are coming from the regulatory side, so I know that there's a lot of global requirements around AI, and aligning all those requirements globally will certainly be a challenge. I know that even smaller countries... I think I remember Chile has AI-specific requirements, and just managing all of that can be really tough.
And then it looks like change control is also up there in terms of things that folks are worried about. And human oversight is on the lower end, which I'm a little bit surprised by: ensuring reviewers stay in the loop as AI output volume increases. Bailey, any thoughts on why that might be, or maybe any comments around our top performers on the poll?
Bailey Canter: Yeah. I mean, it makes sense to me that regulatory alignment is up there, right? Validating AI is really a challenge. It's commonly not understood today, and we don't have clear guidance on exactly how to do that, especially when you implement generative AI or AI learning algorithms. How do you validate that? So that makes sense to me, and we'll definitely show you how you can use that in the platform.
And maybe human in the loop is lower because we can always have you review that output and then take action on it in the system. So I'm definitely interested in learning more about these, and also showing how Ketryx can solve some of these in the demo today.
Jenn Dixon: Awesome. So jumping ahead into maybe some of the regulatory aspects here. I saw that change management was kind of in the middle. I'm a little bit surprised by that, because change management continuously becomes a challenge in our ever-changing world. We're moving very, very fast. Thank you all for the poll responses.
It's important to remember that AI in devices isn't new. The FDA actually approved the first AI/ML device back in 1995. So think, basically before everyone was on the internet: that device, the PAPNET testing system, assisted pathologists in reviewing cervical cytology slides. So all that manual work, putting the slide on there, reviewing it, that's where they applied AI. So nearly 30 years ago, the FDA was already open to the use of AI. What's slowed adoption since isn't really regulatory resistance; it's the change management aspect. How do you adapt your quality management system to support AI development?
How do you adapt your design controls, your standards, to that new world? AI introduces new variables, right? Models evolve, data drifts, validation becomes more of a challenging process, and it's more than just a one-time event, right? There are options out there for continuous learning.
Managing that kind of change within regulated frameworks, across tools, teams, and documentation, is really what makes adoption difficult, because you almost have to solve those risks and problems multiple times across the entire workflow. So the challenges are here: regulatory alignment, data integrity, clinical validation. All those requirements, they're not new problems, but they're amplified when AI is part of the workflow because there are so many unknowns. The opportunity now is to modernize how we handle change control, so innovation doesn't have to slow down every time AI or automation enters the process. So the FDA is trying to help us.
Right? They did put out a draft guidance back in January 2025. I doubt it's gonna get finalized anytime soon, but as always, these draft guidances represent the latest thinking of the FDA. So take a look at it if you haven't already. The agency isn't saying slow down.
In fact, it's saying be structured, right? Use those existing life cycle processes and policies to help you build AI that's safe. So you need a robust QMS that covers the entire product life cycle, including the post-market phase. It says, hey, you've gotta have strong data management.
So that might be kind of new, right? I think in the past, if you were validating and verifying something, you kind of just grabbed the data you needed and said, okay, yeah, we used that data, it makes sense, and we validated the results were good and wonderful.
Okay. And maybe next time you grab the same data or different data. The FDA really highlights how you need to be sure that you're not moving data or reusing data in the wrong places, right? Segregating your datasets. It calls out that UI and labeling is extremely important. And I think we've all experienced that before.
Right? There might be a time when you've gone to an online website and you're thinking, I'm talking to a human in this chat right now, but it turns out it's actually a bot. It's actually an AI. And that lack of transparency is the exact situation the FDA wants to avoid for all medical devices, right? They want all of their users to know if there's AI power behind whatever that functionality is.
The FDA also stresses that AI-specific risk management is something you need to take into account. So imagine, you already have cybersecurity-specific risks. You already have human factors risks. Now you need to think about what is the additional hazard or new technology that you're introducing here that invites these hazardous situations. Model drift is likely going to be one of those things.
Data bias is gonna be one of those things. And so how do you manage that risk? And can you also manage that risk through a predetermined change control plan? Bailey, I don't know about you, but you've interacted with a lot of customers. I'm sure PCCPs come up a lot when you have these conversations.
I find that a lot of folks might actually misinterpret the purpose of a PCCP. They're like, oh, let's just add it onto our submission and avoid future submissions. I think it's a really great way to actually showcase how you are continuously maintaining the safety and effectiveness of the device, because you have a PCCP. It's actually safer because you've gone ahead and planned those changes. Do you find that that's a common misinterpretation?
Bailey Canter: Yes, definitely. And one of them is the challenge in creating your PCCP. What is the designation? What is the workflow that we're going to establish for implementing change in our system? That's the first step, in my mind, of creating a clear PCCP.
The second part is executing the PCCP. How do you align your PCCP with your tool stack and embed it into the correct workflows, to make sure that all changes in your system are coordinated in a controlled way and that you're not cutting any corners or missing an approval where you need one? How do you enforce that PCCP? And then how do you align that with being agile? So yeah, lots of challenges around that, Jen.
Jenn Dixon: For sure. And certainly, how do you make sure all these approvals go through? What is the process you wanna utilize? Chances are you wanna just use your regular quality management system process, or an automated or deterministic process, to do that. Right?
So embedding these directly is a really great way to leverage those controls as evidence towards your PCCP, and you can ensure continuous, controlled innovation. So again, life cycle management, nothing new here. Make sure that you're applying your 13485. You're applying, if you're doing software, 62304. There is another standard out there that's relatively new.
So ISO/IEC 42001. This might be the one you'll encounter if you, like, Google "what is the AI management standard out there?" I always say that this standard is a great one to think about as a North Star. It talks a lot about AI-specific practices, methods, and things to consider for any AI management system. So it's not just constrained to medical devices; it's actually useful in a lot of different areas.
However, I see it as a really great starting point for where you should be and how you should think about your organization, but not necessarily something you wanna hold yourself accountable to. We have yet to really see a lot of the benefits of utilizing the standard, but I do wanna put it out there that there is a way for you to think about what the overall framework is that you wanna use. So what is the governance framework? And then maybe you can build your practices in alignment with that framework and reuse what you already have. Right?
Don't feel like you need to build a 42001 quality management system that's completely separate from your ISO 13485 quality management system.
Bailey Canter: Yeah, and one thing, Jen, I'm thinking about that regulatory experts might find really frustrating or hard to manage: how do you keep up with all of these standards that come out? How do you keep up with the new regulations that are dictated by the EU AI Act or 42001? You always want to make sure you're in compliance and not out of compliance. So it's definitely a challenge there as well.
Jenn Dixon: For sure. Yeah, there are a lot of ways to get more passive notifications when these things come up. Obviously, going to RAPS is a great way to keep up with the latest and greatest regulatory changes. You can always come to our webinars. We'll always try to notify you when something big is happening. But yeah, it is a full-time job to keep up with all the regulations and the latest interpretations.
So we talked about the hype. Now let's talk about ways that AI hasn't succeeded yet, right? A recent MIT study found that 95% of AI pilots fail. Certainly, when I read that, I wasn't super surprised. It's only alarming that it's a whole 95%.
Think of all the eventual waste out there of effort and not really getting the benefit you want. When you dig into the why, though, the reasons are not so surprising to anyone who's actually worked in a regulated environment before, or has a strict operational excellence workflow that they need to achieve. So the first reason they found was a lack of measurable business impact, right? We see a lot of really high-level, lofty ideas of how AI is going to change the world.
But when it comes to working with teams, teams want a very specific business impact or a KPI that they wanna hone in on, and many of those implementations didn't really focus on that. So many AI projects never make it out of that prototype, proof-of-concept mode. Instead, they're generating kind of interesting outputs, but nothing that anyone can back. Second, they found that there was poor integration into workflows.
So maybe the AI was something that lived completely outside of their existing toolset, and they just told folks, hey, every time you do X, Y, Z, make sure you go to this other tool and do some work there and then go to this tool again. Which I think, Bailey, you and I have experienced before, and it is not fun and not efficient in any way whatsoever. And quite often you make a lot of mistakes when you switch tools, right? AI often sits...
Bailey Canter: Oh, yeah. Sorry, Jen, I was gonna interject there. Yeah, absolutely. That context switching really slows you down. Teams just want to stay focused on their preferred tools and the way that they work.
Jenn Dixon: For sure. And because it's sitting on the side, say you're having a really tough day and you just wanna stay focused, you're very quick to put it off to the side and not really work with it. I don't know, does anyone recall Clippy? I didn't wanna use Clippy a lot when I was using Microsoft Word. I know that worked for the first couple of times you opened the application, but afterwards you're like, no, thanks, I don't need Clippy to pop up. So jumping back to the study, I'll not talk about Clippy any longer.
The third reason is the learning gap. Many AI systems don't adapt over time. They don't retain feedback. They can't adjust to the context. They maybe don't learn a lot about how you like to work.
So there's also, are they keeping up to speed? There's a misalignment, and this is something I think, Bailey, you and I talk a lot about. Someone is coming to us with a problem: what's the use case that they'd like to resolve? Teams tend to prioritize high-visibility areas like sales or marketing, while some of the biggest ROI, return on investment, often comes from improving internal, process-heavy areas like quality and documentation. And finally, build-versus-buy trade-offs. I'm sure you know when ChatGPT came out and they made their enterprise version, there were like a thousand ships launched of different folks making their own internal AI, maybe training it in house. There's a lot of effort that goes into refining it.
So projects built fully in house tend to stall due to limited resources or expertise, because you didn't really hire for that role, right? Prompt engineering has almost become its own profession. Whereas if you externally develop something that isn't domain-specific, it might not fit the use. So you're actually looking for something in between, right?
So something maybe externally developed that has resourcing behind it, that is a domain-specific solution. And what you're looking for, especially in the regulated space, is something that's already validated and is maintained continuously, right? If you do something in house and that expert manages to get something that works, as soon as that person leaves, which of course they might, because they might get poached since they're a fantastic AI/ML engineer, it's really hard to maintain that, right? So for quality and regulatory teams, these lessons are especially relevant. AI adoption succeeds when it's measurable, integrated into your daily workflows, and built to evolve safely over time, not when it lives in isolation as a pet project.
So I think folks on this call have probably all heard of AI slop by now. It's all over social media, and it's definitely a cause for concern. For those who don't know, AI slop is low-quality, generic, and often incorrect content produced by AI, and you're kind of surrounded by it. There's so much of it.
I think there are lots of people who are using AI daily, maybe just for their entertainment, but also sometimes for their work. So when AI implementations start to fail, one of the signs that it's failing is all this AI slop that's created, right? All these unintelligible outputs. You've seen it before.
The outputs are overly polished. This kind of slop immediately turns people off, right? So you see something like this, and you're like, hey, this doesn't make any sense. I'm gonna move on to what I know is right and correct. It tends to show up in a few predictable ways: things that don't make sense, repetitive outputs, frequent hallucinations. So it might get something wrong at first, and then get it right if you prompt it again, and then get it wrong again, depending on the time of day, the weather. So that is also something we don't like in quality and regulatory.
And because of this, we actually lose a lot of time. We might potentially lose more time than if we didn't utilize this AI at all, right? Because you're spending all this time manually reviewing the outputs. So what are the mitigation strategies? The good news is that these issues aren't inevitable.
They're symptoms of unstructured use, and there are ways to prevent them. One of them is human oversight: ensure the experts remain in the loop for review and correction. You never want AI-generated content to be just produced willy-nilly. You want someone in the loop for that. And you want a master prompter.
So someone who can help set precise expectations and constraints for the model. This is something that we have expertise in in house at Ketryx. But if you're building your own tool, you really wanna make sure you have someone who can refine prompts for you. And lastly, data integrity: grounding the AI in domain-specific data reduces errors and keeps results relevant.
So think about your AI as almost a child AI that has only grown up with books about patents. For example, I was talking about patents the other day. If it only gets exposed to patents, chances are it's only going to provide you patent-type data outputs, right? So you only want to expose it to data that you expect it to learn from. And lastly, the most effective way to mitigate these risks is with AI agents.
What does that mean? We're agentifying it. We're paring down the amount of data each one of these AIs is exposed to. So we have systems designed to pair generative innovation with deterministic controls. Some of them are only going to generate one type of data, and some of them are only gonna have deterministic processes that they're gonna enforce.
In other words, agents don't just create; they validate, document, and maintain traceability as they go, and they specialize in those things, right? So maybe it's validation of requirements, or reviewing test cases. That's how we move from "AI is a bunch of noise" to something that is reliable and safe, because each of those agents is checked against its own validation and performance criteria, much like humans. Alrighty, so how do AI agents balance generative and deterministic?
I've mentioned that a few times. Let's start with the generative task agent. This is the creative side of the system. I'm sure you've had friends in your life who were extremely creative and came up with these crazy great ideas. Some of them are good.
Some of them are bad. Hopefully mostly good. It executes individual tasks like drafting requirements, writing user stories, and summarizing test cases. It's fast. It's flexible.
It's capable of producing a lot of output quickly, and this is what we typically see when we think about AI these days. But on its own, it doesn't really know whether that output is correct, compliant, or approved. So think about a very enthusiastic intern. I think we've heard that a lot at RAPS. That's where the next type of AI agent comes in.
The deterministic process agent runs in parallel. It enforces your quality system around that generative work. It ensures that every change request, every requirement, and every test follows established workflows, with proper traceability, human review, and synchronization across tools. So again, that pain point of failing to sync things across tools: we now have that agent to do that work for us. So while the generative agent is creative, the deterministic agent validates and controls.
Together, they bridge innovation and structure, allowing teams to move fast but also stay compliant. So think about your left-hand agent, the generative agent: if you're looking for something where being right 80% of the time is good enough, then go for generation. If you're looking for that Six Sigma, 99% quality assurance, your deterministic agent is the way to go.
Now that you've seen how generative and deterministic agents balance each other, let's talk about how that works in practice. In the next section, we'll break down how these agents interact step by step to maintain control within regulated workflows. This is where we move from concept to execution: how agentic AI actually delivers systematic, reliable control in day-to-day use.
Now, if anyone on this call is working on an AI agent themselves for their internal tooling team, maybe their regulatory team, this is where I'd really like you to pay attention. So the first step is the prompt. Obviously, you're using LLMs, and the prompt is what you use to tell the agent what you want, right? So you can phrase it as, review all the software requirements specifications and flag any that need more detail. I know that within 13485 there's a requirement that all requirements must be unambiguous, testable, and traceable.
There's a very clear checklist for a lot of requirements, and sometimes your org has specific checklist items, so it can review all of those things. The quality of the prompt really matters, and that's why I think the prompt should have that checklist embedded in it. Think of it as setting the direction. A strong prompt focuses the agent's analysis so it delivers the insights you actually need. And once the prompt is in place, the agent automatically begins the analysis.
Once the prompt is submitted, the agent moves into the analysis phase. This is where it really adds value, right? It's utilizing all of the hardware, the GPUs, the CPUs, and it has that computing capacity that you maybe don't have time for, and it's performing that analysis. Of course, you can tailor that analysis. That's something that you can definitely customize depending on how you're building it. Next comes the recommendation phase.
This is where the agent turns its analysis into something actionable, instead of automatically making the changes, though. Again, we're working in a regulated space. We're working with that very enthusiastic intern. You want to make sure that the human is in the loop to review those recommended changes.
And the goal here, especially if you're under the AI Act, is that it needs to be as transparent as possible, right? You want to really empower the end users with the information they need to make the right decision. So that human review is very, very important. Most important throughout the entire process. AI agents will never take an action on their own.
I'd say we're not yet at the stage, for some processes, where AI agents should take an action on their own. However, we are starting to approach that, and what I encourage everyone on this webinar to do is think about: have we gotten to the point where we're human-reviewing a bunch of things and most of the time they pass that review? Now think about the impact of saying, what if we just automated it and let the agent make those changes itself? What is the impact, the risk and benefit? That's potentially where you might be able to have even further savings. But I will admit we're not quite there yet. I hope that we'll be there, with the accuracy of our models improving.
I hope we'll be there soon. And then the last step: synchronize. Getting it to sync across all of the systems is something that we deeply, deeply want. This is something you don't necessarily have out of the box, off the shelf, with most models, right? It won't necessarily copy and paste directly into your source systems, and that's why I really enjoy using our Ketryx AI agents, because they work within the systems that we already use.
Alrighty, so I just wanted to highlight a case study that we had recently. We actually have another one that's coming out very soon, which I'll talk about in a minute. Our example comes from a company that's a global leader in clinical research, with a digital clinical research platform. So they work under GxP, and a lot of the folks who are coming to audit them are working in that life sciences world.
They provide infrastructure used by major pharmaceutical and medical device manufacturers, and they run compliant clinical trials. Their platform connects hundreds of research sites worldwide and supports millions of workflows each month, helping teams collaborate efficiently while keeping studies on track. So think about all the documentation that comes out of clinical trials. Because they operate in a GxP environment, every release, end to end, needs to be complete and validated, and they have to have the documentation ready for the world's largest manufacturers.
Their QA and RA teams were spending countless hours manually compiling that validation evidence and generating reports. And so by implementing a requirements management agent, we were actually able to automate the validation and document generation across different versions of the product. So think about that AI coming in early, that requirements management agent coming in early, looking at what those new requirements are and making sure that the validation documentation is generated. Again, that's also part of the agents that generate the documents and make sure that validation was complete. So the agent was continuously tracking the changes, linking the related requirements, and generating audit-ready documentation, turning what is sometimes a multi-week process into something that happens within one or two days, a continuous background process.
So the results are powerful. Documentation time dropped by 80%. Change management became a seamless process across multiple systems, and things like redundant or conflicting requirements were automatically resolved. That wasn't something that came in at the eleventh hour, where you're just starting to say to yourself, oh no, we just realized we actually have three duplicate requirements.
So they gained both speed and confidence, proving that automation can actually make compliance stronger, not weaker, when applied with the right structure. And one thing I will say is that we do have another case study on the way. I was just speaking to our AI team about doing the same thing, but actually with code as your source information. So getting the code that's coming straight off the presses, or straight from Git, and generating documentation from that. And so, Bailey, I'd like to hand this over to you for AI agents and demonstrating that.
Bailey Canter: Wonderful. Thank you so much, Jen. You explained so much, and it was really relevant, especially for me, coming from the regulatory space as a systems engineer. And I'm sure many people on the call resonated with some of those pain points you brought up. Now I'm gonna walk us through a demo of how this actually gets executed in practice. How can we apply some of these ideas that we've been discussing in a platform like Ketryx?
Before we do that, I'd love to give this poll that allows you to interact and decide which use case you are most excited to see. Given time, I'd love to explore all of them, but I'm just curious to know which one is most relevant to the folks on the call today. So I'll just give you all a second to review.
Jenn Dixon: And maybe while we're doing the poll, we'll just quickly talk about this question we have coming from the Q&A: what are some ways to make sure prompts are clear and specific for regulated work?
So I always like to structure prompts with, here's all the contextual materials, and I want you to use these specific links. And you can always ask the question in a way like, if you are a quality and regulatory person, how would you respond to this question? Right? So really providing that context. Bailey, was there anything that you wanted to add there about ways to make prompts clear and specific for regulated work?
Bailey Canter: Yeah, absolutely. So part of it is what you feed the agent. The prompt is so important. And there is a period where Ketryx works with your teams to create validated agents, and there is that refining period. That refining and learning period is so important, because the agent can learn on your QMS, on the data within your system. And that's another key way that we make sure that we're not making any generalizations.
Every action and recommendation is taken from the data within your system. That ensures that we have a complete audit trail for every recommendation that the agent or AI features within the system are suggesting to you, so that we could ask, where did this come from? And it'll say, this came from this SOP in your QMS. And that's an example that I'll walk through today. So making sure that all of the agent suggestions are based in real data is key.
Wonderful. So it looks like we have some responses here, with the highest being producing a final submission document from source material. Of course, if we have some time, we could definitely review some of these other ones, like reviewing SOPs, triaging a complaint, creating a new requirement. But I'll definitely start with producing a final submission document from source material. Wonderful.
So now I'm going to actually share my screen in just a second here. I need to get that poll out of the way. Alright. Let me share desktop one. Wonderful. Are you all able to see my screen okay?
Jenn Dixon: Yes.
Bailey Canter: Great. Thanks, Jen. So for those of you who haven't seen the Ketryx platform before, welcome. This is where we'll show you how you can execute some of these ideas we've been discussing around agentic workflows.
We're mainly going to be in this insulin delivery monolithic system project, which contains our entire SDLC, all the activities that we need to produce the evidence of compliance, from design controls to test executions. And you can see here where we're doing that work. In this case, this team is developing and writing requirements directly in Jira, and tagging or contributing to that evidence directly in the source code. So we're able to actually integrate directly down to the source code, where you may write software item specs, or you may have automated tests that contribute to the documentation that you need to submit to a regulatory body.
Jumping into the insulin delivery system here, we're greeted with the All Items screen. You can think of this as your change control board that allows you to see all the changes in your system, no matter where they live. Again, in this case, we're pulling items directly from Git, so the source code repository, and you can see we're tagging them as software item specs and test cases that live directly in the source code. We also have requirements that live in Jira, and complaints are being managed in Jira as well. And down here, you can see we have some requirements that are acting as risk controls.
With Ketryx configurability, we can configure many different item types to support the way that teams are already working, from requirements all the way to post-market activities. You can think of Ketryx as that overarching layer that sits on top of the tools you're using today to ensure we have a connected system. This gives regulatory and quality folks the ability to have clear insight into the activities that are taking place, all the way down into the source code, and can guarantee that we have compliant and validated workflows as we're working. This ensures that compliance I mentioned, making sure that items are being moved into a controlled state with Part 11 approvals embedded directly into workflows. And here, you can also see the state of items, so ones that are in progress, resolved, or closed.
I'll quickly jump into Jira, and then I'll generate documents before we jump into agentic workflows, to really lay the foundation of how Ketryx works with teams today. Jumping into Jira, you can see the few things that Ketryx adds on top of Jira to allow Jira to be a tool that you can manage requirements in, because natively it doesn't do a great job of that. One of those ways is by embedding this approval workflow directly into the Jira item, as well as this local traceability that allows you to easily understand upstream, downstream, and items that would be impacted if I make a change to this requirement. And this is all interactive, so you can build traceability directly within your source tools as well. The third thing Ketryx does is help you build a robust audit trail.
This is extremely important in general, but also when you start to build in agentic workflows, because you need to keep track of every single change that's made to your system. And here you can do that by using this icon over here. We can see the item approvals, so who needs to approve the item, as well as a robust audit trail that shows me every state change that this item has been through, and you can also see controlled records. In this case, I don't have a controlled record here, but maybe I'll jump to this item. Oops, I'll go back here.
I'll jump to this item in Ketryx and show you what that history looks like when we have a controlled record. Filtering down to controlled records, you can see we have that audit trail. These are clear red lines and green lines that show you exactly what's changed. This ensures that your teams who prefer to work in their dev tools, who prefer to work in Jira, or prefer to work in the source code, can do that, while we experts in the quality and regulatory space can ensure that we're keeping track of all of those changes and we know exactly what's changing. We also ensure that they're moving items through a workflow that's in line with our compliance, our QMS.
I'll jump straight to our release dashboard and go to this 2.0 release that we're working on. In this case, we have a few more controls here that we can embed in our workflow, as well as visibility into what's going on in our system. We have some waterfall-like activities that we've created in our system, because although we want to be as agile as possible, we know that some things are just inherently waterfall, like making sure all of your requirements are gathered and in a controlled state before you can start your design review. Using milestones is one way to enforce that process directly in your workflow, so that developers can't start committing code or making changes to code while your requirements are still being worked on. Going to the documents for this release, these are all the documents that I need to generate and provide as evidence: we have our SRS here, our design, our test report, our traceability matrix.
All the evidence of compliance that we need can be generated with just a click of a button. What we find today is that teams are spending a lot of time, from code freeze to actually deploying to production, gathering all of this information and making sure they have zero gaps in their traceability matrix. But with Ketryx, we give you a real-time view of all of those documents and where you may have gaps in traceability as you're working, instead of at the end. Shifting these processes left really helps you work in an agile way and quickly. So just like that, we were able to generate all of our documents.
I'll actually open this as a PDF for you and show you exactly what that document looks like, for example. In this case, we just generated our SRS. So I showed you that requirement that lives in Jira, and this is us ingesting that information and showing it to you in a templated document here. We have a robust templating language that allows us to template this document so it looks just like the document that teams are generating today. And we pull all of that relevant information into this document for you to view and generate with just a click of a button.
Jenn Dixon: And what's kind of neat about that: an SRS is a software requirements file. If you have different requirements files, you can actually have different agents review each of those requirements differently. So for software, maybe you care more about error handling, so you might have a reviewer that looks at whether that's built into the requirement. But maybe for hardware, you'd have a different agent.
Bailey Canter: Absolutely. That's a great use case, Jen. Now that we've laid a foundation for the Ketryx platform, I'll just show you one more key feature here, which is traceability. This is making sure that we have full traceability from requirements down to where they're implemented in the code, as well as tested. And here, you can see that we integrate directly with the source code and automated tests to show you where you have passed test results or failed test results.
And we want to make sure that we have test executions that are passing. So you can have an agent that reviews that, checks it, and requires all of your tests to be in a passed state before you can release this project. And this icon allows you to see up and down traceability, where you can have agents that work in between each of these columns here to make sure that you have complete requirements implementing this user requirement, and so on. And it can also generate those requirements for you. Let's interact with the assistant now, with just a few minutes left, to show you how you may do this.
So let's ask it, for example, as a regulatory and quality expert... What? Oh my gosh. "What should I ask you?" Wonderful. And clearly, you can see here that my spelling doesn't really matter too much.
But here, you can ask the agent, or the assistant, things like, "What should I ask you? Things that I can ask you that are relevant to me in my role." Of course, these are questions or context that's in line with your QMS and traceability. So here, if I expand this, you can see that it's thinking through all of the items in my system and is able to show me things that I should ask the assistant to get a better understanding of what's going on in my product and where I'm at, and to make sure that this is ready for a release date. So these are important questions that you can ask it without having to go to an SME to understand where you're at in this project.
For example, one key thing you could ask the assistant is to identify a risk that could be introduced. Hi, let's just use the name here, Katie Too. So I'm going to ask it. Taking a risk-based approach is extremely important, especially as you're trying to comply with rigorous standards, or when you're using AI as well, with something like the EU AI Act. What a lot of these guidances say is that you really need to take a risk-based approach. And so being able to interact with an assistant that can take a risk-based approach that's in line with standards like IEC 62304 and ISO 14971 is a way that you can actually have it calculate and suggest where you may have risks, based on everything else in your system.
And so then we can go ahead and create a formal risk in our system together. For now, I'm actually going to jump into agents and explore how you can use agents as well to interact with your system. In this case, we're gonna jump into this requirement expert quality review agent. This is an agent that is set up to run, perhaps daily, that looks at all of my requirements and makes sure that they are robust and not missing acceptance criteria. So it's scanning over all of those requirements.
And as someone who previously worked in a requirements management tool like DOORS, there could be hundreds of requirements that I have to sort through, and I'm not sure if a requirement already exists. And so being able to use an assistant or an agent that can do that and follow SMART criteria for evaluation, we can actually look at all of our requirements, have it filter down to the ones that are relevant to us, and then review suggestions with the agent. So you can see that this agent is requesting that we add clear acceptance criteria and measurable requirements directly into our system. And of course, I can review this suggestion and keep that human in the loop to decide if I actually want to take action on this item, review it, and implement it in my system.
Similarly, we can review some of these other recommendations that it suggested, like editing the field description, and make sure that we have a comprehensive suggestion that's in line, again, with our QMS and activities. My Wi-Fi has been a bit slow today, so apologies for that response not loading. But while it's thinking, I actually want to interact with the assistant here and ask it some key things about my QMS and SOPs. Because Ketryx has an EDMS feature here, we're able to store all of our documents that are relevant to our product. Here, you can go into the QMS, and you can see all of our plans, policies, and procedures that are guiding the development of this product.
So I'm gonna ask the assistant what change procedure I should follow for making changes in my system.
Jenn Dixon: And, Bailey, if I can just jump in here...
Bailey Canter: Yeah.
Jenn Dixon: This is something I wish that I had when I started working. That first one you had up, where it was like, what questions would a quality and regulatory expert ask? Training on how to think like a quality and regulatory expert, and now having direct search access to these SOPs, these plan documents, would have saved so many hours of my time. I had to go chasing people down for those answers.
Bailey Canter: Exactly, Jen. And the first part is giving you visibility into the context of what you need to do as an expert. You can just summarize the current plans that you have today. But once you have that information, how do you actually execute it and use it in your system?
That's another challenge. And so because the assistant can read that information, it can then also suggest, here's where you can actually take the action in your system to execute on what you know as an expert. I think that's really powerful. So here I just asked what change procedure I should follow for making changes in my system. In this case, it actually referenced the exact change management plan, plan one, that lives in my QMS here. So I can actually open this up and show you that plan one does dictate what my change management should be.
And so here, I can use this thinking and planning to actually dictate how I should allow a change request to move through my system. Not only is this valuable for regulatory experts, but it's also important for software engineers or teams that are working on creating a change request. As an expert, you wanna make sure that they are enabled and empowered with the correct tools to follow that QMS. And with Ketryx, we can give them those guardrails to ensure that they are doing exactly that, that they're following the procedures that we have dictated to ensure the quality of our product. Going back to agents, I just want to call out a few more evaluating agents that teams are using to ensure the quality of their product, like this change request expert quality review.
We just went over the requirement expert quality review, but there are some other valuable ones here, like the anomaly expert quality review and the complaint expert quality review, which even looks at post-market activities as well. This is key to not only writing requirements and understanding what we need to do, but taking it the full circle of your SDLC, from writing requirements to observing post-market activities. This really puts the power in everybody on your team, from quality experts to developers to test managers, to ensure that we're enforcing a process, with agents helping us along the way, that creates a high-quality product and, most of all, a product that's safe. Jen, anything else I'm missing here, and anything you wanna review with everyone in the last few minutes we have today?
Jenn Dixon: No, not... How are you? I think we can...
Bailey Canter: I think you cut out a little bit there. Sorry.
Jenn Dixon: Oh, my bad. I'll get closer to the microphone. So one of my favorite things about this screen in particular is that you can definitely play around with the different agents that are accessible. You can keep an agent to yourself first and try to hone in on what you like about it. And then when you're ready to present it to the team, you can make it available to everyone. We have a couple of questions coming from the Q&A.
I think the first one is a very common one we get: how will Ketryx ensure that the information trained on will be confidential? So an enterprise setup for only one company. I know on my side, when we're setting up with clients, we have a zero-data-retention endpoint behind this AI. So everything that's typed into the chat on the right-hand side, when we work with our partners, is not getting retained or used for training.
We actually use Ketryx AI internally, and a lot of our training data actually comes from our internal data. Bailey, anything you wanna add about...
Bailey Canter: Yeah, exactly that. It's making sure that the data it has access to is within your own system. We aren't gathering information outside of the data that's in your project, as well as guidelines from standards like 62304. All of that is based in your data and in standards and regulations when we're making suggestions using agents or assistants.
And yeah, sorry, go ahead. Any other questions, Jen, that were in the chat? I'm just queuing up a response with the assistant here while we're talking.
Jenn Dixon: Yeah, there are a couple. So: what are some best practices to identify, trace, and record machine learning as AI tools train themselves on the data fed to them? This is more for when you're building your own product. Specifically from a compliance and QMS perspective, the data that you use to train the machine learning model for your product is fundamentally a design input, right? The data itself and its makeup are going to change the output of your design. So you need to incorporate that as a design input.
So you should not only have the data, but you should have a meta-analysis on the data. What is it composed of? How is it segregated? You need to have traceability to that level, so that you can present a bias report or a justification, a risk management report that incorporates data bias risks.
Bailey Canter: Absolutely. Yeah. And actually, one thing I wanted to show the team today that I thought was really cool is that you can actually have the agents and assistants make suggestions that are in line with rules that you set up. So again, going back to making sure that the output is dictated by some input, and that input can be rules that are dictated by your QMS. So I set up some rules in my QMS that anytime it suggests an item, make sure that it's written with this "Hello, Webinar," and explain that it's a part of my QMS guidance. So this is just one way that we can create predictable outcomes with specific inputs.
Jenn Dixon: Is it only for medical device regulations, or can this be used for drugs or biologics also?
Bailey Canter: Yeah, great question. We work with teams that are producing many different products, complying with GxP standards, whether they're creating a digital pathology platform, lab-developed tests, or a robot. We help teams in all of those different areas. Whenever you're trying to prove that your product works as you say it should, so it meets its intended use, you could benefit from using Ketryx.
Jenn Dixon: What do you usually see as the biggest barrier for companies when they start using Ketryx or a similar tool? Is it getting leadership on board, or helping teams trust AI in their quality processes? I know I have an answer for this, but I'm curious what you think.
Bailey Canter: Yeah, it's a great question, and it comes up very often. I think the biggest hurdle we face now is internal politics. You're introducing change, and the innovators who want change and who are already using AI are doing it quickly, and they see the benefit. But you're up against teams who've been doing things a certain way with legacy requirements management tools. And change is hard. So it's about trying to help them bridge that gap of understanding that AI and tools like Ketryx are innovative tools that aren't meant to be disruptive and are quite flexible. Introducing that, and understanding how those specific workflows can map into it, is key.
Jenn Dixon: Agreed. I'd definitely say that the biggest problem is very organization-specific. It really depends on what your organization is, how it's composed, what the goals are, that sort of thing. I know that on the client operations team, when we work with clients, we're always getting a sense of what the biggest barriers are. We're always asking those questions, and what business objectives you'd like to achieve. And then we go from there, right?
Bailey Canter: Wonderful. Well, with just a minute left, I just want to take a second to go back to the slides here and queue up our next webinar, Jen, if that makes sense. Yeah, so thank you all for joining us here today.
I know there were a bunch of questions in the chat, which I'm really excited to review and see if we can answer, whether in our FAQ or in a deep-dive session with you all later. But upcoming next, our next webinar is again with Jen and me, on streamlining multi-region submissions. So if you're interested, we would love to see you there. Thank you all so much.
Jenn Dixon: Yes, thank you all for attending. Looking forward to seeing you at the next one.