The Turtle Problem

At Ketryx, we talk to a lot of companies that build medical devices and GxP software. Often, we encounter organizations that have fallen into the trap of doing their own validation for their development tooling. This is an anti-pattern—let’s call it the turtle problem for reasons explained below. It is kryptonite for any product-focused organization. 

To make this argument, let’s discuss why you must validate your dev tooling, how companies slowly slip into the abyss of doing it themselves, and what you can do to stop it.  

‍

Behold the Turtle . . . ¹

Why validate dev tooling to begin with? The answer to this is easy: because you have to. All one needs is 21 CFR Part 820.70(i):

‍

“Automated processes. When computers or automated data processing systems are used as part of production or the quality system, the manufacturer shall validate computer software for its intended use according to an established protocol. All software changes shall be validated before approval and issuance. These validation activities and results shall be documented.”

‍

It’s funny how regulation works sometimes. This line makes up less than 1% of the words included in 21 CFR 820, but its implications are massive. To explain it, let’s use some stupidly simple diagrams.

Ultimately, the FDA cares that the product you put onto the market is validated to an intended use. To a first approximation, this means you:

1. Set a clear intention about what you’re going to build and how it’s going to address the problem you want to solve
2. Follow rigorous processes to build your product
3. Test the product to see if you built something that satisfies your aspirations set in step 1 
4. Document everything you’ve done to an enormous and minute level of detail

I’m oversimplifying to an almost absurd degree here, but the marketing team tells me I have a word limit.

So, cool, the product has to be validated. We’ll illustrate the fact that a software product is validated by coloring it green, like so:

‍

As opposed to a product like a website or a game that does not need to be validated:

‍

Now maybe you say to yourself, “All of those steps sound fairly reasonable, except for step 4. Sounds like it involves a lot of tedious bookkeeping that humans are generally bad at and hate doing. Probably better if I put some software around it to make it easier.” So you end up with:  

‍

“Aha!” you say to yourself. You are satisfied that you have tamed the complexity of the problem.

Then you reread 21 CFR 820.70(i)... you think for a while… you think some more… “Oh no,” you say.

Because what the regulation requires is not just that the product be validated, but every automated aspect of the production process AND the product. You thought you were solving this problem:

‍

But you really have to solve this problem:

‍

And it doesn’t stop there. The only correct interpretation of the clause is that this requirement is recursive. As we say at Ketryx, it’s turtles all the way down. 

‍

How things go wrong  

Now you’re in a quandary. You know you must solve the turtle problem. It’s right there in the regs, but you still don’t want to do it manually. You start to think, “We should validate Jira, then all the product compliance work can be automated.”

STOP. This is a trap.  

First, no. Jira covers only one part of the dev process, so you can’t just stop there. But, for the sake of argument, let’s say you could.

Second, by validating only Jira, you’re just shifting the manual work. Yes, you might automate the product compliance work somewhat, but you will be running a compliance process with spreadsheets and humans for the tooling. You end up doing the manual work you were trying to avoid in the first place.

Third, you might think, “It’s ok to do that if it’s a one-off effort.” But it’s not a one-off effort. Once you get into this validation tooling business (we live it every day here), you realize it is a problem of breathtaking complexity. To implement even simple features requires heroic technical effort.  

And that's the real problem. Notice I said technical effort. Oh, yes. Doing validated automation at any level of sophistication will require time from your dev team. I don’t know any organization on the planet that isn’t limited in some fundamental way by their development capacity. Think about it: when was the last time you heard an executive say, “Hey, you know what I think? We have too many software developers”?

So, now you’ve validated a custom Jira configuration. You reach version 1, but the work is never ending! You have to maintain it, add features, troubleshoot it.  All within a compliance framework. As just an example, ask yourself: when was the last time your team did formal bug tracking for your Jira config?  Never?  Guess what, now you do. 

And here's the kicker, each time you rev your automation, you have to execute another validation cycle to update it.  Soon you see that the time your teams devote to this nonproduct work only grows.  Eventually you realize that a meaningful fraction of your product development velocity is gone. Forever.  

‍

What to do?

Buy. Don’t build. I’m sure that conclusion is obvious considering the source, but I’m not trying to sell you on our platform here. There are products on the market that do this automation and I encourage you to explore long and hard before committing to doing the work yourself. 

Here are a few considerations as you research products and talk to sales teams:

1. How much of the application lifecycle does the tool you’re considering cover?  
2. Does the vendor provide design qualification documentation? Do they validate their product? (This is huge; it gets you out of the endless cycle of managing the validation yourself).  
3. Kick the tires on interface and performance. Tooling in this market is notorious for bad design decisions. 

And, if you’re interested in seeing how we solve the turtle problem in a modern way,  schedule time with us here. 

‍

‍

‍¹ “Behold the turtle. He makes progress only when he sticks his neck out.” - James Bryant Conant, American Chemist

‍