PODCAST: Before the Protocol: What Sponsors Miss About Sites and Human Factors

Emily Friedland: I'm Emily Friedland, VP of Clinical Research at DCN Dx, and this is Expert Insights. Most of the time on this show, we talk with people who develop and submit IVDs. Today, we're starting a short series that looks at clinical research from the site's point of view, the clinics and the people who run the studies, because that perspective rarely makes it back to teams designing the protocol. This is the first of multiple conversations we're having on this topic. Today, we're focusing on the earliest stages before the protocol is final: the human factors work that defines who operates a test and where, and the feasibility work that decides which sites can realistically deliver. The second episode will pick up with what happens once the study is running. I have two guests with me. Missy Farrell is the founder and CEO of ClinicaGen Bio, a site management organization focused specifically on non-interventional IVD, biospecimen collection and remnant retention studies. She's a registered nurse with more than 20 years in clinical research and has sat on the site, CRO, IRB, and sponsor sides of this work. And Dr. Karen Seidler is CEO and Principal at Usable Solutions LLC, where she works on human factors and usability for diagnostic and medical device developers under FDA's Human Factors Guidance and IEC 62366. Missy, Karen, welcome.

Karen Seidler: Thank you for having me.

Missy Farrell: Thanks, Emily.

Emily: Before we dive into the details, I'd love to hear a little bit, Karen and Missy, about how you got involved. Karen, with human factors, and Missy, your first experiences with human factors.

Karen: I have a serendipitous origin story for human factors. I was on a very different career path, and by chance sat next to a human factors person on a plane. And by the end of the flight, I was just so fascinated by a field that had the sole goal of making products easier to use and safer that I did a 180-degree pivot and switched careers and never looked back. And so over the years, I've done human factors for many regulated industries, but happily landed in the medical device industry about 20 years ago, which is an industry that can certainly benefit from the particular objectives and outcomes of good human factors process.

Emily: Missy, I'd love to hear from your perspective how human factors has entered your life on the clinical research side.

Missy: So human factors did enter my life on the clinical research side via Karen. It was during Covid and we were in the midst of conducting many, many studies trying to get over-the-counter and point-of-care tests available nationwide. And we had a particular diagnostic device that was being researched by the sponsor. And a very long story short, did not go as intended. And after the FDA submission and after negative feedback from FDA, then Karen was brought in. And if my tone doesn't allude to it enough, that was the inappropriate pathway. But, you know, it was fortuitous that I got to meet Karen. I knew that something like human factors had to exist in clinical trials of diagnostic devices. I just didn't know what it was called. And then I met Karen and we worked together on this product. And then we've been working together ever since. And in the grand scheme of things, if things go perfectly, clinical research and human factors shouldn't even know each other. It shouldn't have anything to do with one another. But here we are, and we're doing a podcast about this too.

Emily: Thank you both. I appreciate the background. Diagnostics, I think, is newer to formal human factors. And from my perspective, this is going to be one of our more exciting conversations. So I'm really looking forward to it. Missy, when you founded ClinicaGen Bio, you founded it to focus on a part of the research process that gets very little attention. When a sponsor asks about site performance, what do they usually mean, and what are they not asking that they should be?

Missy: There's a lot to unpack there. I think one of the first things to clarify is that in vitro diagnostics are not interventional pharmaceutical studies. They don't even fall under Part 312, right? And I've heard it said in the past that IVD studies are essentially the same thing as pharma, but easier. Or I've actually heard derogatory terms like "dumbed-down pharma." And that's not accurate as well. Some of the site qualifications or the criteria that you would look at to qualify a site and see if they are feasible as a site for conducting this study wouldn't even apply. So a site that has a pharmacy or a crash cart, or one of the things you see on the IRB submission is distance from the nearest emergency room. When you're talking about non-significant risk, which is the IRB category that they put these studies in, when you're talking about non-significant risk studies, you're not looking at that. We are not focused on the safety of the human subject ingesting a pharmaceutical product. We are really looking at, can this site identify and engage and enroll eligible subjects that can use this product or have this product used with them during a clinical workflow? That is a big key element.

Emily: I once had a manager in an organization who came from the pharma side into diagnostics, and she said kind of exactly what you said, Missy, which was, "Well, this is just half of what we do on the pharma side. It can't be that hard." And I remember her coming back to me not very long after, going, "This is a lot more complicated than I thought it was." And I think, you know, people forget that there's a lot of science that goes into diagnostic products and that we're asking people to engage with science in a casual way, or at their workplace, where they might not have normally done it previously. And that takes some planning and care. You said exactly this, which is that the pharma model gets misapplied to non-interventional studies. What does that look like in practice, and what does it cost a study when that happens in practice?

Missy: I think it begins at the inception. Our professional bodies that accredit us, there's not even any content hardly out there for this space of diagnostic devices. Yet the industry has exploded over the past couple of decades, and I see that as a big gap. You know, we just don't get the structured, formal training that we should. And so people that get training on clinical research are just kind of trying to figure it out and wing it, which, that's not really appropriate for something that's as highly regulated as clinical research. The thing that ends up getting missed is that, and you alluded to it, you touched on it just momentarily ago when you said, you know, we're enrolling people during a clinical visit. I want to stress, these are diagnostic devices. That means we're enrolling sick people during their sick visits, when they don't feel well, the children don't feel well. We're asking them to have additional samples collected. We're asking them to collect additional data and stay in the clinic longer, and asking the clinic to figure out what to do with them while they're there for this extended period of time, and they're sick. Operationally, when these things aren't taken into consideration, it's a poor experience for our patients, which, you know, I think all of us in an industry, we want our patients that are engaging to get a better experience from clinical research. And then it ends up being a poor experience for the clinic and ultimately can negatively impact the study.

Emily: Karen, a lot of teams treat human factors as a summative usability test you run on the finished device. Where does the human factors work start in the development lifecycle, and why does use-related risk analysis belong this early?

Karen: One of the biggest misconceptions I encounter in the IVD space is that human factors is just a validation study that's conducted near the end of development. But in reality, the validation study is only the final step in a much broader engineering process that should start at the very beginning of the product development cycle, and it continues throughout product development and through commercialization. And so, there's a lot of misunderstanding about what human factors is, but essentially, in the context of a medical device, it helps you understand your intended users, your use environments, who's going to be using your device, where they're using it, what they're trying to accomplish, and, most importantly, where use-related risks might arise that could actually affect the safety or effectiveness of using that device. And every design decision should be informed by this understanding. So, as I mentioned, human factors is a process, so there's a myriad of activities that are involved that just help you understand the intended user, the use environment, and the risks. And a key activity that you asked about is the use-related risk analysis. And this is where you're asking questions like, what could reasonably go wrong when someone's using the device, and why might that happen? How can we design the device so that those use errors are less likely to occur, or if they do occur, are less likely, at least, to cause serious harm? And those types of conversations are much more valuable while the device is still flexible, and not after the device has been designed, or the instructions, the packaging, the software. We don't want to be asking these questions after all of that is finished, because if you wait until the human factors validation study to discover that users don't understand an instruction, or they can't interpret your test results, or they're consistently stumbling over a critical task step, then you've actually discovered the problem at one of the most expensive points in the development process. So now you might find yourself having to revisit design. You might have to repeat your validation study, or even worse, you might delay or have to repeat your clinical study. And so validation testing isn't where human factors start, it's where you actually confirm that all the work you've done up to that point has been successful, that the remaining use-related risks have been reduced at least to a reasonable level, and that intended users can use the device safely and effectively. And at that point, you're much more confident that the device is ready for your clinical study.

Emily: When I started in industry many years ago, one of my first jobs was acting as that last step out of development before it went into trials, in validating systems. But that was when I was a trained user, a scientist in a lab working on large or medium-sized, complicated, highly complex systems. Now an IVD gets to be used by different operators. Sometimes there are people like who I was a trained lab tech in a lab, but now it can be a medical assistant in a busy clinic or a patient at home. How does the intended user and use environment change how a device gets designed and validated?

Karen: All the different user groups, they drive the design. And one of the first things that we do in the human factors process is try to understand who the intended user is and where they're going to be using the device. And different user groups, the ones that you mention, all bring to the table different backgrounds, training, experience. There could be a wide range of physical and sensory and cognitive differences, abilities and limitations within those groups that affect how people actually are going to interact with the device. The use environment is just as important, where the device is going to be used. So someone collecting, say, a swab sample for a home STI test may have very limited counter space or nowhere to stabilize a collection tube, whereas in a clinic environment, there might actually be a tube rack available that supports the task. So those types of differences can influence the design of the device and the packaging and the labeling. And home use, you mentioned the lay user, home use often expands those considerations even further, because now you have to think about pediatric or older users, just a range of users, caregivers assisting with testing, people with maybe visual or dexterity issues. And this is why we spend a lot of time in human factors understanding the intended users and the conditions under which they're going to be using your device at the outset, and then identifying what the potential use-related risks for these different groups are, and then try and design them out of the system. Essentially, the earlier that you identify these use-related issues, the more opportunities you have to solve it through good design.

Emily: And I think this is becoming especially more complex and important as we're integrating technology with home use, software, all of those things that people sometimes interact with on a daily basis, but maybe don't interact with it in their healthcare, or with something else, having to manipulate something else. This is a really important topic. Missy, when a study lands at one of your sites, how often do you find that the device design assumes conditions that the clinic can't meet? And what are the common ways that there are mismatches between clinic use and instructions for use?

Missy: You asked if it's common, I would say yes, it is common that there's some element of the device design or the packaging or the instructions for use that's really not fit for the intended user or in that use environment. I would say it's pretty common. And, you know, as Karen was alluding to, clinical trial is not the place for this to be happening. These products, the packaging, the labeling and the workflows, and I'll talk more about the workflows in a minute, these things should all be ironed out. And the CRO and the sponsor developers should have a very good idea of how the product is going to perform in the clinical setting. The device itself, the instructions, the only thing that we really should be doing is testing it in the hands of the intended users, in the intended use setting, and working towards capturing data for that sensitivity, specificity, in a PPA. These things that we do during clinical trial, we shouldn't be tweaking it at all at this point. It should be fit for purpose.

Emily: I think we've all seen too many times an obvious problem once it gets into the hands of the users, that we then lose time and money, frankly, and sometimes patients, right? Because we have to go back and rework labeling to meet the need of the user. So we want to work that out early.

Karen: Absolutely. I mean, I see that all the time, the later in development that you put the device in the hands of the user, the fewer options you have for addressing issues. And certainly by the time you get to a clinical study, it's expensive to go back and revisit.

Emily: So in that regard, where do human factors and clinical studies overlap, and what does the sponsor get when these two pieces are planned together instead of separately? And I think when we're talking about this, we don't necessarily mean when they're planned to be run together, but just that we are thinking about human factors and thinking about our clinical study together.

Karen: I actually don't see much of an overlap between human factors and the clinical study. I view it as a handoff from human factors to the clinical world. Human factors and clinical studies, they're answering very different questions. So human factors is asking, can the intended user use the device safely and effectively? And clinical studies are asking, does the device perform clinically as intended? And when the human factors work, when it's completed, first, you have the very tangible benefit of having already identified many of the foreseeable use issues and usability issues that you might otherwise see, and you've been given then the opportunity to refine the design and demonstrate, essentially, that representative users can perform the critical tasks safely and effectively. And so that de-risks the clinical study. So you get to go into your clinical study with much greater confidence that users aren't going to get tripped up by usability of the device. And so now the clinical study can focus on what it's intended to do and evaluate, which is clinical performance, rather than revealing usability issues that could have and should have been addressed already, well before you ever got to the clinical study. So human factors, essentially, again, not so much of an overlap, but a handoff, it helps set up the clinical study for success by essentially reducing surprises.

Emily: I used to always tell people that we wanted to mitigate risk in our clinical trial as much as possible, by feeling as confident as we could that the results we would get in that trial would meet our expectations. And I think now some folks are getting a little bit confused, because we do a usability portion of a study during a clinical trial, often with point-of-care and CLIA-waived and over-the-counter tests. But in my opinion, and Karen, I'd love to have your opinion on this, that should be usability.

Karen: That should just check all the boxes. We should know that they're going to be able to complete those critical tasks well ahead of entering those tasks into the clinical trial. If, during the clinical trial, additional usability information is collected, that's wonderful, but that's really not the time that you want to be discovering usability issues. It's expensive to discover at that late stage. It could seriously impact your clinical study if you have to go back and redesign. And often, at that late stage, what you're left with in terms of options for fixing any usability issues are really just throwing in things into your instructional materials and hoping that that's going to address the issue, as opposed to, if you had considered the user and the use environment, and explored through what we call formative testing, which are studies, usability studies that inform the design, had you done that along the way during development, you would have had so many other options available to you. The best way to address usability issues is by actually building the solution into the physical device itself, and not rely on instructions. We all have seen how users treat instructions for use or QRIs, it is so hard to get them to use them. And, you know, particularly if you're designing for a home environment or lay user, there's a range of abilities and health literacy, and literacy just in general. And if you can design usability issues out of the product or the physical device to begin with, that's the best way to go, not try and just do patchwork. You know, these days, visible mending is so popular, you don't want your instructions for use to be this visible mending that you're doing and trying to just fix it late-stage. But again, the two studies themselves are so different from each other, they're asking different questions, the methodologies are significantly different between the two types of studies. So there's a number of logistical considerations that make it tricky to combine the two studies. For example, a human factors validation study relies on trained observers and interviewers, because we're not just documenting whether someone completed a task correctly, we're actually observing how the task was performed, and then, very importantly, conducting what's called a root cause analysis to understand why usability issues occurred. And this root cause analysis is very important to interpreting the human factors validation study findings, because it's informing the residual risk analysis, which is essentially the assessment of whether the residual risk that's been observed, maybe, during the validation study is acceptable or not. And this is very difficult to build into a clinical study, because I imagine you'd have to have a trained observer for every single sample that's being tested. Essentially, another logistical consideration involves who the actual participants are who are needed for the studies. Missy and I chat about this a lot, in the context, say, of a CLIA-waived study, the participants in a human factors validation study, they're not the people donating the sample. Our participants are the untrained participants. In clinical studies, whereas maybe you only have to use a minimum of three sites and three untrained users per site for a validation study, we typically need a minimum of 15 representative participants for each user group. So that would be, in this case, we would need a minimum of 15 untrained users in order to properly conduct the validation study. Logistically, that's very difficult to do in a clinical study. And so there's a number of these methodological considerations that make it very difficult to combine these two studies.

Emily: Thank you so much. Missy, going back to the site side, you pointed to site feasibility as a place where important context falls through. What do the standard questions ask, and what are we missing? Those of us that are sending you these site questionnaires, some of the things that are missing, we can point back to regulatory, right?

Missy: So the sponsors and the CROs are supposed to select sites that are qualified for the study, that's the whole point of the feasibility process. During that process, we routinely do get correct questions that ask, "Have you engaged in any IVD studies in the past?" This is where it falls off the cliff. The next question that comes up is, "What is your site's prevalence of Covid, prevalence of gonorrhea and chlamydia?" Okay, these two things, looking to see if it's asking if a site has experienced conducting IVD studies, check that box, okay, that's a good one. The next question really should be, "What was your site's most recent experience? What type of IVD study was it? How many did you enroll? What was that prevalence? What were your timelines? Did you submit all your data on time?" They need to ask study-specific questions. There's no confidentiality issues with that. But instead, often we get asked questions about prevalence. That really is not a good proxy for how you're going to enroll. If you have a site that has really good prevalence of STIs, for example, there's other questions that you need to tease out to see if the site can actually capitalize on enrolling those subjects. You need to ask a lot of questions about workflows. Just because a site has a patient population, that's not a guarantee, as I said, it's not a proxy for their ability to enroll those people in a study.

Emily: I'm hearing a lot of the questions that we ask routinely, and I understand that, and I do agree that we have a tendency, I'm including myself in this as part of the CRO process, right, where we're trying to meet a client's needs, and they want to enroll in the shortest timeline they can and get the most positives in the most populations. They can minimize their costs. And I'm realizing, as I'm talking to you, that there are better ways to understand that versus, we had, you know, 300 positive gonorrhea tests in the last year. That doesn't mean that those patients are going to be accessible or meet your inclusion criteria, or they might not even be the right sex for your study, right? Maybe we had 300 male gonorrhea patients, and we're trying to find those elusive female gonorrhea patients, and there's less partner notification at this site or something like that. So I am hearing and onboarding this feedback for our own site feasibility questionnaire. As part of that, site selection is supposed to be driven by the intended-use population and the operational capacity, which is where I think some of these questions come from, even if we're not asking them the right way. What does a sponsor see on paper that turns out to be wrong once the study starts?

Missy: Well, I think you started down that path with your buildup to the question. I think, often, looking at raw numbers without asking the next steps on how we'll get to those numbers, often, you know, there's an assumption that sites already have these numbers just laying around, that clinical sites just routinely run prevalence, which couldn't be farther from the truth. Some sites throughout the U.S. pay no attention to their prevalence whatsoever. Other sites, depending on the type of medical facility they are, do run prevalence, but it is for a very specific government-required purpose. So typically, the way we ask for prevalence in studies is unique to research, and I can give a really good example of how we need to tease out more data from that. So, a minute ago, you said, you know, we asked the sites how many gonorrhea cases they have in a year. One of the things that I routinely see is a lack of follow-up on the breakdown of those. Now, again, I'm saying this in two voices, I'm telling you, we need more granularity, but I'm also saying that granularity is hard to get because it isn't something that is a standard electronic medical record report that you can pull. It's often a manual process to get these. One of the biggest areas, and STIs is a perfect example, if you look at CDC, and you look at data at CDC, it says that the population that has probably some of the highest prevalence of STIs is adolescents. Adolescents, although we do enroll them in studies, adolescents, they can, in most states throughout the United States, get diagnosed and treated for STIs without parental consent. However, that does not bridge over to clinical research. Adolescents cannot sign consent forms for themselves to enroll in an STI study, even though, non-interventional, you're not putting anything experimental into this adolescent, they still can't enroll for themselves, and most adolescents are seeking diagnostics and treatment for STIs without their parents. So here's a huge gap, we have these numbers that a site reported, how many of those are adolescents that you are probably not going to be able to get in your study, even if your study does include adolescents. So that is a really big one. You know, that answer, Emily, is multifactorial. It is digging into data to get the information that you need to decide if this site can enroll, looking at the site's data to see what their population breakdown is, males, females, adolescents. It's also asking for data that might be manual. I can't tell you how many times, you know, I'll reach out to a site and I'll ask them questions about their prevalence, and they'll say it's going to take me a couple of weeks to get the information, because it is a manual process. Even with everyone on electronic medical record, the way that you may have to run reports, pull different reports, you may have to go to a person that has access to run those reports, because not every nurse or clinician in the office has the access privileges in their EMR to run those reports. So there's a lot around feasibility that is poorly understood and is frankly misunderstood, especially in certain areas like STIs. Yeah. And I think, even when we take a step back, before we're even in full feasibility, as a CRO, right, we're under contract with a client, and now we are fully working the sites on feasibility. Before that, when a client comes to us for a quote, they want a guarantee of these things, and they also want a quick turnaround time. And it's kind of like, also, today, this might be the prevalence, but in three to six months, when you make your decision, I can't, one, guarantee the site is available; two, that this population is going to stay the same; or three, that this prevalence is relevant to the design that we ended up with in this study.

Emily: So these are all very, very good points, Missy. And to that point, I think you touched a lot on the mechanics of a site and the maybe misunderstandings, misconceptions about how readily available data are. A lot of developers come from a data-science background and have never had to think about how the U.S. healthcare system works and how fragmented it is. What surprises them most when they meet the clinical reality?

Missy: I think they would be shocked at how few patients actually receive standard-of-care testing. It is shocking, you know, from STI testing to respiratory testing during flu and Covid season, the number of people who are symptomatic and, in a perfect world, should get tested, don't. Now, what does that do to our studies? It gives us skew in our feasibility, because we don't have perfect numerators and denominators, right? It also gives us skew in our user population. And I say that with a great big sigh, because, you know, when people are coming into a clinic, they're sick. They want to be diagnosed. They want to know what's wrong. They want some sort of treatment for this illness. They want to feel better. Now we're going to approach this patient population that is ill, doesn't feel well, and wants a diagnosis and treatment, in order to be in one of our studies. It's great, that's not an issue there, but it's understanding that the folks that are probably the most likely to have an infection and be our best subject on paper, if we don't have mechanisms at those clinics, or within our studies, to help contribute to their diagnosis, then those subjects are less likely to enroll in our study. That's, you know, just the long and short of it.

Emily: We've talked a bit about STI studies, Missy and I overlapped on a lot of STI studies in her background, and so we talk about that a lot. And I think they're a good example of how recruitment isn't the same for every population, specifically in sex-based organization of a study, or enrollment of a study. Where do male and female recruitment diverge, and what can a site realistically promise as a result?

Missy: I think, in the context of STIs, when we're conducting STI studies, it's frequent that women go to OB-GYNs, and they're going to an OB-GYN to get tested for symptoms of an STI, which are usually urogenital. Men are not going to an OB-GYN to get tested for their urogenital symptoms, unless there's something going on, you know, a male and his female partner are going to an OB-GYN. So men usually go to a different place, or go to primary care, or they'll go to a municipal health department or something like that, or a different community-based clinic setting. Going into this, there needs to be an assessment of where do we go for, even just gender? You know, where do males go for STI testing, where do females go for STI testing? You know, the challenge is finding the sites that test adolescents and women and men. And so it is definitely something that you need to consider beforehand. One of the things that we do at my company is we understand that this is time-sensitive for the CRO, we understand it's time-sensitive for the sponsor, and we understand that there is a shopping aspect to it, that, you know, sponsors have to shop around and find, you know, what sites, what CROs, we get that we conduct our own feasibility in-house, and we conduct that routinely throughout the year. Sometimes, you know, as much as two or three times a year, depending on how many new sites that we're working with. And we keep that data banked so that, you know, we have relatively fresh data, because, at least for STIs, it is somewhat cyclical. Now, conversely, let's talk about respiratory. Respiratory is not something that you can bank that data. You can't say, "How was your participation on a Covid study last year? How was your participation, how was your site, you know, how well did they do? What is your prevalence on flu?" That changes year to year to year, because, you know, we have different strains of these respiratory viruses in circulation, and it varies regionally. You know, one year we may have a horrible flu season up in New England, and, you know, in the Western states, not so much. So really knowing how to manage the data is very important. And some of this, this is not, you know, information or a critique of CROs and sponsors, this is on the site side and site management side, you have to understand what it is the CROs and sponsors need and have that information ready for them. And not, you know, if you're engaged in a lot of these studies, that should be something that you can prep ahead of time and have that information ready, because the information that is needed to determine if your site is a good site or if it is eligible, really, some of the questions are kind of the same thing. And the extra information that I'm alluding to is something that we, you know, try to gather on our end, because we know that it's important to be able to provide that to you guys on the CRO side, but then also sponsors.

Emily: We talked a lot about prevalence and reaching patients for prevalence, and all of those things. There are other factors that go into determining whether a site is going to be successful or not successful in a study, including past performance, access to populations, things like that. Many of the clinics that can reach the right patients are resource-constrained and they're at their operating capacity. How does the site management organization, like ClinicaGen Bio, make those populations reachable without overwhelming the site?

Missy: The things that we do, you know, it's part of our business model, is we try to lift up and lift off so much of the administrative that, you know, kind of bogs down sites, you know, in terms of the regulatory, in terms of the data transcription, in terms of advising the site, or, in some cases, the CRO, we're the best point in the visit, or in the entire research portion of the visit, to collect the data, is we try to do that. We tried to find creative ways to work in standard-of-care testing, because, as I alluded to, if a patient's going to a clinic and they want a test, they can't get a test, there's no federal programs that pay for tests. And now you're offering a study, if we could just have a test result in that study, that could make a huge difference, and it could definitely impact how receptive that patient is to engaging in this study, and give them what they were there in the clinic to get in the first place.

Emily: Karen, when a device has to interact with a very broad population of users and a broad set of intended use settings, we have a lot of clients who want to make dual claims, right? They want to make point-of-care and over-the-counter claims. Maybe there's a professional-use claim and a lay-user claim. And then there's this variety of patient populations that are included in it, elderly, young people, infants, parents, you know, testing an infant, people testing themselves, women doing their own genital swabs versus men collecting a urine sample, just a group of examples. What types of things do you need to consider, and how does this affect, in general, the usability work that needs to happen in advance of launching into a clinical validation?

Missy: Could I interject one point there that I think is a point of clarification? I think one of the things to consider is, when we are enrolling human subjects in trials, Karen's enrollment of participants in simulated trials is very different. We are enrolling sick patients in clinical settings, collecting human biospecimen samples and utilizing them on these diagnostic devices. Karen, when she's conducting her trials, they're simulated in the same user populations, but completely different contexts. There is no biologic samples, there is no clinical setting where there's other sick people, and, you know, so it's a very different context in a lot of cases. And I'm kind of staying in this vein of, you know, sick visits for respiratory or STIs or things like this. But Karen, if you could elaborate based on that, I think that might be an important distinction.

Karen: And what you're talking about there is more in the validation testing, or maybe formative testing, and who we're recruiting for that. And again, for any human factors study, you always want to recruit your representative users, whether they be the lay user. And then, within that population, if there are differences in how the different users within that user group will interact with your product, you might have to parse that out into more than one user group. For instance, for the Covid test, or any in vitro diagnostic test for the home user, maybe you might have self-testers versus caretaker/care-recipient testers, so that might break it out into two user groups at that point. Again, the approach, just in general, to the design and to the testing is still the same, it's important to be listening to your intended user groups, whoever they might be, and understanding how they're using your product, or will be using your product, what are the conditions under which they're going to be using your product, and the different characteristics that might impact how that product is used. Whether you're designing for, say, a lay user, if you've come up with design solutions that help a lay user, but you also have your untrained user, everyone benefits from good design. And so solutions that might work for one user group to make it easier for them to understand how to interact with your product, they're going to make it easier for everyone to use. But you do, in your design and your testing, have to consider all of your user groups. It is very important, because the differences, the experiences, the educational backgrounds, just their dexterity issues, visual issues, all sorts of things, can impact how your product is being used, and you need to consider all of those differences.

Emily: If the sponsor brought you both in before the protocol was written instead of after, what is the first thing each of you would change?

Missy: If we're talking about a protocol that's combining both the clinical and the validation study, I would be asking them why we're trying to combine those two studies in the first place. And again, for some of the reasons we touched on earlier, the human factors validation study should be conducted before the clinical study. It's supposed to be demonstrating that the design mitigations address the critical issues or use issues. And so you don't want those use issues following you into the clinical study.

Karen: So, so I guess what I would ask to change is, first off, not run them together, not in the same time frame. But if, for some reason, if there was an absolutely compelling reason for the studies to occur in the same time frame, maybe there's some narrow seasonal enrollment window that you can't miss, so you have to conduct both types of studies in the same time frame, I think it would be important to differentiate between running the studies in parallel versus actually combining them. I would recommend that they not be combined, that the protocols be as independent as possible, for reasons that we touched on a little bit earlier, just by nature that the studies themselves are just asking very, very different questions, and, as a result, the methodologies are so different, and it's just very difficult to run them in tandem. I would also, though, caution the client that you should not take that risk, even running them in parallel, unless you have done your human factors homework along the way, to begin with, that this is not the first time that you're putting the device in the hands of the end user. Again, you do not want to be discovering usability issues during clinical studies, or even the human factors validation study. You want to have the opportunity to have the representative user groups interacting with your product beforehand, so you can uncover usability issues and use-related risks, and then design those out of the product. And then you can more confidently go into your human factors validation study, which, again, is supposed to be demonstrating that you have, in fact, mitigated design issues that could lead to serious harm. And also, by conducting the studies more sequentially, and not in parallel and not in tandem, you're helping to de-risk the clinical study itself, as Missy spoke to earlier.

Emily: And Missy, what advice, from maybe a site perspective, would you want a sponsor to think about as they're designing their clinical protocol?

Missy: Definitely concur with everything that Karen has said. And this is actually something over the past year and a half that I've started doing, when I get an opportunity to interface with clients, I point-blank ask, "Have you conducted thorough human factors work?" Because if not, we see it in the clinical trial, and it does impact the clinical trial. And I think, also, I would ask about their pre-sub status. Sometimes it's an indicator, you know, it doesn't mean that we won't work with them, but if they've not gone through the efforts to evaluate their product and the packaging and the instructions very thoroughly, and they haven't done a pre-sub, it kind of portends that things are going to be bumpy in that clinical trial. And it's unfortunate, because they come to you with a position that they don't know what they don't know. If they don't know why they should have done a pre-sub, or why they should have done human factors work, then there's an unhealthy focus on what is happening at the site level, and there's blame on the site level that shouldn't be there. One of the things I would love to be able to do and find out from a client or sponsor ahead of time is be able to look at the instructions for use or any of the ancillary products, we're talking about a fingerstick, and, you know, there's a capillary tube, I'd really like to see that before we get started, because there's some variability with how these ancillary products work. And, yeah, just being able to discuss, you know, hear from them about how they think their product is going to be integrated into a clinical setting. And in the United States, you have regional differences in medical settings, you have differences from multi-specialty groups to very small private doctor's offices, and how medicine is performed. And, often, I hear clients describe how their product is used in a clinical setting that I've never heard of before, it just doesn't exist, maybe not in the United States, you know, and that is something that you have to consider. A lot of products are initially developed outside the U.S., and there's an influence on medical systems that don't exist here in the United States. So, yeah, I would, you know, just being able to get information about how they envision this product being used, the populations, being able to look at the product, being able to look at the package insert, the QRI, being able to look at the packaging itself, asking some questions, you know, if this test is supposed to be used by an untrained user, and it's intended for CLIA-waiver use, and we're conducting it in front of the patient, how do we maintain the blind on the patient? The patient's not supposed to see the result. Sometimes very simple questions like this, are we supposed to pick it up and walk out of the room? Do we kick the patient out of the room? I mean, how do you envision these tests being used, and just the real nitty-gritty of it? I would love the opportunity to have these dialogues ahead of time.

Emily: Well, you set me up for two of my favorite soapbox topics, which are, one, the Q-Sub process is free and you should use it, number one, if I can tell anybody anything, it's that. And number two, capillary samples are really hard to do. People really struggle with them. So those are my two favorite soapbox topics, and I definitely heard you touch on both of those here.

Karen: Oh, I was going to say, if I could jump in, even things like a capillary sample, I mean, with the untrained user, I think sometimes there's the expectation that clinical staff or clinical users have knowledge that they may or may not necessarily have. And sometimes the supporting materials, even the instructions, might leave out the instructions for how to collect a sample using the capillary tube, or they take for granted that, because these are clinical users, they bring with them a certain level of knowledge, which may or may not necessarily be the case, depending on who they are. Missy and I sometimes interact this way, we do a quick and dirty, maybe formative study beforehand, just to see where we are with the product, and try and catch some of those use issues, just to see if there's at least some quick fixes. I mean, again, ideally, human factors would have been incorporated into the process along the way. Unfortunately, the reality is that's often not the case. So what can we at least do at that stage? Is there anything we can do with maybe the QRI? The weakest solution is usually trying to pack things into the instructions for use, but if you're at that point in the development process, that might be your only option.

Emily: Well, we've done studies in patients or populations that aren't considered by the mainstream very often, like people who inject drugs, or people who are unhoused. And even just the choice of lancet used on people who may be out in the cold, or work with their hands, or have had other, you know, reasons why they have deep calluses or difficult fingers to stick, aren't considered in the design of the product, even though the product is intended primarily to be used with that group of individuals, right? So my hands are as delicate as they come, and even I have trouble sometimes getting a capillary sample out of myself. These are all good, very good points about consideration of all of the factors that go into running a product. Missy, going back to the site side and the feasibility questionnaire, I think we covered a lot of this, but if you could just add one question to every feasibility questionnaire, what would it be?

Missy: We often have no idea why you've put a given question into the feasibility questionnaire, you know, and we don't know if the response to that question means we're going to get selected or not. You know, there are some questions that are just really befuddling sometimes. So it's not, I don't think it's so much a critical point that you must always ask this. I do lean back into GCP, if the obligation is for the sponsor and CRO to choose sites that are qualified, I think the best proxy for whether a site is qualified or not is: have they done these studies? And if they have done those studies, how did they enroll? What was their prevalence? You know, some other indicators for performance around those specific studies. You can get that information in a way that, if the trial is not out there in the public realm, in terms of ClinicalTrials.gov or something, you can get that information in a way that stays online.

Emily: Missy, what's the most common wrong assumption you hear? One from the site, and then, Karen, one from human factors.

Missy: Again, going back to feasibility, it's often treated as one-and-done. So we get a draft protocol, or, even better, we get a synopsis that's like 200 words, and we complete feasibility off of this, and no one comes back to ask, "Hey, now that we have a full protocol," or "Now that we've added males to the study that was originally females, what does that do for our feasibility? Can you give me some numbers?" You know, when feasibility is completed and the study doesn't go as planned, the knee-jerk assumption, that a site failing to enroll, or not having the prevalence that was projected out as the clinical prevalence, there's this knee-jerk in the industry that the site somehow falsified their data or inflated their data. And, when, in reality, the industry is not taking ownership for the fact that they did a poor job at assessing the data and assessing the site, or they have an incomplete understanding of the medical system and how it varies from clinic to clinic to clinic. I think the biggest assumption is that there is malintent there, and there's never, I can't think of once in my entire career when a CRO or a sponsor has come back and said, "Hey, Missy, these are the numbers that were completed, this is the data that was completed on this feasibility, and this is how things happened with the study, help us understand, give us some feedback on what we could have done to project better, what happened that caused skew to where the site didn't enroll what they had estimated they would enroll." We never get that opportunity.

Emily: We often hear that somebody lied, or somebody falsified their numbers in order to get in the study, which is, it's further from the truth. It's definitely a perspective on human nature that I don't generally have. I don't think most people are trying to game our system to win clinical studies, because ultimately you lose one if you don't meet expectations, because you falsified some, we're not going to select you for another study, right? So it is an interesting way that people look at things. Karen, from the human factors side, what's the most common misconception?

Karen: Oh my gosh, so many. One common misconception, I think, I encounter is that human factors isn't needed because the IVD device seems simple, or it looks very similar to products already out there. And, in reality, even small design differences, whether they be in the instructions, or the device itself, or the packaging, can sometimes have very unintended consequences that, at minimum, some testing needs to be done just to identify if, in fact, there are any new use issues introduced with the product. An example that just comes to mind, it seems so innocuous, but a tube holder. I've actually seen this across a number of different products, where the manufacturer wanted to do the right thing and include a tube holder in the packaging to help stabilize the tube for the user. However, the illustrations in the instructions didn't properly convey the context and show where the location of that tube holder was, and it had the unintended consequence of the user taking the tube and thinking that the tube holder was actually part of the device, just because of the view that was shown in the illustration looked a little bit like the sample well on the device. And the users took the tube and just kept twisting it into the sample well, and, of course, damaging the device. And so what was just such a minor design feature, and so well-intentioned when it was implemented, really was a serious issue with the device. And nothing about that was obvious to the development team beforehand, and it would have gone undetected until the clinical study, had the formative study not been conducted in that case. And so, to me, that just illustrates the benefit of doing even small, focused human factors activities early in development, they can uncover the kinds of issues, or these types of issues, that are relatively easy to address, and ensure that the clinical study is evaluating clinical performance and not these avoidable use-related issues.

Emily: Thank you so much. And people are still the most interesting animal out there, I think.

Karen: Yeah, I mean, in human factors, it's not the user, it's the design that somehow communicated to the user an incorrect action.

Emily: Well, Karen and Missy, thank you both for joining this first podcast. You've definitely both put the human in human factors for me, and the expert in Expert Insights, and really appreciate it.

Missy: Thank you for the conversation.

Karen: Thank you for having us.

Emily: This conversation is part of a broader series on the site's view of IVD clinical research, where we look at different stages of a study and the realities sponsors don't always see. If you're planning an IVD clinical study and want to talk through any of this, you can reach us at DCNDx.com/contact. Thank you for listening to Expert Insights.