Living under the ROC

To diagnose a condition, physicians often resort to different tests. The performance of each of these diagnostic methods can be measured using different characteristics:

  • Predictive value is the probability of correctly identifying a subject's condition given the test result. ✅

  • Youden's J is the likelihood of a positive test result in subjects with the condition versus those without the condition —probability of an informed decision. 👀

  • Sensitivity is the proportion of people who actually have a target disease that are tested positive (true positive or detection rate).📍

    A negative result in a test with high sensitivity can be useful for ruling out disease, because it has a lower type II error rate.

  • Specificity is the proportion of people who do not have a target disease that are tested negative (true negative rate). The false positive rate (1 − specificity) is the probability of false alarm. 🚫

    A positive result in a test with high specificity can be useful for ruling in disease, because it has a lower type I error rate. 
When used to compare diagnostic accuracy, this enables the choice of the best test or combination of tests for a specific purpose.
 
Example: Rapid antigen tests for SARS-CoV-2 detection have 100% specificity and lower sensitivity compared to RT-PCR detection (see here). 🧪

One of the challenges in interpreting the results of diagnostic tests is that they may yield continuous measures. Therefore, it is necessary to determine thresholds.🚦

Enter receiver operator characteristic curves (ROC): plots of true positives against false positives for all cut-off values. 


👆We expect the ROC curve of a diagnostic test with reasonable accuracy to be in the upper left quadrant above the reference line.  

😡 When a loose cut-off point value is applied, the point moves upward and to the right along the curve.

😊 When a strict cut-off point value is applied, the point on the curve moves downward and to the left along the curve. 

 

But this plot provides more information than just determining cut-off points.

The most commonly used parameter to compare the efficacy of two (or more) diagnostic tests or markers is the area under the curve (AUC). 💯

For any test to be statistically significant, the lower 95% confidence interval value of the AUC must be >0.5 (above the red reference line in the graph). 

However, for a diagnostic test to be clinically meaningful (meaning, for clinicians to truly base their decisions on its results), only an AUC ≥0.8 is considered acceptable.

Clinical interpretation of the value of a diagnostic test based on AUC ROC (see here).

This is also used in clinical research to measure the inter-observer variability, when two or more observers measure the same continuous variable. 👥

 

And that's it regarding study design for now. 

Next, I'll look at examples from my own work.

Comments

Post a Comment

Leave a comment!

Pride and Respect

Image
Pride month just finished... So why is this relevant for academic publishing? I have a conflict of interest, as this is personal to me. This also means I feel obligated to clarify, educate, and mitigate biases whenever possible.  Sex refers to a set of biological attributes in humans and animals that are associated with physical and physiological features, including chromosome expression, hormone function, and reproductive/sexual anatomy. Gender refers to the socially constructed roles, behaviours, and identities of female, male, and gender-diverse people. It influences how people perceive themselves and each other, how they behave and interact, and the distribution of power and resources in society. Terms that define identity Apart from gay men, lesbians, and bisexual people, it is necessary to recognize, Asexual : An individual who experiences a lack of attraction to other persons of any sex or gender. Pansexual : An individual who experiences attraction to people regardless of g...
Read more

London Book Fair 2024: OA and AI

Image
In a survey on the trends of academic publishing conducted in 2023, reflecting on the need to reformulate the current business model was front of mind. The need to optimize operational efficiencies may stem from the increased adoption of open access (OA) policies. For those not in the know, OA is a practice that enables research outputs to be distributed online, free of access charges or other barriers, including the ability for copy or reuse, by applying an open license for copyright. The mandates  advocated by the Holdren Memo in the US, the Nelson Memo in the UK, and Plan S for the EU consolidated the view that unrestricted access to research publications is part of an integral right for transparency—we want to view the outcomes of taxpayer-funded research. However, many publishers follow a gold OA model and charge an article processing charge (APC), which is typically paid through institutional or grant funding. This means that readers don't pay, because t...
Read more

London Book Fair 2024: challenges for publishers

Image
I went to London with a purpose. It’s always good to meet collaborators and friends, but I had other goals with my visit. You see, it’s never good when the decisions that impact us are arrived at without our input. I wanted to bring awareness of my new peer review standards project to those in the publishing industry that can benefit from them. I'm comfortable researching and reviewing, even editing and managing the peer review process, but hearing editors and publishers speak of the big picture issue was an eye-opener. Change is ever present The publishing industry fancies itself as having evolved to explore the heritage and lived experience of minorities, by more frequently placing a spotlight on hidden and undervalued communities. Jonathan Karp, who spoke on the current model of employee ownership that underpins his role as CEO at Simon & Schuster , perfectly exemplified this—with profits now being reinvested in the company, there’s a new collaborative...
Read more