#
Why a Strong Diagnostic Test May Add Little New Information
[Chat GPT 6]
My colleague Luis Cano recently posted a useful example of a question that diagnostic developers should ask much earlier: What does the new technology add to what we already know?
His post described a pancreatic neuroendocrine tumor preprint involving 1,438 patients and approximately 30,000 histology images. An AI model extracted prognostic information from routine H&E slides.
Yet, as Cano reported, a model using established clinicopathologic information achieved a C-index of 0.875; adding the AI increased it to 0.883. [Sivasubramanian]
The AI could predict recurrence. The clinical information could already predict recurrence very well. Those findings can coexist.
My comment on his post drew on a lesson I remember from MBA statistics. Given an outcome and ten candidate variables, variables A, B, and C might predict the outcome extremely well. Then variables D, E, and F might do almost as well. That does not mean we have discovered two separate reservoirs of predictive power that we can simply add together. We may have found two ways of measuring much of the same thing.
For molecular diagnostics, digital pathology, and AI, this distinction can separate an impressive publication from a compelling case for clinical adoption.
The arithmetic works but the attribution changes
We expect addition to be commutative: 3 + 5 = 5 + 3. It is tempting to assume that combining predictors works the same way.
An important qualification makes the lesson precise. Ordinary correlation is symmetric: the correlation of X with Y equals the correlation of Y with X. And if we fit the same ordinary regression model to the same data with the same predictors, simply rearranging the columns does not change its final predictions.
- What depends on order, is the incremental credit we assign to each source of information.
“How well does this test predict recurrence?” and “How much does this test improve prediction after we use the pathology report?” are different questions.
The first measures performance in isolation. The second measures added value against an existing baseline. A test can perform very well on the first and make only a modest contribution on the second.
Here, “clinical first” means choosing the clinical model as the benchmark.
Two impressive predictors may share the same information
Tumor size, grade, stage, and nodal involvement have long informed prognosis. Simple Ki-67 rated low-medium-high can also provide any pathologist with prognostic information. Gene expression signatures [MAAA tests] may capture related biology through another measurement method.
These are not interchangeable measurements, and their overlap must be studied in each intended population. Nevertheless, there is nothing surprising about a proliferation signature and a histologic assessment of aggressive growth predicting the same recurrences.
Imagine two observers examining the same underlying disease behavior. One describes it through morphology and anatomy; the other through molecular expression. Both may be accurate. Their agreement does not create two independent quantities of information.
Technological sophistication does not establish informational novelty. Measuring familiar biology with greater precision may be useful, but that usefulness must be demonstrated against the information already available.
A numerical example with R²
Consider an illustrative ordinary linear regression example. These are invented teaching numbers, not recurrence-study results.
| Information used | R² |
|---|---|
| Clinical variables alone | 0.60 |
| Molecular score alone | 0.65 |
| Clinical variables and molecular score together | 0.68 |
R² measures the proportion of outcome variation explained by this kind of model.
Start with the molecular score. It explains 65% of the variation. Add clinical variables and the figure rises to 68%. The presentation practically writes itself: “Our molecular score captures nearly all the predictive information; clinical variables add only three percentage points.”
Now start with clinical variables. They explain 60%. Add the molecular score and the figure rises to the same 68%. The molecular increment is eight percentage points, rather than the 65 percentage points it appeared to contribute in isolation.
The molecular score did not lose its predictive ability. Much of that ability was already represented in the clinical model.
In this example, the molecular score contributes 0.08 beyond clinical information; clinical information contributes 0.03 beyond the molecular score. The remaining 0.57 is shared explanatory contribution under this simple decomposition.
Whichever set enters first gets credit for the shared contribution. Whichever enters second is evaluated on what it adds.
That decomposition is specific to this illustration. More complicated models can involve interactions or suppression effects, so shared contribution is not always a neat positive slice.
The same lesson applies to AUC
Now consider another invented example:
| Information used | AUC |
|---|---|
| Clinical variables alone | 0.82 |
| Molecular score alone | 0.80 |
| Clinical variables and molecular score together | 0.83 |
A marketing presentation might emphasize that the molecular score achieves an AUC of 0.80 and that adding clinical information improves performance by just 0.03.
A clinical evaluator can ask the reverse question. Clinical information already achieves 0.82. Adding the molecular score improves AUC by 0.01.
Both calculations are correct. They answer different questions.
AUC is a ranking measure, not a percentage of variation explained. An AUC of 0.80 does not mean “80% of recurrence is explained” or “80% of patients are correctly diagnosed.” Nor can we divide AUC into shared and unique portions as we did with R² above.
Nevertheless, its increment depends on the baseline against which the new predictor is evaluated. Research on biomarker evaluation explicitly demonstrates that the same candidate marker can produce different AUC gains depending on the strength of the starting model. PMC
Cano’s example uses a C-index, a related measure of discrimination applied to survival outcomes. The conceptual question is the same: how much does the image model add beyond the clinical model? C-index and ordinary binary-outcome AUC are not numerically interchangeable.
The product may already contain the clinical information
An additional complication arises when a commercial score combines a new and proprietary molecular measurement with everyday tumor size, nodal status, age, or other established variables.
That can be an entirely sensible product design. But the combined score’s performance is not automatically the molecular component’s performance.
A useful evaluation distinguishes three models:
- Established clinical information alone.
- The molecular component alone.
- Clinical information plus the molecular component.
Suppose the third model performs beautifully. We still need to know how much of that performance was available from the first.
Likewise, showing that a commercial score adds information beyond any one clinical variable is not equivalent to showing that it adds information beyond a well-constructed model containing the relevant clinical variables together.
A weak comparator can make a modest advance look large.
Three meanings of independent
“Independent” is particularly slippery in diagnostic marketing. At least three distinct claims can hide behind it.
| Marketing Claim | What it actually asks |
|---|---|
| The test is uncorrelated with clinical variables | Do the measurements vary together in the association being examined? |
| The test has an independent association with recurrence | Does its association with outcome persist after appropriate adjustment? |
| The test adds useful predictive information | Does including it improve validated predictions enough to matter? |
These are different questions. A significant adjusted association does not by itself establish a clinically meaningful improvement in prediction. Conversely, correlation with existing predictors does not automatically make a test useless.
Even a genuine absence of pairwise linear correlation would not establish statistical independence: nonlinear relationships may remain. And checking correlations against clinical variables one at a time does not settle what their joint model already captures.
The clinically relevant analysis evaluates the proposed test within the information environment in which it will actually be used. PMC
The p = 0.055 maneuver
A particularly misleading statement is: “The molecular score had no correlation with the clinical variables.”
The supporting result turns out to be p = 0.055.
That result does not establish no correlation. It says that the particular statistical test did not cross a chosen 0.05 threshold. The distinction matters because “no correlation” can become the premise for a much larger claim: that the test contributes entirely new information.
Failure to demonstrate an association at a selected threshold is not demonstration that the association is absent. And it may be pretty strong.
And a p-value does not measure the strength of the correlation. That requires the estimated association and its uncertainty, interpreted in the context of sample size and study design. The American Statistical Association explicitly cautions that p-values do not measure effect size or importance. doi.org
For a concrete illustration, under a conventional two-sided Pearson correlation test:
| Sample size | Observed correlation r | Approximate p-value |
|---|---|---|
| 10 | 0.622 | 0.055 |
| 1,000 | 0.061 | 0.055 |
The same p-value can accompany a large observed correlation in a tiny sample or a small observed correlation in a large sample. These are calculated illustrations, not clinical findings.
Thus, a strong observed correlation (like 0.6220 can indeed be described as “not statistically significant” and then misleadingly repackaged as “no correlation.” To assess the actual finding, ask for r, its confidence interval, the sample size, and the method used.
And p = 0.049 versus p = 0.055 does not mark a line between “associated” and “unrelated.”
Why clinical reviewers can miss the problem
These problems can survive clinical peer review because the individual analyses may be competently performed while the central inference remains unsupported.
A paper may show a significant association between the molecular score and recurrence. It may show an attractive ROC curve. It may report that a correlation with a clinical variable did not reach statistical significance. It may then show that adding clinical variables to the molecular score makes little difference.
None of those findings, individually or together, necessarily answers the question: What does the molecular test add after we use the clinical information already available?
A reviewer can scrutinize the patient cohort, pathology, endpoint definitions, and laboratory methods while accepting the authors’ framing of the comparison. An unmet evidentiary need does not have to look like a calculation error.
Publication therefore does not relieve the reader of examining the baseline model and the direction of the comparison.
Why evaluators like MolDx begin with clinical information
For an add-on test, an evaluator such as MolDX has a practical reason to start with established clinical information. The clinician already has it. The proposed purchase is the additional test.
MolDX’s EndoPredict coverage policy provides a concrete example of this reasoning: it describes a score combining molecular and clinicopathologic information, and discusses evidence of prognostic information beyond clinical and pathologic features alone. cms.gov [test is covered]
The evaluative question is therefore not whether the new test can predict recurrence from a standing start. It is whether the test improves the decision facing a clinician who already has a pathology report.
“Clinical first” gives the baseline information credit because it is already part of the relevant care pathway. The reversed analysis can be mathematically legitimate while answering a different commercial or scientific question.
An assertion of independent association is also only one step. The next questions concern the size and reliability of the predictive improvement, then whether that improvement supports better decisions.
A small increment is may still be enough
A small overall increase in AUC or C-index should prompt examination, not an automatic declaration that the test has no value.
A marker might improve risk estimates near a treatment threshold without greatly altering rankings across the entire cohort. It might help a prespecified subgroup or offer more reproducible measurement. Each possibility needs evidence.
Conversely, an impressive AUC does not show that predicted absolute risks are accurate, that management should change, or that patients benefit.
A fair evaluation therefore uses the same intended population, endpoint, prediction horizon, and validation patients for the competing models. It gives the clinical comparator appropriate modeling attention, guards against overfitting, reports uncertainty in the increment, and examines calibration and decisions at clinically relevant thresholds.
The question is not simply how much higher the curve rises. It is what the additional information allows us to do better.
A strong association establishes that a test knows something about the disease. Demonstrating added value establishes that it tells the clinician something useful beyond what the clinician already knows.
Before admiring the new test’s predictive power, ask how much of that power was already in the chart.
##
##
Behind the blog. Prompt.
I prompted with this instruction and also uploaded Cano article and my comment. GPT Sol 6.1 medium in /work mode.
my colleague luis cano wrote this linked in article and you see my comment. Use this as an intro. Then, write a sort of mini class on how correlation statistics like R^2 or AUC are not indifferent to order of operations. Tumor size and grade are strong predictors of recurrence and aways have been. Ki67 molecular experssion may also be strong - but not independent. Marketing likes to show that clinical adds little more to the molecular propretiary supertest, but if you do CLINICAL first, then the fancy molecular test may add little more, certainly, much less than if it's used in isolation. (there's also the canard that they will claim that clinical variables and new molecular test "have no correlation" because the corelation is p=.055 (actually reflecting a damn strong correlation). plese write this up as a teaching essay to the (surprising to some) non commutabililty of correlation studies like these. Your audience is savvy about diagnostics but NOT savvy about points as subtle as this, they do know what p values are, what AUC looks like. They'll ready your essay and feel like they've really learned something. As I mention evaluators like MOLDx are hawklike interseted in correlations residual AFTER clinical variables, while your marketing department insets that clinical variables (which the test incorporates) add little and there's no overall correlation, a misstatement of what p .055 means.