The ranking that did not survive a new seed

On a Sunday in late May an audit of my own code came back with a list of small fixes. One of them added a single argument to the function that splits the data.

In April the table had a clear winner. We were comparing ways to recalibrate a mortality model for patients hospitalized with COVID-19 in Brazil: 6,011 patients from fourteen hospitals, nearly one in four of whom died before discharge. The question was whether the predicted risk meant the same thing for an old patient and a young one, for a man and a woman, in the North and in the South. To rank the methods we built a composite score that weighs calibration, the gap in calibration between subgroups and any loss of discrimination. A multicalibration method built on gradient boosting came first. That family of methods is what my PhD is about.

The argument from the audit spread deaths evenly between training and test. With it in place and nothing else changed, the same method went from first to last. Running it with a different set of library versions moved it to second. When I tried to reproduce the April table, I could not, because nobody had written down which library versions produced it.

That was my mistake. I had read a ranking off a single random partition of a cohort of moderate size and treated it as a finding. I had also skipped the most boring part of the job, recording the environment, so even the wrong number could not be checked.

The size of the regions explains part of it. The North contributed 294 patients and 54 deaths from two hospitals. With a fifth of the data held out for testing, the North's calibration error was computed on about sixty people and eleven deaths, sorted into ten bins of predicted risk. Two or three deaths landing in one bin or another are enough to move that number, and the composite score inherits every bit of that noise. Across partitions, the North's error for the uncalibrated model averaged 0.107 with a standard deviation of 0.036. A single split tells you where one draw landed.

Most clinical prediction papers still report one split, and I understand why. It is the default in every tutorial. Reviewers ask for the best number. Repeating the whole pipeline dozens of times used to cost days of computing and a lot of patience, and a distribution makes a less exciting table.

The computing is no longer the obstacle. An agent rewrote our comparison to run over forty partitions, loading the original methods without rewriting them. With the residual of the boosting correction on the logit scale, temperature scaling had the best mean score, 0.636, and finished last in 5% of partitions. The boosting method had the worst mean, 0.449, finished last in 40% of them, and beat temperature scaling in 30% of paired comparisons (95% interval, 18% to 45%). It widened the calibration gap between subgroups, compared with the uncalibrated model, in 65% of partitions.

There are limits to what this proves. Forty partitions of the same cohort are still one cohort, collected during one pandemic, with the same hospitals and the same gaps in the records; it is no substitute for external validation. And the verdict on the boosting method depends on an implementation detail. With the residual on the probability scale, it ties with temperature scaling and wins 48% of the paired comparisons. A method that looks bad or acceptable depending on one line of code is not a method I can recommend, but I also cannot say the idea behind it is wrong.

The method that lost belongs to the family my thesis is about. The decisions from that week are on record. The logit scale became the primary analysis and the probability scale a sensitivity check. The multicalibration method I built myself still has to be run across all forty partitions before I say anything about it.

What changed in practice is easy to list. No number in the manuscript comes from a single run anymore. Every table is regenerated from the forty partitions by a script, and the library versions are saved next to the results. The methods are compared by mean, standard deviation and paired win rate with an interval, and the dashboard shows how often each one finished first or last. The paper now leads with the least glamorous method in the comparison, because it was the most consistent.

Forty partitions are cheap now. Report all forty.