What counts
as evidence?
Welcome to a special issue of the BBSW Community Newsletter, dedicated to our 2026 Annual Conference. What a two days it was. On November 5–6, around 450 of you filled the Crowne Plaza in Foster City for The Adaptive Data Scientist: From Data to Impact: two keynotes, a featured AI panel, eight sessions with more than 38 speakers, a poster session, partner demos, and a great many conversations in between.
Thank you to every speaker, panelist and session chair who shared their work so generously, to our poster presenters (and congratulations to the award winners), and to our 52 sponsors and partners. A special thank-you to conference chair and BBSW president-elect Ning Leng, BBSW president Xiaolong (Ron) Yu, and the organizing committee, Jing Du, Jingjing Gao, Yannan Tang and Anqi Zhu, and all of our volunteers, for another successful conference. BBSW has been volunteer-run since 2018, and this event was another reminder of what this community can build together.
Warm regards,George Wu, Andy Shen, Aida Yazdanparast · on behalf of BBSW
The Conference
Report.
Anyone who has worked in drug development knows the moment. The analysis is finished, the slides are polished, the meeting is well under way, and someone at the end of the table asks, quite reasonably, whether the result actually supports the decision. It is an excellent question. It would have been an even better one at the first meeting, which is where that person ought to have been invited all along.
At BBSW 2026 the question kept turning up, each time in slightly different clothes. Sometimes it asked how much checking an AI-generated analysis needs before anyone should trust it. Sometimes it asked when external data can genuinely improve a trial, or what makes a prediction useful enough to change a development plan. The wording varied, but the underlying worry did not: what, exactly, counts as evidence?
For biostatisticians and statistical programmers, these are no longer occasional philosophical puzzles but ordinary weekday decisions. The speakers offered concrete ways to make them and, just as usefully, several good reasons to keep asking awkward questions early, while the answers can still change something.
The audience, as it turned out, had been asking much the same thing. During the Featured AI Panel, chaired by Ruixiao Lu with Judy Li, Shaobo Liu, Hao Zhu and Scooter Plowman, live polls asked the room where it stands with AI. Adoption proved broad but uneven. GenAI is now a daily tool for code and literature review, but far less so for regulated outputs, and the barriers people named were about process and validation much more than cost: i.e., another question of evidence.
With validation and regulatory acceptance so near the top of that list, the panel’s regulatory perspective was especially welcome. FDA is actively seeking partnership with industry to advance innovation in statistics and AI-related health policy, led by Hao Zhu, Director of FDA’s Division of Pharmacometrics. The talks, as it happens, spent much of their time on exactly these barriers, and on what happens when AI hands off its work to humans.
The first thing to say about an AI-generated specification or report draft is that it can be ready remarkably fast, certainly in time for a demonstration. The second is that getting it ready for a study team takes rather longer, because somebody still has to check how it interpreted the protocol, reconcile it with the data and put their name to the result.
That less glamorous stretch of work is where several speakers chose to spend their time. Yun Bai and Yuyang Du described a literature workspace whose requirements grew as colleagues began to question it, starting with the deceptively simple matter of who checks what the system screens out. Ning Leng argued for domain workflows built on established computational tools and tested against realistic benchmarks, and Joseph Paulson drew a firm line between a demonstration that works and a capability an organization can actually rely on. The talks were different, but the problem was shared: how to find the errors, and then how to make the whole process repeatable.1, 2, 3
Statistical programmers already know most of what this requires. A table has long needed a specification, an executable derivation, identifiable inputs and a record of review, and CDISC’s Analysis Results Standard offers a structured way to carry results and their metadata through exactly that process.4
The catch is that agreement and correctness are not the same thing. Two tables can match perfectly because both use the same wrong denominator, and two programs can faithfully reproduce the same mistaken reading of the protocol. Agreement is reassuring, but correctness is the point. Useful verification therefore has to test the scientific specification itself, including the cases where different populations ought to produce different counts.
By the same logic, review needs defined checks. A signature records that someone approved the output, but it says nothing about which errors that person was in a position to catch. Martha Cao’s reflections on adopting AI at a small biotech raised exactly these questions of validation, traceability and sign-off. Nor does regulatory language offer an easy way out: the FDA’s draft guidance on AI excludes certain operational efficiencies only when they do not affect safety, quality or reliability, so calling a workflow “operational” does not settle its consequences.5, 6
All of which means review time belongs in any honest productivity estimate. A faster draft is valuable if it helps the team reach an accepted result with less total work. If it does not, the time has not been saved, only moved to the reviewer.
Faster drafting is one promise of AI. A bolder one is that it will improve the analysis itself, which made it all the more charming that one of the conference’s quiet success stories was linear regression.
In Bingkai Wang’s benchmark of 50 randomized trials, none of the machine-learning approaches evaluated improved the precision of covariate adjustment over plain linear regression, a method that had apparently neglected to read the announcements of its own retirement. The finding has honest limits. The trials were publicly available, the comparisons were correlated and the machine-learning methods were not tuned, all of which leaves room for other methods to do better in other settings.7
The expanded analysis was more revealing still. Informative baseline text could improve precision, whereas running structured covariates through a language model added little. What seemed to matter, in other words, was not the sophistication of the tool but the information it made available.7 Lilia Bouzit’s digital-twin work offered a second reason to take simpler models seriously: in external validation on trial data, pooled logistic regression held up well against more complex alternatives.8
None of this makes new information worthless; quite the opposite. Daniela Ferrara showed what it can look like in ophthalmology, where retinal images hold measurements and patterns that a conventional analysis table tends to leave out, and where AI can help extract them.9
The trick is to keep the value of new measurements separate from the choice of model. A clean way to do that is to compare three things: the current model, the same model given the new information, and a more complex model using that information, all evaluated in a comparable validation setting against the same performance criteria.
And even a very good prediction has a limited remit: knowing which patients are likely to do poorly is not the same as knowing which treatment will help them.
Several presenters showed how much prognostic signal can be found in unexpected places. Petr Jordan extracted prognostic information from routine CT scans, Swati Kaushik linked a molecular subtype to poor prognosis and to spatial biological hypotheses, and Xiaoyong Fu traced molecular changes across treatment trajectories. Each approach can reveal patterns well worth pursuing. Yet an association with outcome, however striking, does not establish that a treatment benefits some patients more than others. That distinction, between prognostic and predictive biomarkers, is set out formally in the FDA–NIH BEST resource.10–13
Happily, not every useful decision requires a treatment-benefit prediction, and real-world data can earn its keep in more modest ways. Nayan Chaudhary’s breast-cancer example used it to examine a restrictive eligibility definition. Broadening that definition raised the estimated eligible fraction from 41% to 51.7%, while observed outcomes and mutation prevalence fed into the design discussion.14
It is worth being precise about what that number means. The roughly 26% relative increase describes the estimated eligible population, not an observed improvement in recruitment, since sites still have to find, screen and enroll the patients. What the analysis gave the team was a sound basis for considering broader eligibility while checking that the trial’s assumptions would still hold.
Wenjuan Gu’s MRD proposal took on a harder question: whether evidence supporting an intermediate endpoint in one myeloma setting carries over to CAR T-cell therapy. Published-trial synthesis, real-world mediation and internal prediction could each contribute a different piece of the answer, but only if they show which assumptions they share and how any disagreement among them would be resolved. Otherwise, three analyses with the same blind spot may simply have formed a very agreeable committee.15
The broader lesson is that prognosis, eligibility and endpoint transfer each call for a different validation study. The intended use even changes how the data are prepared, from which patients qualify to when measurements become available and which outcomes need to be derived.
“Digital twin” is a wonderfully efficient phrase: two words, several possible meanings and almost always comes with a promising slide. Jingyuan Yang’s overview spanned uses from prognostic adjustment all the way to in-silico trials, which is exactly why a team evaluating such a model needs to say which of those uses it has in mind.16
The difference is considerable. A baseline prognostic score can help sharpen precision within a randomized trial, but replacing or supplementing a comparator asks far more of both the model and the data. Historical patients may have received different supportive care. Registries may measure progression differently. And the patients with repeat scans or biopsies are, almost by definition, the ones who stayed alive and observable long enough to have them. A neatly harmonized dataset, in short, does not mean the underlying circumstances have agreed to cooperate.
Target-trial thinking helps by making the design explicit: who is eligible, which treatment strategies are being compared and when follow-up begins. That discipline can prevent avoidable timing and selection errors, though it cannot conjure up confounders that were never measured.
Bouzit’s work, to its credit, tested predictions in trial populations separate from the development data. Even so, agreement with observed survival does not establish an unbiased counterfactual for every treated patient, because good average calibration can coexist with poor performance in precisely the subgroup that drives the decision.8
This is why the more cautious designs drew attention. Ruben van Eijk emphasized augmenting concurrent controls rather than replacing them, so that randomized observations remain available as a yardstick for judging the external information. Austin Hand turned to the borrowing procedure itself, including what happens when the external data fail the compatibility test.17, 18
Taken together, these talks made the case for evaluating the full procedure: tracking how external information changes the treatment-effect estimate, its uncertainty and the chance of reaching a wrong conclusion.
Jessica Overbey’s illustrative confirmatory design put numbers on both sides of the trade: what borrowing external evidence buys, and what it costs. At a true hazard ratio of 0.85, dynamic borrowing raised power from 61.4% to 80.2%, the kind of improvement that makes a design team sit up. The receipt, however, had a second line. In a scenario where the new trial had no effect despite favorable prior evidence, the false-positive rate was 7.7%, compared with 2.5% without borrowing.19
A team choosing this design would have to weigh both figures against how credible the prior evidence really is. Scaling back borrowing under strong conflict does not guarantee nominal frequentist error control across every degree of conflict. In basket trials, likewise, checking only the all-null scenario can miss the problems that appear when some cohorts benefit and others do not.
James Travis and Mark Rothmann laid out what regulators expect from such evidence. The FDA’s draft guidance on Bayesian methods addresses prior justification, success criteria and operating characteristics, while its revised draft on substantial evidence of effectiveness discusses how a trial plus confirmatory evidence may together support an evidence package. Neither document, it should be said, makes the relevance or independence of supporting evidence automatic.20–23
Meeting those expectations is a job for statisticians and programmers working side by side. The statistician specifies the plausible conflict scenarios and the decisions to be made at each look, and the programmer implements every interim, selection and fallback branch. The final formula can be perfectly correct while the simulated trial takes the wrong route to reach it, and the code, unfortunately, will not send an apologetic email.
Other efficient designs come with conditions of their own. Hongjian Zhu examined inference after treatment selection in seamless trials, and Jung Yeon Won showed how a rare-disease crossover design depended on reversible effects, high retention and credible assumptions about carryover. Calling a trial seamless does not remove the seams; it just makes checking them somebody’s job.24, 25
A design recommendation, then, should arrive with its conditions attached: when it works, when it fails and why the team is willing to accept the tradeoff.
Even a well-supported recommendation can arrive too late, or answer a question nobody at the table was really asking. The leadership sessions took on this problem directly, asking how quantitative experts can help shape a decision while there is still time to change it.
Patrick Loerch described roles that now stretch across traditional data-science boundaries. Conny Irl and James Reimann urged statisticians to understand the clinical and commercial choices their analyses support, and Marla Curran showed what that can look like in practice: using AI to help prepare scenarios so that a team could agree on its decision rules before the readout.26–28
Jingyuan Chen described similar responsibilities for programming leads. Chen’s cases involved addressing colleagues’ concerns about new methods and organizing shared work across studies, and in each one, agreement depended on practical answers about support, timelines and contingency plans.29 That kind of negotiation rests on listening, the skill Richard Zink explored through applied improvisation. For those of us who have occasionally rehearsed a rebuttal while a colleague was still asking the question, it is a more demanding skill than it sounds.30
The partner demonstrations, meanwhile, gave teams no shortage of tools to evaluate, and the full list of demos from Day 1 and Day 2 is in the conference app. Whatever the tool, when a vendor describes an AI workflow, an agent or a design optimizer, it is worth asking for the intended use, reference cases, change controls and reproducible exports. Better still, ask to run a representative study problem, and inspect the failures as carefully as the successful output.
For a first evaluation, choose a task the team already understands well enough to check independently, then compare the new workflow with current practice, counting review and rework in the time recorded. The result should tell the team whether to adopt the tool, narrow its use or keep experimenting.
And if the tool does free up capacity, decide deliberately what to spend it on. It might fund the sensitivity analysis that keeps getting deferred, an earlier design discussion or a programmer’s contribution to the analysis specification. Saved time left unassigned tends to vanish quietly into the calendar; given a job, it can make the evidence better.
Adapt the work.
Expand the impact.
The adaptive data scientist, as these two days sketched them, learns new methods, checks where they work and brings that knowledge into development decisions early. Faster tools, like the AI that more of us now use every day, earn their place when they give a team better evidence, or more time to think through its choices.
Start with the decision. Choose tools and methods around a real development question, not the other way round.
Make the claim precise. Prediction, treatment-effect estimation and reliable production each need a different kind of evidence.
Test the whole process. Compare with credible alternatives, examine what happens when assumptions fail, and count verification in the measure of value.
Use the capacity to contribute earlier. Bring statistical and programming expertise into design, evidence strategy and cross-functional decisions before the slides are ready for approval.
For those of you who are wondering: yes, this report was written by our AI reporter with human editors.
- Yun Bai & Yuyang Du. Ask, and Be Asked: Building an AI Community by Inviting Others to Shape the Work. BBSW 2026.
- Ning Leng. Beyond the AI Demo: A Community Effort to Build Pharma Skills and Benchmarks. BBSW 2026.
- Joseph N. Paulson. Beyond the AI Proof of Concept: Building a Center of Data Excellence for Drug Development. BBSW 2026.
- CDISC: Analysis Results Standard. Official standards documentation.
- Martha Cao. Reflections from a Small Biotech: Adopting AI in Data Preparation. BBSW 2026.
- FDA: Considerations for the Use of AI to Support Regulatory Decision-Making. Draft guidance.
- Bingkai Wang. When Prediction Meets Randomization: Regression, Machine Learning, or LLMs? BBSW 2026.
- Lilia Bouzit. Leveraging Real World Data for a Digital Twins Framework, Validated in the Clinical Trial Setting. BBSW 2026.
- Daniela Ferrara. From Data to Impact: Artificial Intelligence to Transform Drug Development. BBSW 2026.
- Petr Jordan. Real-World Imaging Biomarkers in Oncology: Onclara IO and Applications in Drug Development. BBSW 2026.
- Swati Kaushik. Multimodal AI Identifies a High-Risk, Immune-Cold NF1-Mutant sqNSCLC Patient Subtype Enriched by a 31-Gene Prognostic Signature. BBSW 2026.
- Xiaoyong Fu. Real-World Clinical Multi-Omics Analyses Reveal Evolutionary Trajectories for Drug Resistance in Non-Small Cell Lung Cancer. BBSW 2026.
- FDA–NIH BEST: Prognostic versus Predictive Biomarkers. Official biomarker terminology resource.
- Nayan Chaudhary. RWD/E for Trial Design: Expansion of Trial Eligibility for a Phase 3 Trial Using Clinico-Genomic Real World Data. BBSW 2026.
- Wenjuan Gu. Evaluating Intermediate Endpoints for CAR T-Cell Therapy in Multiple Myeloma: A Multi-Source, Multi-Method Analytical Framework. BBSW 2026.
- Jingyuan Yang. Digital Twins for Clinical Development: From Prognostic Adjustment to In-Silico Trials. BBSW 2026.
- Ruben van Eijk. Digital Twins and Synthetic Controls: From Concept to Regulatory-Grade Evidence. BBSW 2026.
- Austin L. Hand. Augmenting Randomized Clinical Trials with External Data: Opportunities and Limitations. BBSW 2026.
- Jessica Overbey. The Design of Bayesian Confirmatory Trials. BBSW 2026.
- James Travis. Overview of the FDA Draft Guidance on Bayesian Methods in Clinical Trials of Drug and Biological Products. BBSW 2026.
- Mark Rothmann. Substantial Evidence of Effectiveness: 2026 Revised Draft Guidance. BBSW 2026.
- FDA: Bayesian Methodology Guidance Recap. Official explanation of the draft guidance.
- FDA: Demonstrating Substantial Evidence of Effectiveness. Revised draft guidance, June 2026.
- Hongjian Zhu. Investigation and Recommendation of Analysis Approaches for Adaptive Seamless Phase II/III Clinical Trials. BBSW 2026.
- Jung Yeon Won. Crossover Design in a Rare Disease Setting: Statistical and Practical Considerations, a Case Study in Aqneursa. BBSW 2026.
- Patrick Loerch. Data Science in Transition: The AI-Driven Evolution of Roles in Biopharma. BBSW 2026.
- Conny Irl & James Reimann. Power Players: Statisticians as Strategic Drug Developers. BBSW 2026.
- Marla Curran. All Sand, No Rocks: Creating Capacity Without Surrendering Judgment. BBSW 2026.
- Jingyuan Chen. Influencing Across Complexity in Projects. BBSW 2026.
- Richard C. Zink. Statistics and Applied Improvisation: Strange but Necessary Bedfellows. BBSW 2026.
Got something
to share?
A reflection on the conference, a speaking idea for next year, a resource: if it would help this community, we want to hear it.
Contact Us
