Scientific Approaches to Generative AI at the SciPy Conference

Co-author: Michael Zargham

In a  previous post, we covered the SciPy Conference, which took place in Minneapolis, MN, USA in July 2026. In this blog post, we’ll talk in depth about our Humane Intelligence Birds of a Feather (BoF) session,  Building Scientific Approaches to Generative AI. BoFs are interactive sessions, often panel discussions or community conversations, designed to bring conference attendees together to discuss topics that are relevant to the SciPy community. In our BoF, we led a guided community discussion around AI evaluation.

SciPy Conference attendees are scientists, software developers, and data practitioners who have built the technical foundations that have enabled the explosive growth of generative AI, generative AI-driven products, and their technical ecosystems. However, unlike the scientific computing software that this community has traditionally built, generative AI sometimes lacks the rigorous quality control that is typical of powerful systems and expected of scientific endeavors. 

The SciPy community’s deep experience with scientific practice and software development grounded our conversation. Our BoF participants were both users and critics of generative AI. Rather than uncritically championing or demonizing the technology, the conversation covered requirements that need to be met in order for generative AI to be trusted in a particular situation, and how to evaluate deployed systems against those requirements, while upholding the same standards for computational science that are applied in natural systems.    

We had participants assemble into groups with a designated notetaker and  we shared 3 sets of discussion questions. Our first set set the context, reminding participants of their own experiences with generative AI and asking them to think about them through their scientific and technical expertise.

There is a Real and Urgent Need for Scientific Rigor in AI Evaluations

Generative AI is often described as something that will soon be a major part of every aspect of our lives. Our BoF participants pointed out that this is already happening. Generative AI has been firmly embedded in many common tools, interfaces, and packages that were already part of our daily lives and it is being used across domains such as finance, medicine, energy, nuclear engineering, and more. Participants also mentioned use cases such as document and image creation, writing code, and conducting analysis. While these changes are not as easily detectable to a lay audience, when asked, our BoF participants easily identified many examples of generative AI that is part of their professional and personal lives. 

With this reality, our BoF participants stressed the real and urgent need to bring more scientific rigor into AI evaluations, especially those  context-driven and human-centered.

Much Higher Performance Requirements

One of the most common vulnerabilities that we test in our AI red teaming evaluations at Humane Intelligence is hallucinations. Likewise, hallucinations were of particular concern to our BoF participants. One of them mentioned noticing hallucinated citations. The threshold of acceptable factual inaccuracies in science is extremely low by nature of the work involved. While the volume of research being produced has surged, vetting for quality using pre-generative AI standards has become unduly challenging.

The other failure modes (aside from hallucination) were doing a bad job (e.g., poorly written code that might not run correctly) and serious issues around private data leakage and broken guardrails. In terms of prevention, the discussion included simply not opting for generative AI products for use cases in which it doesn’t perform well (e.g., scheduling), bringing in domain knowledge and oversight around the domain in which generative AI is being used, having links to verifiable sources, and breaking down tasks into smaller, more tractable pieces. 

The scientific and open source backgrounds of the participants was evident. A recurrent theme was bringing in human oversight, which is aligned with scientific programming pursuits. Being able to build and run code is not enough, you need to be able to contextualize the outputs against scientific process and facts. 

Our second set of questions built on the first, asking participants to draw on their expertise and discuss what it would take to bring scientific rigor to the AI evaluation space.

AI is Skipping Valuable Processes

At Humane Intelligence, we are often at the nexus of various community, funder, and client conversations about labor augmentation, increased productivity, and reduced technical work barriers to entry that generative AI can enable. Our SciPy BoF participants were generally in agreement with these ideas, and many participants regularly use generative AI in their work. However, one insight that stood out was that generative AI can enable users to bypass decades of designing, refining, and implementing the good software development practices that have been championed by maintainers in the SciPy community. 

One participant mentioned linting as an example of something that would have made for better generative AI output. Linters have been around for decades to identify code bugs in static environments. Today, anyone can vibe code using a few prompts to create public-facing applications. While vibe coding or AI assisted coding can certainly reduce barriers to help more people output code, it also allows a user to bypass valuable processes that ensure their code is secure, sustainable, and logical. This was of particular concern to our BoF participants as members and users of open source software projects. Similar to the increase in the volume of papers, there has been a surge in the volume of pull requests to open source projects that included  generative AI-produced code. This has created the need for projects to develop new governance mechanisms to handle new pathways for accepting work into their code base. 

There was broad consensus in this discussion that the way to bring scientific rigor is to treat evaluations as a scientific process. Recommendations included stating hypotheses, making assumptions explicit, having a valid control, and ensuring that the evaluation process is reproducible and measurable. The technical requirements for scientific rigor included domain-appropriate practices, such as running automated testing and code inspections for AI-generated code, as one would do with human-written code. Another topic that was brought forward was about not only acknowledging the non-deterministic output of generative AI, but creating contextually-appropriate frameworks. In general, evaluation was seen as a technical requirement, with humans in the loop and automated methods being mentioned. Similar approaches have been taken in non-AI software engineering, systems analysis, and qualitative research. 

Our final topic was the most challenging and provided the most ambiguous answers. We touched on measuring the impact of a system or how someone can tell that a system is working in the real world where it’s being used.

The discussions that ensued elucidated verification (did we build it right?) versus validation (did we build the right thing?).  Verification covers correct construction. However, generative AI systems don’t exist in a vacuum; they are deployed and used in specific circumstances to fulfill specific purposes.  Validation tells us whether a deployment is appropriate for the situation in which it is deployed. Context sensitivity and human-in-the-loop were discussed as important validation components. End-to-end real world testing was mentioned as a way to probe efficacy. However, this is still an unresolved question. Impact measurement is tricky and often the outcomes that you are trying to move are challenging to measure and attribute directly to prior efforts. 

Our BoF yielded important discussions,  even though there are still many open questions. As Humane Intelligence and DSG build our approach to using ontologies for contextual AI evaluations, we were encouraged to hear the approach we are developing is strongly aligned with the advice the SciPy community gave. It reinforced several ideas that guide how we will approach creating AI evaluation tooling that enables scientifically rigorous evaluation:

• Evaluations need to be contextualized in domain knowledge and should follow the scientific method with a declared question, stated assumptions, and reproducible process
• Coverage should be measurable 
• Domain experts, i.e., humans, need to be part of this process

Practitioners should walk away from an evaluation with a scientifically-backed understanding of whether or not they trust a generative AI system.

These takeaways give us a clearer path forward, even with open questions, and they were made possible by the conversations we had in the BoF. Thank you to everyone who attended this session and provided input into this discussion. If you’d like to stay in touch, we invite you to stay connected with us by following us on LinkedIn, keeping an eye on our Insights, or writing to us.