AI Contextual Evaluations

One of the most common questions Humane Intelligence clients ask is, “How do I know if an AI model or system performs well in a particular use case?” When we unpack this statement, we see that accounting for AI contextual evaluation complexities can be difficult. Guessing which scenarios to test in an AI red teaming workshop or examining underlying data without a clear direction is insufficient to capture the breadth of possible failure points. That’s why our OB-CAIE methodology first defines a problem space before creating the evaluation.

With our OB-CAIE methodology, Humane Intelligence is helping clients better understand their problem space coverage and gaps, and how to use compute and resources more effectively. This approach creates more mathematical and statistical rigor and clarity around AI model and systems performance, evaluation replicability and limitations, and ultimately, better product go/no-go decisions. 

Every AI contextual evaluation is different! While there’s no single way to determine which is better, here’s a quick guide to decide whether a taxonomical or ontological based methodology is more appropriate:

Taxonomies are better for:

• Smaller AI evaluation budgets
• AI red teaming workshops that are primarily for educational purposes
• Rapid, one-off evaluations
• Well-defined and well documented problem spaces

Knowledge graphs / ontologies are better for:

• Evaluations that build or expand over time
• Testing scope reproducibility and comparability
• Teams with a high degree of subject matter expertise
• Complex or high-stakes problem spaces that are constantly evolving