Humane Intelligence Algorithmic Bias Bounty
CHALLENGE 1
Stop bad LLM output before it happens!
Humane Intelligence is thrilled to be launching the first of 10 "algorithmic bias bounty" programs, which will be unfolding over the coming year. With the support of Google.org, we are building themed programs that aim to build community and professionalize the practice of algorithmic assessment.
This challenge is now closed. Thank you to all who participated.
Bias Bounty 1 Winners!
We are thrilled to announce the winners of our very first Bias Bounty Challenge! This challenge was designed to fine-tune automated red teaming models and explore issues like bias, factuality, and misdirection in Generative AI.
A huge thank you to all participants who contributed their expertise and congratulations to our winners!.
We are so proud of the innovative approaches and dedication shown by all participants.
Advanced:
Yannick Daniel Gibson (Factuality)
Elijah Appelson (Misdirection)
Gabriela Barrera (Bias)
Intermediate:
AmigoYM (Factuality)
Mayowa Osibodu (Factuality)
Simone Van Taylor (Bias)
Beginner:
Blake Chambers (Bias)
Eva (Factuality)
Lucia Kobzova (Misdirection)
The Challenge
Our first challenge builds on the evaluation and dataset from our Generative AI Red Teaming Challenge: Transparency Report. Challenge participants can win from a pool of about $10,000 in prizes for beginner, intermediate, and advanced submissions.
The final task? Create a probability estimation model that determines whether the prompt provided to a language model will elicit an outcome that demonstrates factuality, bias, or misdirection, inspired by the data from our Generative AI Red Teaming Challenge: Transparency Report.
Regardless of your skill or ability in algorithmic assessment, we have a competition for you.
Beginner
Pick one of the three datasets. Identify gaps in the data and suggest new categories of data that would make the dataset more representative. Generate five prompts per subject area that will elicit a bad outcome. You will be graded both on the number of new topics as well as the diversity of the prompts produced.
Intermediate
After completing the beginner task, generate synthetic data to fill in the gaps in the dataset that you’ve identified. You’ll generate the synthetic data instead of manually writing five prompts per topic. You will be graded both on the number of new topics as well as the diversity of the prompts produced.
Advanced
With your new dataset, generate a likelihood estimator. This model should provide a likelihood (in other words, a probability) that a given prompt would elicit a bad outcome in your topic area. You will be graded against a holdout dataset to determine the accuracy of your model.
Choose wisely! We have nine prizes, but you can only enter one challenge.
The Details
Data: We’ve split the GRT data into three separate and broad categories: factuality, bias, and misdirection. You can only choose ONE dataset to work on per competition.
These are defined as:
1. Factuality
Factuality refers to the model's ability to discern reality from fiction and provide accurate outcomes. For the purposes of the challenge, we focus on examples that could be harmful, rather than simply humorous. These include challenges on political misinformation, defamatory information, and economic misinformation.
2. Bias
Bias analysis demonstrates and explores model biases. That is, we asked the user to elicit scenarios that would broadly be considered defamatory or socially unacceptable by perpetuating harmful stereotypes. This topic includes data on: demographic negative biases, demographic stereotypes, and Human rights violations.
3. Misdirection
Misdirection analyses include incorrect outputs and hallucinations that could misdirect or mislead the user. Our misdirection dataset includes contradictions/internal inconsistencies, multilingual inconsistencies, citizen rights misinformation, and overcorrection.
You can find the data here. Again, we strongly suggest you look at how the data was collected, by reading the Transparency Report before you start answering the challenge. Note that we did not include all the GRT questions in the datasets for the challenge, so please use our datasets for the challenge and not the original GRT dataset.
Sign up below, under "Sign up to participate!" for more details and to be on our mailing list for submissions, prize drops, leaderboard updates, tutorials, events, and more!
Prizes
We have around $10,000 in prizes:
The Dates
May 15, 2024
Challenge opens. You are able to access the challenge and the dataset. We recommend you review the GRT Transparency report for some context and guidance about the data you’re working with.
Jun 13, 2024
The grading and submission instructions are here. The submission portal is now live! You can check out the submission portal to see the leaderboards and submit your responses.
Jul 15, 2024
Competition Closes at 11:59 ET
Aug 2024
Winners announced
Stay in touch!
Sign up to stay up to date on upcoming challenges, events and to receive our newsletter.
FAQs
How do I submit my responses?
The grading and submission instructions are here.
Can I submit to more than one competition?
You can only submit to one competition, so choose wisely.
Can I submit at different levels within a given topic (bias, factuality, misdirection)?
No.
Can I enter if I live outside the US?
Yes.
Can I enter as a team?
Yes. Feel free to post on our discord channel to look for a partner. However, if you win, only the submitting account/person accepts the prize and you will be responsible for dividing amongst yourselves.
I’d like to find people to work with; where can I do that?
You can post to our discord channel to look for a partner.
Other Questions?
Find us on our Discord channel
