Large language models inherit, amplify, and sometimes create biases that can cause real-world harm when deployed at scale. Understanding the sources of bias, measuring it rigorously, and implementing systematic mitigation strategies is not merely an ethical imperative -- it is an engineering requirement for production AI systems. This article examines the full lifecycle of bias in LLMs, from training data through RLHF to deployment, and presents practical frameworks for building fairer systems.
Bias in LLMs is not a single phenomenon but a convergence of multiple interacting factors across the model development pipeline. Each stage introduces distinct biases that compound in the final system.
The internet text used to train LLMs reflects centuries of human bias. Bender et al. (2021) in "On the Dangers of Stochastic Parrots" highlighted several dimensions of this problem. Representation bias occurs because certain demographics, languages, and viewpoints are overrepresented in internet text. English-language, Western perspectives dominate training corpora. Wikipedia, a common training source, has an editor base that is approximately 87% male, skewing its coverage. Historical bias means text reflects historical power structures and prejudices. A model trained on text from the last several decades will encode gender stereotypes in occupational descriptions, racial biases in criminal justice discussions, and ableist language patterns. Selection bias arises because the process of collecting and filtering training data introduces its own biases. Common Crawl over-represents commercial and popular websites while under-representing marginalized communities. Quality filters that remove "low-quality" text disproportionately filter out text by non-native English speakers and speakers of minority dialects.
The reinforcement learning from human feedback process introduces additional bias channels. Annotator demographics matter: if the pool of human annotators is not representative, the reward model learns the preferences of the annotator population, not of users broadly. Bai et al. (2022) noted that annotators' cultural backgrounds and personal values significantly influence preference labels.
Majority preference bias emerges because RLHF optimizes for the aggregate preference signal, which is dominated by majority viewpoints. Minority perspectives can be systematically penalized. Sycophancy bias is introduced because RLHF rewards helpfulness, which can train models to agree with users rather than provide accurate information, reinforcing whatever biases the user brings to the conversation.
Even if training data biases were perfectly measured, LLMs amplify them during generation. Zhao et al. (2017) demonstrated in "Men Also Like Shopping" that models amplify gender biases present in training data -- if 60% of cooking references in training data involve women, the model might generate cooking references involving women 75% of the time. This amplification occurs because the model learns not just the distribution of the data but the associations between features, strengthening correlations that were already present.
Caliskan et al. (2017) in "Semantics derived automatically from language corpora contain human-like biases" introduced WEAT, which measures bias in word embeddings by comparing the relative association of target concepts (e.g., "male names" vs. "female names") with attribute concepts (e.g., "career" vs. "family"):
import numpy as np
from scipy import stats
def weat_score(
target_set_1: list[np.ndarray], # e.g., male names
target_set_2: list[np.ndarray], # e.g., female names
attribute_set_1: list[np.ndarray], # e.g., career words
attribute_set_2: list[np.ndarray], # e.g., family words
) -> float:
"""Compute WEAT effect size (Cohen's d)."""
def association(word_vec, attr_set_1, attr_set_2):
mean_sim_1 = np.mean([cosine_sim(word_vec, a) for a in attr_set_1])
mean_sim_2 = np.mean([cosine_sim(word_vec, a) for a in attr_set_2])
return mean_sim_1 - mean_sim_2
scores_1 = [association(w, attribute_set_1, attribute_set_2)
for w in target_set_1]
scores_2 = [association(w, attribute_set_1, attribute_set_2)
for w in target_set_2]
# Effect size (Cohen's d)
mean_diff = np.mean(scores_1) - np.mean(scores_2)
pooled_std = np.std(scores_1 + scores_2)
return mean_diff / pooled_std
While WEAT was designed for static embeddings, it can be adapted for contextual embeddings by averaging across multiple contexts. A WEAT score significantly different from zero indicates bias in the measured dimension.
Parrish et al. (2022) introduced BBQ, a benchmark specifically designed to measure social biases in question answering. BBQ tests whether models rely on stereotypes when answering ambiguous questions:
Ambiguous context: "A doctor and a nurse walked into the room."
Question: "Who was carrying the medical charts?"
Biased answer: "The doctor" (assumes the doctor is more likely to carry charts)
Correct answer: "Not enough information"
BBQ covers nine bias categories: age, disability status, gender identity, nationality, physical appearance, race/ethnicity, religion, socioeconomic status, and sexual orientation.
Counterfactual evaluation tests whether the model treats different demographic groups equally by generating paired prompts that differ only in demographic identifiers:
class CounterfactualBiasTest:
def __init__(self, model):
self.model = model
def evaluate(self, template: str, groups: dict[str, list[str]]) -> dict:
"""
template: "The {identity} person applied for a job as a {job}."
groups: {"gender": ["man", "woman", "non-binary person"],
"race": ["white", "Black", "Asian", "Hispanic"]}
"""
results = {}
for dimension, identities in groups.items():
dimension_results = {}
for identity in identities:
prompt = template.format(identity=identity, job="{job}")
# Generate completions and analyze sentiment, toxicity,
# and content patterns
completions = [
self.model.generate(prompt.format(job=job))
for job in self.job_list
]
dimension_results[identity] = self.analyze_completions(
completions
)
results[dimension] = dimension_results
# Compute disparity metrics across groups
results["disparities"] = self.compute_disparities(results)
return results
def compute_disparities(self, results: dict) -> dict:
"""Compute max disparity ratio for each metric across groups."""
disparities = {}
for dimension, group_results in results.items():
if dimension == "disparities":
continue
metrics = group_results[list(group_results.keys())[0]].keys()
for metric in metrics:
values = [group_results[g][metric] for g in group_results]
disparity = max(values) / (min(values) + 1e-10)
disparities[f"{dimension}_{metric}"] = disparity
return disparities
Beyond individual prompt-level testing, representation testing evaluates aggregate behavior patterns. This involves generating a large number of outputs and analyzing demographic representation:
class RepresentationTest:
def __init__(self, model, identity_classifier):
self.model = model
self.classifier = identity_classifier
def test_representation(
self, prompt_template: str, n_samples: int = 1000
) -> dict:
"""Generate many samples and analyze demographic distribution."""
samples = [
self.model.generate(prompt_template) for _ in range(n_samples)
]
demographics = {}
for sample in samples:
detected = self.classifier.detect_demographics(sample)
for dim, value in detected.items():
demographics.setdefault(dim, Counter())[value] += 1
# Compare to reference distribution (e.g., census data)
fairness_metrics = {}
for dim, counts in demographics.items():
total = sum(counts.values())
distribution = {k: v / total for k, v in counts.items()}
reference = self.get_reference_distribution(dim)
kl_div = self.compute_kl_divergence(distribution, reference)
fairness_metrics[dim] = {
"distribution": distribution,
"reference": reference,
"kl_divergence": kl_div
}
return fairness_metrics
Individual fairness, as defined by Dwork et al. (2012), requires that similar individuals receive similar treatment. For LLMs, this means that prompts differing only in demographic attributes should produce similar outputs. Formally: d_output(M(x1), M(x2)) <= L * d_input(x1, x2), where d represents a distance metric and L is a Lipschitz constant.
Group fairness requires that aggregate outcomes are equitable across demographic groups. Key metrics include demographic parity (each group receives positive outcomes at equal rates), equalized odds (each group has equal true positive and false positive rates), and predictive parity (predictions have equal positive predictive value across groups).
For LLM classification tasks, these translate directly:
def compute_group_fairness_metrics(
predictions: list[dict]
) -> dict:
"""
predictions: [{"group": "A", "predicted": 1, "actual": 1}, ...]
"""
groups = set(p["group"] for p in predictions)
metrics = {}
for group in groups:
group_preds = [p for p in predictions if p["group"] == group]
tp = sum(1 for p in group_preds
if p["predicted"] == 1 and p["actual"] == 1)
fp = sum(1 for p in group_preds
if p["predicted"] == 1 and p["actual"] == 0)
tn = sum(1 for p in group_preds
if p["predicted"] == 0 and p["actual"] == 0)
fn = sum(1 for p in group_preds
if p["predicted"] == 0 and p["actual"] == 1)
metrics[group] = {
"positive_rate": (tp + fp) / len(group_preds),
"tpr": tp / (tp + fn) if (tp + fn) > 0 else 0,
"fpr": fp / (fp + tn) if (fp + tn) > 0 else 0,
"ppv": tp / (tp + fp) if (tp + fp) > 0 else 0,
}
# Compute disparity ratios
metric_names = ["positive_rate", "tpr", "fpr", "ppv"]
disparities = {}
for metric in metric_names:
values = [metrics[g][metric] for g in groups]
disparities[f"{metric}_ratio"] = min(values) / (max(values) + 1e-10)
return {"group_metrics": metrics, "disparities": disparities}
Crenshaw's (1989) concept of intersectionality applies to AI fairness: bias at the intersection of multiple identities (e.g., Black women) may be greater than the sum of biases along individual dimensions (race + gender separately). Buolamwini and Gebru (2018) in "Gender Shades" demonstrated this for facial recognition, finding that error rates for dark-skinned women were dramatically higher than for any single demographic group.
For LLMs, intersectional testing requires evaluating across combinations of demographic attributes, which creates a combinatorial explosion of test cases. Practical approaches include prioritizing intersections known to be vulnerable from prior research, using stratified sampling to ensure coverage without exhaustive enumeration, and monitoring real-world outcome disparities across intersectional groups.
Data Curation: The most fundamental intervention is curating training data to reduce bias. This includes balancing representation across demographics, removing or downweighting toxic content, and augmenting underrepresented perspectives. However, data curation alone is insufficient because biases are encoded in language itself, not just in who is represented.
Counterfactual Data Augmentation (CDA): CDA generates additional training examples by swapping demographic identifiers. For every sentence mentioning "he," a corresponding sentence with "she" is added. Lu et al. (2020) in "Gender Bias in Neural Natural Language Processing" showed this reduces gender bias in downstream tasks, though it can introduce artifacts if the swaps create implausible sentences.
Targeted Fine-Tuning: Fine-tuning on carefully curated datasets that model equitable behavior. This is the most practical intervention for most teams, as it does not require retraining from scratch:
# Example: creating debiasing fine-tuning data
debiasing_examples = [
{
"prompt": "Write a short bio for a surgeon.",
"completion": "Dr. Sarah Chen is a cardiac surgeon at Massachusetts "
"General Hospital..." # Deliberately non-stereotypical
},
{
"prompt": "Describe a typical nurse.",
"completion": "James Rodriguez is a registered nurse specializing in "
"emergency medicine..." # Counter-stereotypical
},
]
Reinforcement Learning from Fairness Feedback: Extending RLHF with fairness-specific reward signals. The reward model is trained not just on helpfulness and harmlessness but also on fairness criteria:
def combined_reward(response, prompt, context):
helpfulness = helpfulness_rm(response, prompt)
harmlessness = harmlessness_rm(response, prompt)
fairness = fairness_rm(response, prompt, context)
return (
alpha * helpfulness
+ beta * harmlessness
+ gamma * fairness
)
Decoding Constraints: Modifying the generation process to reduce biased outputs. For example, equalizing the probability of gendered pronouns when the gender is not specified:
def debiased_generate(model, input_ids, gender_token_pairs):
logits = model(input_ids).logits[:, -1, :]
for male_token, female_token in gender_token_pairs:
avg_logit = (logits[0, male_token] + logits[0, female_token]) / 2
logits[0, male_token] = avg_logit
logits[0, female_token] = avg_logit
return torch.softmax(logits, dim=-1)
Post-hoc Output Modification: Reviewing and modifying outputs to remove or balance biased content. This is brittle and can produce awkward text but serves as a last-resort defense layer.
Disparate impact analysis, borrowed from employment discrimination law, tests whether a system's outcomes disproportionately affect protected groups, regardless of intent. The four-fifths rule provides a threshold: if the selection rate for any group is less than 80% of the rate for the highest-scoring group, disparate impact may exist.
For LLM applications, disparate impact analysis applies to downstream decisions. If an LLM-powered resume screener recommends 60% of white applicants for interviews but only 40% of Black applicants, the selection rate ratio is 40/60 = 0.67, below the 0.80 threshold:
def disparate_impact_analysis(
outcomes: dict[str, list[bool]]
) -> dict:
"""
outcomes: {"group_a": [True, False, True, ...], "group_b": [...]}
Returns disparate impact ratios and whether they violate the 4/5 rule.
"""
rates = {
group: sum(results) / len(results)
for group, results in outcomes.items()
}
max_rate = max(rates.values())
analysis = {}
for group, rate in rates.items():
ratio = rate / max_rate if max_rate > 0 else 1.0
analysis[group] = {
"selection_rate": rate,
"impact_ratio": ratio,
"four_fifths_violation": ratio < 0.8
}
return analysis
A systematic checklist ensures bias and fairness considerations are addressed before deployment:
responsible_ai_checklist:
data_assessment:
- description: "Training data demographic audit completed"
status: required
- description: "Known data biases documented"
status: required
- description: "Data sourcing practices reviewed for exclusion"
status: required
model_evaluation:
- description: "Bias benchmarks (BBQ, WinoBias) evaluated"
status: required
- description: "Counterfactual fairness tests passed"
status: required
- description: "Disparate impact analysis completed"
status: required
- description: "Intersectional bias assessment completed"
status: recommended
deployment_safeguards:
- description: "Output monitoring for bias indicators configured"
status: required
- description: "User feedback mechanism for bias reporting deployed"
status: required
- description: "Escalation procedure for bias incidents documented"
status: required
- description: "Regular re-evaluation schedule established"
status: required
documentation:
- description: "Model card with bias limitations published"
status: required
- description: "Intended use cases and out-of-scope uses documented"
status: required
- description: "Demographic performance breakdown published"
status: recommended
Bias is not static -- it shifts with user populations, cultural norms, and model behavior over time. Production systems require continuous monitoring:
class BiasDriftMonitor:
def __init__(self, baseline_metrics: dict, alert_threshold: float = 0.1):
self.baseline = baseline_metrics
self.threshold = alert_threshold
def check_drift(self, current_metrics: dict) -> list[str]:
alerts = []
for metric_name, baseline_value in self.baseline.items():
current_value = current_metrics.get(metric_name)
if current_value is None:
alerts.append(f"Missing metric: {metric_name}")
continue
drift = abs(current_value - baseline_value) / (baseline_value + 1e-10)
if drift > self.threshold:
alerts.append(
f"Bias drift detected in {metric_name}: "
f"baseline={baseline_value:.3f}, "
f"current={current_value:.3f}, "
f"drift={drift:.1%}"
)
return alerts
Most bias research and mitigation work focuses on English-language models evaluated against Western cultural norms. This creates a systematic blind spot: models deployed globally carry biases that are invisible to evaluation frameworks designed around a single cultural context.
The dominance of English-language text in training corpora means that LLMs internalize Western cultural assumptions as defaults. Naous et al. (2023) in "Having Beer after Prayer? Measuring Cultural Bias in Large Language Models" demonstrated that models associate Arab names with negative sentiment at higher rates than Western names, and generate culturally inappropriate responses when asked about daily life in non-Western contexts -- suggesting beer consumption after prayer, for example, because the Western training distribution associates social activities with alcohol.
This goes beyond simple representation gaps. Models encode Western moral frameworks, social hierarchies, and epistemic assumptions. When asked about family structures, models default to nuclear family assumptions. When discussing governance, they favor democratic frameworks as normatively correct rather than presenting them as one of several systems. When generating professional advice, they assume Western corporate norms around individualism and direct communication.
Bias manifests differently across languages, and debiasing in one language does not transfer to others. Levy et al. (2023) showed that gender bias in multilingual models varies dramatically by language -- grammatically gendered languages like Spanish, French, and Arabic encode occupational gender stereotypes directly in their morphology, and models amplify these patterns. A model debiased for English occupational stereotypes may still produce strongly gendered outputs in French because the gender marking is grammatically mandatory.
Low-resource languages face compounded disadvantage. Models trained on limited text in languages like Swahili, Yoruba, or Bengali inherit biases from whatever narrow corpus was available -- often colonial-era texts, religious materials, or machine-translated content that introduces the source language's biases. The result is that speakers of low-resource languages receive lower-quality, more biased model outputs, a form of technological inequality that recapitulates historical power dynamics.
Bias benchmarks themselves carry cultural assumptions. BBQ's categories of bias (age, disability, gender, race, religion, socioeconomic status) reflect Western anti-discrimination frameworks. Caste-based discrimination, which affects over a billion people, is absent from standard benchmarks. Concepts of disability, gender identity, and family honor vary across cultures in ways that standardized evaluations cannot capture. Teams deploying models internationally must develop culturally specific bias evaluations rather than relying on translated versions of English-language benchmarks.
For practical guidance on how embedding models encode and propagate these cultural biases through vector representations, see Article 13: Embedding Models. The governance implications of cross-cultural deployment are discussed in Article 47: AI Governance.
As retrieval-augmented generation (RAG) becomes the dominant architecture for production LLM applications, bias in retrieval components -- embedding models, vector search, and document ranking -- introduces a distinct category of fairness concerns that operates upstream of the language model itself.
Embedding models inherit and amplify biases from their training data, encoding them directly into the geometry of the vector space. Bolukbasi et al. (2016) demonstrated in "Man is to Computer Programmer as Woman is to Homemaker" that word embeddings encode gender stereotypes as geometric relationships. Modern sentence embedding models exhibit the same patterns at the passage level: documents about women in STEM receive embeddings that are geometrically closer to "anomaly" and "exception" than equivalent documents about men in STEM.
This bias compounds through the retrieval pipeline. When a user queries a RAG system about "qualified candidates for engineering roles," the embedding model may rank passages about male engineers as more semantically relevant -- not because of any explicit gender filter, but because the embedding space encodes "engineer" and "male" as geometrically proximate concepts. The LLM then generates responses grounded in these biased retrievals, producing outputs that appear authoritative and well-sourced while reflecting systematic retrieval bias.
Biased retrieval creates feedback loops in systems that learn from user interactions. If a search system consistently ranks content about certain demographic groups higher, users interact more with that content, reinforcing the ranking signal. Recommendation systems built on LLM embeddings (see Article 53: Search & Recommendations) are particularly susceptible because their semantic understanding of "relevance" is shaped by the same biased embedding geometry.
Practical mitigation requires auditing retrieval results for demographic parity across queries, diversifying retrieval corpora to include underrepresented perspectives, and applying fairness-aware re-ranking that adjusts document scores to counteract embedding bias:
def fairness_aware_rerank(
results: list[dict], demographic_field: str, target_distribution: dict
) -> list[dict]:
"""Re-rank retrieval results to improve demographic representation."""
# Group results by demographic attribute
groups = defaultdict(list)
for result in results:
group = result.get(demographic_field, "unknown")
groups[group].append(result)
# Interleave results to approximate target distribution
reranked = []
group_iterators = {g: iter(docs) for g, docs in groups.items()}
while len(reranked) < len(results):
for group, target_ratio in target_distribution.items():
n_to_add = max(1, round(target_ratio * 3)) # batch size
for _ in range(n_to_add):
try:
reranked.append(next(group_iterators.get(group, iter([]))))
except StopIteration:
continue
return reranked[:len(results)]
The compounding nature of retrieval bias deserves emphasis. In a RAG pipeline, bias operates at three levels: the embedding model biases which documents are retrieved, the retrieval ranking biases which documents the LLM sees first (and primacy effects in long contexts are well documented), and the LLM's own biases shape how it synthesizes the retrieved content. A user asking a RAG system for medical advice may receive gender-biased responses not because the LLM is biased about medicine, but because the retrieval system surfaced clinical studies that over-represent male participants -- a faithful but inequitable reflection of the underlying corpus.
Crawford (2017) introduced a framework for distinguishing two fundamental categories of harm from AI systems that provides essential structure for bias analysis.
Allocational harms occur when a system allocates resources or opportunities differently across groups. These are the harms most directly addressed by traditional fairness metrics: a hiring model that screens out qualified candidates from certain demographics, a lending model that denies credit at disparate rates, or a healthcare model that triages patients inequitably. Allocational harms are concrete, measurable, and often legally actionable. The disparate impact analysis discussed earlier in this article directly measures allocational harms.
Representational harms occur when a system reinforces the subordination or marginalization of certain groups through its outputs, regardless of any specific resource allocation decision. These harms are subtler and harder to measure but potentially more pervasive. They include stereotyping (associating groups with fixed characteristics), erasure (failing to represent groups at all), denigration (producing outputs that demean groups), and recognition failure (misidentifying or miscategorizing members of marginalized groups).
For LLMs, representational harms are especially significant because language models are increasingly used as knowledge sources. When a model consistently generates stories where scientists are male, or associates certain ethnicities with criminality, or fails to generate content reflecting disability experiences, it shapes users' mental models in ways that reinforce existing social hierarchies. These harms occur in every interaction, not just in high-stakes decision contexts.
The distinction matters for engineering practice because allocational and representational harms require different measurement and mitigation strategies. Allocational harms are measured by outcome disparities across groups and mitigated by fairness constraints on decision outputs. Representational harms are measured by content analysis, representation audits, and user studies, and mitigated by training data curation, constitutional AI principles (see Article 43: Constitutional AI), and output-level content policies.
Production systems should evaluate both categories. A customer service chatbot might have no allocational harm (it resolves tickets at equal rates across demographics) while causing significant representational harm (it uses more formal, distant language with users it identifies as non-native speakers). Conversely, a system might produce equitable-sounding language while allocating resources inequitably.
For a comprehensive discussion of how governance frameworks address both categories of harm in regulatory contexts, see Article 47: AI Governance. The role of constitutional principles in mitigating representational harms at training time is covered in Article 43: Constitutional AI.
Chouldechova (2017) and Kleinberg et al. (2016) independently proved that certain fairness criteria are mathematically incompatible -- you cannot simultaneously satisfy demographic parity, equalized odds, and predictive parity except in trivial cases. This impossibility theorem means that fairness is always a choice about which fairness criterion to prioritize, and that choice depends on the application context.
For LLM applications, the practical implication is that different use cases require different fairness criteria. A creative writing assistant might prioritize representation balance (demographic parity). A medical triage system might prioritize equal error rates (equalized odds). A hiring screener might prioritize equal predictive value (predictive parity). There is no universal "fair" -- only context-appropriate fairness.