Participatory Moral AI Is Not Neutral: The Invisible Hand of Developers

Overview§

This paper challenges the assumption that voting-based moral AI alignment is neutral or objective. The authors dissect the moral preference elicitation pipeline into three stages where developers exert hidden normative influence: (1) feature scoping (which aspects of a dilemma are put to a vote), (2) voter sampling (which participants are polled), and (3) question framing (how the dilemma is worded). Through two empirical phases with N=809 participants across three deployment contexts—AI kidney allocation, AI agents simulating absent workers, and generative AI depicting the deceased—they show that each stage can materially alter the resulting preference profile.

Methodology§

The study follows a common moral AI elicitation design: participants are presented with hypothetical dilemmas and asked to judge the acceptability of AI decisions that trade off different moral features. The authors systematically manipulated each pipeline stage.

Phase 1: Feature Scoping§

Participants rated the moral relevance of features (e.g., patient age, worker autonomy, authenticity of a deceased person's likeness) across the three contexts. They found that the same feature can be morally relevant in one context but irrelevant or even negatively weighted in another. For example, efficiency might be a dominant feature in kidney allocation but secondary in simulating an absent worker. This indicates that feature schemas are context-dependent and cannot be naively transferred.

Phase 2: Voter Sampling and Political Ideology§

Participants were classified by political ideology (liberal vs. conservative). For roughly one-third of features, ideological groups expressed significantly different preferences, and some differences reversed direction across contexts. The ideological composition of the voter pool therefore directly alters the aggregated preference profile. Let $p_{i,f}$ be participant $i$'s preference rating for feature $f$. The aggregated score is:

$$ \bar{p}_f = \frac{1}{N} \sum_{i=1}^N p_{i,f} $$

If the sample is skewed, $\bar{p}_f$ shifts. The paper demonstrates that this is not just noise—it is systematic and predictable from ideology.

Phase 3: Question Framing§

The same dilemma was worded in different ways (e.g., using emotional vs. neutral language, or emphasizing recipient vs. donor outcomes). Framing changed aggregate judgments by up to a full scale point (e.g., on a 5-point Likert scale) and widened or narrowed ideological gaps. For a pair of framing conditions $A$ and $B$, the framing effect on preference for feature $f$ is:

$$ \Delta_f = \bar{p}_f^{(A)} - \bar{p}_f^{(B)} $$

The authors also show that framing modulates how moral foundations (e.g., care, fairness, loyalty) correlate with judgments, altering the apparent psychological basis of decisions.

Implications§

Together, these results demonstrate that the moral AI elicitation pipeline is not a neutral aggregation mechanism. The developer choices embedded in feature scoping, voter sampling, and question framing function as an "invisible hand" that pre-determines outcomes. Aggregation alone cannot yield fair or transparent AI; each pipeline stage must be explicitly audited and disclosed.

Implementation Sketch§

The following Python snippet illustrates how one might compute feature-level preference aggregations and compare framing conditions, similar to the paper's analysis pipeline.

import pandas as pd
import torch

# Simulated data: rows = participants, cols = features, values = Likert ratings
# Condition column indicates framing condition

df = pd.DataFrame({
    'participant': range(100),
    'feature_efficiency': torch.randint(1, 6, (100,)).numpy(),
    'feature_fairness': torch.randint(1, 6, (100,)).numpy(),
    'feature_authenticity': torch.randint(1, 6, (100,)).numpy(),
    'ideology': np.random.choice(['liberal', 'conservative'], 100),
    'framing': np.random.choice(['neutral', 'emotional'], 100)
})

# Aggregate by framing condition
agg = df.groupby('framing')[['feature_efficiency', 'feature_fairness', 'feature_authenticity']].mean()
print(agg)

# Compute ideological gap per feature within a fixed framing
for feat in ['feature_efficiency', 'feature_fairness', 'feature_authenticity']:
    liberal_mean = df.loc[(df['ideology'] == 'liberal') & (df['framing'] == 'neutral'), feat].mean()
    conservative_mean = df.loc[(df['ideology'] == 'conservative') & (df['framing'] == 'neutral'), feat].mean()
    print(f'{feat}: liberal-conservative gap = {liberal_mean - conservative_mean:.2f}')

# In practice, one would bootstrap or use regression to estimate uncertainty
# and test for significant framing x ideology interactions.

In a full implementation, one would use multi-level regression models to estimate the variance attributable to participant demographics, context, and framing, rather than simple group means. The key takeaway is that the aggregation function itself is not the problem—the upstream choices are.

Conclusion§

The paper provides strong empirical evidence that moral AI elicitation is value-laden at every step. It calls for a shift from "aggregate everything" to "audit everything": each decision about features, voters, and wording should be documented and made transparent to stakeholders. This is critical for building AI systems that are genuinely accountable in morally loaded domains.