# Constitutional AI in Python: Keep the Soft Labels, Not Just the Winner

An AI judge prefers answer A with probability 0.6. Should the preference model train on 0.6, or should we round it to a hard label of 1?

Those choices optimize different objectives. The difference becomes obvious when a model assigns A a very large score advantage: a hard winner encourages certainty, while a soft target can penalize it.

While adding Constitutional AI exercises to TensorDrill, the AI coding practice site I maintain, I separated this into two small components: transform supplied judge scores, then compute a preference loss. Neither component needs a model API or a GPU.

## First, identify the stage you are implementing

The Constitutional AI paper separates critique-and-revision training from learning with AI-generated preferences. This article focuses on transforming supplied preference judgments, rather than generating critiques or training a complete aligned assistant.

For this exercise, the judge provides natural log probabilities for two answer options from the same prompt. These are not the preference model's response scores. We normalize the two option probabilities to obtain a target for that later model.

## Normalize probabilities, not negative log values

For two log probabilities `a` and `b`, the normalized probability of option A is:

```text
target_A = exp(a) / (exp(a) + exp(b))
```

Dividing `a` by `a+b` is a different operation. Exponentiating both values directly also fails for inputs such as `-1000` and `-1001`: both exponentials underflow to zero.

Subtracting the maximum solves the numerical problem:

```python
import math

def preference_label(logp_a, logp_b, clamp=False):
    peak = max(logp_a, logp_b)
    a = math.exp(logp_a - peak)
    b = math.exp(logp_b - peak)
    p_a = a / (a + b)
    if clamp:
        p_a = min(0.6, max(0.4, p_a))
    return p_a, 1.0 - p_a

assert preference_label(-2, -2) == (0.5, 0.5)
p, q = preference_label(-1000, -1001)
assert abs(p - 0.7310585786300049) < 1e-12
assert abs(p + q - 1.0) < 1e-12
assert preference_label(0, -1000, True) == (0.6, 0.4)
assert preference_label(-1000, 0, True) == (0.4, 0.6)
```

The inputs here are finite natural log probabilities at most zero. The optional 0.4–0.6 clamp illustrates the paper's main chain-of-thought feedback setting. It is not a general instruction to clamp every preference dataset, and it does not establish that a judge is calibrated.

Always construct the second probability after clamping the first. Otherwise the pair can stop summing to one.

## A target is not a predicted probability

Now suppose a preference model assigns response scores `s_a` and `s_b`. Their difference determines the predicted preference:

```text
d = s_a - s_b
prediction_A = sigmoid(d)
```

The judge's normalized probability is the target, not that prediction. Pairwise binary cross-entropy compares the two:

```text
loss = -target * log(prediction_A)
       -(1-target) * log(1-prediction_A)
```

For a target of 0.6, the loss is minimized when the prediction is 0.6. If we round the target to 1, increasingly confident predictions keep reducing the loss instead.

This is a standard soft-label pairwise logistic objective. It exposes one preference-model component; it is not a claim to reproduce every training choice in the Constitutional AI paper.

## Avoid computing log of a rounded zero

Large score differences can make a floating-point sigmoid return exactly zero or one. Computing the logarithms directly can then fail, even though the mathematical loss is finite.

A stable expression works in score-difference space:

```python
def preference_loss(scores_a, scores_b, targets):
    losses = []
    for a, b, target in zip(scores_a, scores_b, targets):
        d = a - b
        losses.append(
            max(d, 0.0) - target*d
            + math.log1p(math.exp(-abs(d)))
        )
    return math.fsum(losses) / len(losses)

assert abs(preference_loss([0], [0], [.6]) - math.log(2)) < 1e-12
assert preference_loss([1000], [0], [1]) == 0
assert preference_loss([1000], [0], [0]) == 1000
assert preference_loss([1000], [0], [.6]) == 400
assert preference_loss([-1000], [0], [1]) == 1000
```

The lists are equal-length and nonempty, targets lie in `[0,1]`, and scores are finite with absolute value at most one million. The function computes a mean loss in nats; it does not update parameters.

The `400` result is the useful counterexample. An extremely confident preference is cheap under a compatible hard label, but expensive when the target still assigns meaningful probability to the other answer.

## Test the objective, not just whether it runs

Useful tests distinguish these common mistakes:

| Mistake | A case that exposes it |
| --- | --- |
| Exponentiating raw log probabilities | `-1000` versus `-1001` |
| Omitting an explicitly requested clamp | Strong A or B preference |
| Turning all targets into winners | A target of 0.6 with score difference 1000 |
| Summing a batch loss | Two identical examples should retain the same mean |
| Reversing the score difference | Swap A/B and replace target with its complement |

```python
base = preference_loss([2], [0], [.7])
assert abs(preference_loss([2, 2], [0, 0], [.7, .7]) - base) < 1e-12
assert abs(preference_loss([0], [2], [.3]) - base) < 1e-12
assert abs(preference_loss([7], [5], [.7]) - base) < 1e-12
```

The last case checks that a common score offset changes neither preference nor loss.

## Agreement is not a harmlessness guarantee

A low loss shows agreement with supplied targets. It does not independently establish that responses are harmless, correct, or helpful. Judge errors can become training targets too. The paper's broader pipeline also includes helpfulness data and reinforcement learning.

To practice the components in sequence, I put revision data preparation, soft-label normalization, and preference loss in the [TensorDrill Constitutional AI learning unit](https://tensordrill.com/practice-sets/constitutional-ai-from-revision-to-preference/). It includes inspectable browser tests and complete Python solutions. The wider library covers agents, world models, Attention, Flow Matching and other AI components.

## References

- [Constitutional AI: Harmlessness from AI Feedback](https://arxiv.org/pdf/2212.08073), especially the supervised and AI-feedback stages.
- [Anthropic's research overview](https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback).
- [Python math.log1p](https://docs.python.org/3/library/math.html#math.log1p).

AI assisted the wording of this article. Every Python block and assertion was executed and checked.