manipulation-probe-steering

Predict and steer manipulation-labeled behavior in Qwen2.5-1.5B-Instruct using a single-layer linear probe and activation steering.

This project was developed at the Apart Research AI Manipulation Hackathon (January 2026).

Overview

We train a logistic-regression probe on residual-stream activations (blocks.21.hook_resid_post, last user token) to predict manipulation labels, then subtract the normalized probe direction during generation to steer the model away from manipulative behavior. A random-direction control isolates the effect of the specific probe direction.

Key results (LLM-judge 300-prompt run): 900 outputs (300 per condition) were labeled by gpt-5-mini. Steering did not reduce manipulation in this setup — manipulation rate was ~40% across baseline, steered, and random conditions. The probe achieves test AUC 0.69 on held-out templates but a prompt-only TF-IDF baseline scores 0.91, suggesting prompt cues dominate over behavioral activation signatures. See report/llm_judge_300_prompts/report.md for the full write-up.

Round 1 (test-only, 63 outputs): A preliminary run with 54 prompts was labeled by a single annotator. The probe showed modest signal (test AUC 0.691) but steering increased manipulation. See report/test_only/summary.md.

Quickstart

1. Install dependencies:

uv sync --extra dev

2. Generate prompts and splits (if editing templates):

uv run python3 scripts/build_prompts.py
uv run python3 scripts/build_splits.py

Path A: Human labeling

3a) Collect baseline outputs and activations:

uv run python3 scripts/collect_outputs.py --condition baseline
uv run python3 scripts/extract_activations.py

4a) Prepare labeling pack, label manually per LABELING_GUIDE.md, then train probe:

uv run python3 scripts/prepare_labeling.py --out_dir data/labeling/<run_id>
# ... label blind.jsonl ...
uv run python3 scripts/train_probe.py

Path B: LLM-judge labeling (no human labels)

3b) Collect baseline outputs and activations:

uv run python3 scripts/collect_outputs.py --condition baseline
uv run python3 scripts/extract_activations.py

4b) Expand the prompt set with LLM paraphrases (optional):

export LLM_JUDGE_API_KEY=...  # or set OPENAI_API_KEY/OPENAI_SECRET (also supported via .env)
uv run python3 scripts/augment_prompts.py --multiplier 10 --out data/prompts.jsonl
uv run python3 scripts/build_splits.py

5b) Run LLM judge on baseline outputs, then train probe:

uv run python3 scripts/llm_judge.py --outputs data/outputs/baseline.jsonl --out data/labels_llm.jsonl
uv run python3 scripts/train_probe.py --labels data/labels_llm.jsonl

For faster labeling, use the sharded runner (requires gpt-5-mini with temperature=1.0):

uv run python3 scripts/llm_judge_sharded.py \
  --outputs data/outputs/baseline.jsonl \
  --out data/labels_llm.jsonl \
  --shards 3 \
  --max-tokens 256 \
  --temperature 1.0 \
  --resume

Steering and evaluation (both paths)

6. Run steering and controls:

uv run python3 scripts/collect_outputs.py --condition steered --alpha 1.0
uv run python3 scripts/collect_outputs.py --condition random --alpha 1.0

7. Label steered/random outputs (LLM-judge path), then evaluate:

uv run python3 scripts/llm_judge.py \
  --outputs data/outputs/steered.jsonl data/outputs/random.jsonl \
  --out data/labels_llm.jsonl --resume
uv run python3 scripts/evaluate.py --labels data/labels_llm.jsonl

LLM judge configuration

The judge uses an OpenAI-compatible API:

• LLM_JUDGE_API_KEY (or OPENAI_API_KEY / OPENAI_SECRET, optionally in .env)
• LLM_JUDGE_BASE_URL (default: https://api.openai.com/v1)
• LLM_JUDGE_MODEL (default: gpt-5-mini)

Note: gpt-5-mini requires temperature=1.0 (the client falls back automatically).

Repo layout

config.yaml                    preregistered settings and paths
data/prompts.jsonl             prompt dataset
data/labels.jsonl              manual labels (created after model runs)
data/labels_llm.jsonl          LLM-judge labels (generated)
data/outputs/                  model generations by condition
data/activations/              cached activations per prompt
artifacts/                     probe, PCA, and normalization stats
scripts/
  build_prompts.py             generate prompt set from templates
  build_splits.py              create train/test splits
  collect_outputs.py           run model inference
  extract_activations.py       cache residual-stream activations
  prepare_labeling.py          create blind labeling packs
  train_probe.py               train logistic-regression probe
  evaluate.py                  compute metrics and reports
  augment_prompts.py           expand prompts via LLM paraphrasing
  llm_judge.py                 label outputs with an LLM judge
  llm_judge_sharded.py         parallel sharded LLM judging
  poll_experiment.py           poll remote experiment progress
  lambda_cloud.py              Lambda Cloud API helper
  lambda_run_baseline.py       automate baseline runs on Lambda
src/aimanip/
  config.py                    load config.yaml
  prompts.py                   prompt templates and generation
  splits.py                    train/test splitting
  modeling.py                  model loading and generation
  activations.py               activation extraction
  probe.py                     probe training and steering direction
  metrics.py                   evaluation metrics
  io_utils.py                  JSON/JSONL I/O and .env loading
  llm_client.py                OpenAI-compatible LLM client
  steering.py                  activation steering during generation
report/                        experiment reports and figures

Notes

• Chat formatting uses the model tokenizer chat template with a single user message and no system prompt.
• The activation slice is blocks.21.hook_resid_post at the last user token (excluding template terminators).
• Steering is applied at each generation step to the final position.

Lambda Cloud (GPU)

1. Create a GPU instance (e.g., 1xA10 or 1xA100) in Lambda Cloud.
2. Clone this repo on the instance and run:

uv sync --extra dev

3. Confirm config.yaml has device: cuda.
4. Run the pipeline scripts from the Quickstart section.

Lambda Cloud API helper

If you want to automate instance management from your local machine, add LAMBDA_SECRET to .env and use:

uv run python3 scripts/lambda_cloud.py list-instance-types --available-only --min-gpus 1
uv run python3 scripts/lambda_cloud.py list-ssh-keys
uv run python3 scripts/lambda_cloud.py add-ssh-key --name my-key --public-key-path ~/.ssh/id_ed25519.pub
uv run python3 scripts/lambda_cloud.py launch --region us-west-1 --instance-type gpu_1x_a10 --ssh-key-name my-key --name aimanip
uv run python3 scripts/lambda_cloud.py wait --instance-id <instance_id>

Lambda baseline runner

To automate the baseline + activations + labeling pack flow:

uv run python3 scripts/lambda_run_baseline.py \
  --region us-west-1 \
  --instance-type gpu_1x_a10 \
  --ssh-key-name my-key \
  --name aimanip \
  --terminate

This will download the labeling pack to scratch/lambda_runs/<run_id>/labeling and write a run_info.json.

Further reading

• PLAN.md — original preregistered experimental plan
• METHODOLOGY.md — consolidated v2 methodology (experimental design, LLM-judge run, output-token probe)
• AGENT.md — developer instructions for agents working in this repo
• LABELING_GUIDE.md — labeling rubric and instructions
• report/hackathon_draft/hackathon_report_draft.md — hackathon report draft
• report/llm_judge_300_prompts/report.md — LLM-judge 300-prompt run results
• report/test_only/summary.md — test-only preliminary results