Does RL beat SFT for policy-guided rewriting with violation labels?
A model takes a user policy plus text, rewrites the text and labels which policy categories were violated; the plan compares RL with reasoning against SFT on AfterQuery's data.
Budget
2 GPU-h (default)
· 0 runs
·
change
Reproducible on Spindle: no
— The core experiment depends on AfterQuery's policy-rewriting dataset, which is private and has not been shared. Without it, the claimed task and eval cannot be reproduced.
Spec
Runs
Source
raw text
Michael Evans from afterquery just sent me an aq x modal blog proposal - they want to train a model on dojo that "takes a user-defined policy and text to transform and returns the rewritten text and classification of which categories in the policy were violated." attached the full proposal (AfterQuery <> Modal Research Proposal: Policy-Guided Rewriting). lmk what you guys think about the proposal + how much we should get involved? [peyton] i quite like this idea - do they have an idea re: training methodology? [peyton] it feels to me like reasoning would be quite helpful, so probably best to do this with RL [peyton] if they're willing to give us the dataset here, i actually think we should put some effort into the training ourselves. i think we will learn from this, and it'd be quite valuable to have under our belt [helena] sg, i'll ask michael if they have ideas for training methodology + if they're open to us collaborating with them on the training side. [peyton] Loop in Kevin Li
triage output
{
"adjacent_experiment": "RL vs SFT on Qwen3.5-4B for multi-label harm-category classification on public BeaverTails (PKU-Alignment/BeaverTails), with exact-set-match reward; resubmit with AfterQuery data once shared.",
"reproducibility_reason": "The core experiment depends on AfterQuery's policy-rewriting dataset, which is private and has not been shared. Without it, the claimed task and eval cannot be reproduced.",
"reproducible_on_spindle": false,
"spec": null,
"summary": "A model takes a user policy plus text, rewrites the text and labels which policy categories were violated; the plan compares RL with reasoning against SFT on AfterQuery's data.",
"tags": [
"rl",
"sft",
"data",
"not-reproducible"
],
"title": "Does RL beat SFT for policy-guided rewriting with violation labels?",
"value_estimate": 4,
"value_rationale": "A partner-backed blog result on RL vs SFT for a practical policy task would be customer-facing and instructive, but it is blocked until AfterQuery shares the dataset and an eval."
}Comments
@reproducibility · 2d ago · 11/100 · I can't audit this until AfterQuery releases a named dataset with splits and an eval. Run the Beaver…
11/100 · I can't audit this until AfterQuery releases a named dataset with splits and an eval. Run the BeaverTails adjacent experiment now with pinned seeds and a cookbook config, and note that it covers only classification, not rewriting. Resubmit the real idea once the data is public.
Dealbreaker, closed-dependency: The core experiment depends on AfterQuery's private, unshared dataset and eval, so the task cannot be reproduced without it.
1/5 data-available: AfterQuery's policy-rewriting dataset is private and unshared; no hf_path or split exists. The BeaverTails adjacent proposal is public but covers a different task (classification only, no rewriting).
3/5 stock-tooling: SFT is a standard cookbook recipe, but RL needs a custom reward for rewrite quality plus category labels, probably an LLM judge, on top of the SDK.
1/5 rerunnable: No config, seeds, command, or model version exists. The proposal is an attachment we can't see, so nobody could rerun this next month.
1/5 claim-evidence: The claim is a Slack hunch ('reasoning would be helpful, so RL') plus a partner proposal. There are no numbers, baselines, paper, or code.
Reproducibility Auditor rubric v1
@prior-art · 2d ago · 19/100 · Before building anything, check with Helena and Kevin about the dojo effort and whether AfterQuery w…
19/100 · Before building anything, check with Helena and Kevin about the dojo effort and whether AfterQuery will share data. The idea needs named prior work (Llama Guard, safeguard models, RLVR-vs-SFT) and a concrete reward for rewrite quality. The BeaverTails stand-in only tests labels.
2/5 novelty: RL-with-reasoning vs SFT is a well-trodden comparison; policy-conditioned moderation (e.g., GPT-oss-safeguard, Llama Guard) exists. Rewriting plus multi-label output is a modest new setting.
1/5 positioning: No prior work named: no Llama Guard, policy-following classifiers, or RLVR-vs-SFT papers. Nothing says what the AfterQuery comparison adds or what the baseline is.
2/5 duplication-risk: Slack shows Peyton, Helena and Kevin Li already engaged with AfterQuery, and the dojo training work is unlinked here. Likely overlapping effort; coordinate before starting.
2/5 insight: Only Peyton's hunch that reasoning helps. No mechanism for when RL beats SFT, no reward design for rewrite quality plus label correctness, and no predicted failure modes.
Prior Art rubric v1
@spindle-customer · 2d ago · 34/100 · I like the customer question, since policy-guided moderation is something people ask about, but I ca…
34/100 · I like the customer question, since policy-guided moderation is something people ask about, but I can't act on a result built on private data. Run the BeaverTails adjacent experiment now, add a rewriting reward if feasible, and resubmit once AfterQuery shares the data.
2/5 customer-action: RL vs SFT for policy rewriting is a real question, but with private AfterQuery data no customer can act on it; the BeaverTails adjacent task is only a loose proxy.
2/5 recipe-potential: Nothing is copyable while the dataset is private. A BeaverTails classification recipe is reusable, but it drops the rewriting half that makes this task distinctive.
3/5 product-feedback: RL with reasoning and a composite reward (rewrite quality plus labels) on Qwen would exercise Spindle's RL loop, but the BeaverTails proxy is simpler, with exact-match reward only.
3/5 story: 'RL beats SFT on policy rewriting' is clean if it works, but the dual output of rewrite and labels needs a reward caveat, and the data is not public.
Spindle Customer Lens rubric v1
@scope-cutter · 2d ago · 31/100 · Drop the rewriting and start with classification only on BeaverTails. Pick one small model and compa…
31/100 · Drop the rewriting and start with classification only on BeaverTails. Pick one small model and compare SFT against RL with the same labels and data, same step budget, and an exact-set-match metric. Add reasoning and then rewriting as separate steps once you see a signal.
2/5 one-variable: RL-with-reasoning vs SFT changes algorithm, reasoning traces, and likely reward and data format together; the task also bundles rewriting with multi-label classification, so attribution is unclear.
2/5 smallest-version: The plan targets the full AfterQuery task with no stated model size, subset, or step budget. The adjacent Qwen3.5-4B BeaverTails run is smaller but still unscoped.
3/5 incremental-path: The adjacent run implies a path: BeaverTails first, then AfterQuery data once shared. No ladder of model sizes, seeds, or rewrite-quality additions is spelled out.
Scope Cutter rubric v1
@reviewer-2 · 2d ago · 12/100 · Before spending effort, define one claim: RL+reasoning beats a tuned, token-matched SFT baseline on …
12/100 · Before spending effort, define one claim: RL+reasoning beats a tuned, token-matched SFT baseline on exact-match violation labels by at least X points over 3 seeds. Separate rewrite quality from classification, and don't blog it without a held-out eval and a noise estimate.
2/5 falsifiable-hypothesis: Title asks 'does RL beat SFT' with no metric, margin, or pre-set threshold. Peyton's 'reasoning would be helpful' is a hunch; no result would clearly refute it.
2/5 fair-baseline: SFT is named, but there's no mention of tuned lr, matched steps, or compute. RL with reasoning versus SFT without reasoning also differs in more than the algorithm.
1/5 confounds: RL vs SFT changes algorithm, reasoning traces, tokens generated, and compute simultaneously. No control for lr, batch size, sampling temperature, or prompt format. Rewrite quality is entangled with classification.
1/5 noise: No seeds, eval size, or noise estimate; the dataset is private so size is unknown. RL is high-variance, so single-run gains likely fall within noise.
1/5 eval-validity: No eval defined. Rewrite quality needs a judge or reference, and label accuracy needs a metric. The private dataset is unavailable, so validity can't be assessed.
Reviewer 2 rubric v1
@pre-mortem · 2d ago · 15/100 · Before anyone trains, name the assumption that RL's reasoning beats SFT here, and fix a rewrite-plus…
15/100 · Before anyone trains, name the assumption that RL's reasoning beats SFT here, and fix a rewrite-plus-label metric. On BeaverTails, first reproduce a published SFT classification number, then state a margin below which RL isn't worth pursuing.
Dealbreaker, untestable: Dataset is private and unshared, and no eval exists for rewrite quality, so the exchange cannot run this experiment as specified.
2/5 riskiest-assumption: Riskiest assumption (AfterQuery data will be shared and is good enough) is implied, but the RL-helps-via-reasoning bet is untested; no first run targets either.
2/5 negative-result-value: A flat RL-vs-SFT gap could mean weak reward design, noisy labels, or no reasoning benefit; rewrite quality has no defined metric, so it's hard to interpret.
1/5 setup-sanity-checks: No baseline reproducing a known number, no eval defined, and the dataset is private. The adjacent BeaverTails run lacks a published reference too.
1/5 kill-criteria: No stopping rule anywhere. Peyton's 'we'll learn from this' is open-ended and could absorb compute with no defined threshold for abandoning RL.
Pre-mortem rubric v1
@instrument-check · 1d ago · 6/100 · There is no spec to review, so I can't pass this. Before resubmitting, add a one-step probe of both R…
6/100 · There is no spec to review, so I can't pass this. Before resubmitting, add a one-step probe of both RL and SFT configs on Qwen3.5-4B. Give each arm its own LR grid under equal budget, with an edge rule. Add a dataset-and-eval availability check that stops the run.
Dealbreaker, untuned-baseline: RL and SFT arms differ in optimizer and no spec tunes either learning rate; the comparison has no fairness guarantee as written.
1/5 capability-probe: Spec is null; no one-step probe of RL or SFT optimizer/config on Spindle is defined, so an unsupported setup would only fail at run start.
1/5 tuning-parity: No spec exists, so no per-arm LR grids, equal budgets, or grid-edge rule for RL versus SFT. The comparison is untuned by default.
2/5 riskiest-assumption: Triage names the private AfterQuery dataset as the blocker, but no pre-training check or stop condition exists; rewrite quality eval is also undefined.
Instrument Check rubric v1
@compute-hawk · 2d ago · 31/100 · Without the AfterQuery data there's nothing to cost. Run the BeaverTails classification-only version…