Does DPPO's divergence mask beat GRPO clipping on GSM8K?
DPPO swaps PPO/GRPO ratio clipping for a trust region based on policy divergence; we compare GSM8K test accuracy against GRPO clipping at equal rollouts and updates.
Budget
2 GPU-h (default)
· each run plans 2 GPU-h, 120 min wall cap
· 4 runs
·
change
Reproducible on Spindle: yes
— DPPO replaces GRPO's ratio clipping with a divergence-based token mask. It is client-side loss code via forward_backward_custom in the cookbook RL loop on GSM8K. No fork is needed.
Spec
DPPO divergence-masked loss vs GRPO ratio-clipped loss, Qwen3.5-2B GSM8K LoRA RL
Hypothesis. On Qwen3.5-2B LoRA RL on GSM8K with 2 policy updates per rollout batch, the DPPO loss reaches higher greedy exact-match accuracy on the 1319-item GSM8K test set than the GRPO clipped-ratio loss. The lower bound of the paired 95% CI on DPPO minus GRPO must be above 0.
Method. The requested Qwen3.5-0.8B is not served, so we use the closest served small model, Qwen/Qwen3.5-2B. Both arms run the same client-side RL loop: the tinker-cookbook math_rl GSM8K env, renderer, prompts, reward and data order. Each step samples groups_per_batch=32 GSM8K train prompts with group_size=8 rollouts at temperature 1.0 and max_tokens=512.
Qwen/Qwen3.5-2B
RL
100 steps
batch 256
rank 32
gsm8k
eval GSM8K test (openai/gsm8k main/test) · greedy exact-match accuracy
2 GPU-h
120 min cap
Success criterion. On the 1319 GSM8K test items, the paired 95% Hoeffding CI for DPPO accuracy minus GRPO accuracy has a lower bound above 0.0, roughly a 7.5-point gap at this n. A smaller positive gap is reported as inconclusive, not validated.
full spec
{
"activation_gate": {
"applicable": true,
"checks": [
{
"max": 0.3,
"measures": "fraction of loss tokens masked by the DPPO divergence rule on the second minibatch update, mean over pilot steps",
"metric": "dppo_masked_token_fraction",
"min": 0.001
},
{
"measures": "90th percentile of |p_current - p_sampler| over loss tokens at the second update",
"metric": "policy_drift_abs_p_q90",
"min": 0.005
},
{
"measures": "fraction of tokens clipped by the eps=0.2 ratio clip in the baseline loss on the same pilot batches (confirms the regime is off-policy for the baseline too)",
"metric": "grpo_clip_fraction",
"min": 0.0005
},
{
"max": 2,
"measures": "optim_step calls per rollout batch as logged",
"metric": "executed_updates_per_batch",
"min": 2
},
{
"measures": "model name reported at client creation",
"metric": "executed_base_model",
"one_of": [
"Qwen/Qwen3.5-2B"
]
}
],
"reason": "DPPO only differs from GRPO on tokens whose divergence exceeds the trust region. A 5-step pilot of the treatment arm, with 2 updates per batch, must show that the mask fires on a nontrivial but not dominant share of tokens and that the policy actually drifts. Executed config values must match the plan."
},
"arms": [
{
"loss": "grpo_clip_eps0.2",
"lr": 2e-05,
"name": "grpo_seed0",
"seed": 0,
"updates_per_batch": 2
},
{
"loss": "dppo_divergence_mask",
"lr": 2e-05,
"name": "dppo_seed0",
"seed": 0,
"updates_per_batch": 2
}
],
"base_model": "Qwen/Qwen3.5-2B",
"baseline": "GRPO clipped-ratio loss (eps=0.2) through the same custom loss path. Identical rollouts per step, 2 minibatch updates per batch, lr, rank, seed, data order and eval.",
"batch_size": 256,
"budget": {
"gpu_hours": 2.0,
"max_wall_minutes": 120
},
"command": "Custom tinker SDK loop; the loss function is the custom part. Sketch:\n\nimport os, tinker\nsc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\ntc = sc.create_lora_training_client(base_model='Qwen/Qwen3.5-2B', rank=32)\n\n# Rollouts: reuse tinker_cookbook.recipes.math_rl GSM8K env and renderer.\n# group_size=8, groups_per_batch=32, max_tokens=512, seed=0.\nfor step in range(50):\n sampler = tc.save_weights_and_get_sampling_client(f's{step}')\n batch = rollout(sampler, gsm8k_train_prompts[step], group_size=8) # stores sampler logprobs as old_lp\n adv = group_mean_center(batch.rewards)\n for mb in split(batch, 2):\n tc.forward_backward_custom(mb.data, make_loss(LOSS, old_lp=mb.old_lp, adv=mb.adv)).result()\n tc.optim_step(tinker.AdamParams(learning_rate=2e-5)).result()\n\n# LOSS='grpo': -min(r*A, clip(r, 0.8, 1.2)*A)\n# LOSS='dppo': -(r*A) * keep_mask(div(p, p_old) <= delta), with div and delta pinned from the paper code\n# Final eval: greedy decoding on openai/gsm8k test (1319 items), numeric exact match.",
"correctness_preflight": {
"paths": [
{
"baseline_revision": "UNRESOLVED: pin tinker-cookbook commit whose stock ppo/importance-sampling loss defines GRPO clipping",
"id": "grpo_dppo_custom_loss",
"kind": "loss",
"reference_revision": "UNRESOLVED: pin commit of the official DPPO implementation for arXiv 2602.04879",
"tolerances": {
"baseline_gradient": {
"atol": 1e-05,
"min_cosine": 0.9999,
"rtol": 0.001
},
"baseline_loss": {
"atol": 1e-05,
"rtol": 0.0001
},
"reference_gradient": {
"atol": 1e-05,
"min_cosine": 0.9999,
"rtol": 0.001
},
"reference_loss": {
"atol": 1e-05,
"rtol": 0.0001
}
}
}
],
"required": true,
"status": "unresolved: DPPO reference code revision and exact divergence/threshold not yet pinned (paper digest failed); no full sweep until resolved"
},
"dataset": {
"hf_path": "openai/gsm8k",
"n_examples": 1600,
"name": "gsm8k",
"split": "main/train"
},
"decision_rule": {
"alpha": 0.05,
"criteria": [
{
"bounds": [
0,
1
],
"id": "primary",
"margin": 0.0,
"metric": "accuracy",
"weights": {
"baseline": -1,
"treatment": 1
}
}
],
"estimand": "Difference in greedy GSM8K test exact-match accuracy, DPPO minus GRPO, for these fixed final checkpoints of Qwen3.5-2B LoRA; larger is better",
"method": "paired_hoeffding",
"seeds": [
0
],
"unit": "fixed_checkpoint_items",
"version": 1
},
"eval": {
"aggregation": null,
"benchmark": "GSM8K test (openai/gsm8k main/test)",
"metric": "greedy exact-match accuracy",
"n_examples": 1319
},
"experiment_features": {
"adaptive_selection": false,
"checkpoint_restore": false,
"custom_loss": true,
"custom_optimizer": false,
"generated_training_data": false,
"heldout_loss": false,
"learned_generator": false,
"paired_comparison": true,
"paired_training_arms": true,
"warmup_adaptation": false
},
"feasibility": null,
"feasibility_check": {
"gpu_hours": null,
"known_gpu_hours": 0,
"phases": {},
"problems": [
"Measured feasibility not recorded"
],
"status": "unknown",
"wall_seconds": null
},
"feasibility_required": true,
"generation_audit": {
"answer_comparison": "numeric_exact",
"max_eval_overlaps": 0,
"max_label_errors": 0,
"min_sample_size": 100,
"required": false
},
"hypothesis": "On Qwen3.5-2B LoRA RL on GSM8K with 2 policy updates per rollout batch, the DPPO loss reaches higher greedy exact-match accuracy on the 1319-item GSM8K test set than the GRPO clipped-ratio loss. The lower bound of the paired 95% CI on DPPO minus GRPO must be above 0.",
"lora_rank": 32,
"max_seq_len": 1024,
"method": "The requested Qwen3.5-0.8B is not served, so we use the closest served small model, Qwen/Qwen3.5-2B.\n\nBoth arms run the same client-side RL loop: the tinker-cookbook math_rl GSM8K env, renderer, prompts, reward and data order. Each step samples groups_per_batch=32 GSM8K train prompts with group_size=8 rollouts at temperature 1.0 and max_tokens=512. Advantages are group-mean-centered (GRPO).\n\nThe rollout batch is split into 2 minibatches. That gives 2 optim_steps per batch, so the policy drifts from the sampling policy and the trust region can act. With one update per fresh batch, neither mask nor clip fires. Both losses are computed in one forked custom loss path through forward_backward_custom, using the sampler logprobs as p_old.\n\nThe arms differ in exactly one switch:\n- Baseline (GRPO): token loss -min(r*A, clip(r, 1-0.2, 1+0.2)*A).\n- Treatment (DPPO, arXiv 2602.04879): token loss -r*A, with tokens masked out when the divergence between the current and sampling policy exceeds the threshold delta in the direction the advantage pushes. The divergence is estimated from the token probabilities, e.g. |p - p_old|, rather than the single-sample ratio.\n\nTriage only received a truncated digest of the paper. The exact divergence estimator, delta and masking direction must be pinned from the paper's released code before any training. Until then the correctness preflight is unresolved and no full sweep runs.\n\nTraining: LoRA rank 32, AdamW lr 2e-5, 50 rollout steps (100 optim steps) per arm, seed 0. Both arms start from the same base weights.\n\nEvaluation: the final checkpoint of each arm is scored with greedy decoding on all 1319 GSM8K test items, using the cookbook's numeric answer grader.",
"optimizer": {
"lr": 2e-05,
"name": "adam"
},
"primary_endpoint": "gsm8k_test_greedy_exact_match_accuracy@final_checkpoint (1319 items)",
"restoration_preflight": {
"applicable": true,
"fixed_batch_required": true,
"groups": [
{
"arms": [
{
"name": "grpo_seed0",
"optimizer": "adam",
"optimizer_config": {
"beta1": 0.9,
"beta2": 0.95,
"eps": 1e-08,
"learning_rate": 2e-05
}
},
{
"name": "dppo_seed0",
"optimizer": "adam",
"optimizer_config": {
"beta1": 0.9,
"beta2": 0.95,
"eps": 1e-08,
"learning_rate": 2e-05
}
}
]
}
],
"reason": "Both arms are LoRA-trained from the same base. Before the first update, verify identical initial state: compare the logprobs on a fixed tokenized GSM8K batch across both freshly created clients, which must be bitwise or within 1e-6. Also verify the active optimizer is AdamW with the planned config after creation. Server identity cannot be introspected and stays unverified."
},
"steps": 100,
"success_criterion": "On the 1319 GSM8K test items, the paired 95% Hoeffding CI for DPPO accuracy minus GRPO accuracy has a lower bound above 0.0, roughly a 7.5-point gap at this n. A smaller positive gap is reported as inconclusive, not validated.",
"title": "DPPO divergence-masked loss vs GRPO ratio-clipped loss, Qwen3.5-2B GSM8K LoRA RL",
"training_type": "rl"
}previous versions (1)
DPPO loss vs GRPO loss for LoRA RL on GSM8K
spec
{
"base_model": "Qwen/Qwen3.5-0.8B",
"baseline": "Same model, LoRA rank, learning rate, seed, rollout settings and step count, trained with the stock tinker_cookbook math_rl GRPO-style loss.",
"batch_size": 256,
"budget": {
"gpu_hours": 2.0,
"max_wall_minutes": 120
},
"command": "Baseline: python -m tinker_cookbook.recipes.math_rl.train model_name=Qwen/Qwen3.5-0.8B base_url=$TINKER_BASE_URL env=gsm8k group_size=8 groups_per_batch=32 learning_rate=4e-5 max_tokens=512 max_steps=100 log_path=/tmp/rx-dppo-baseline. Treatment (tinker SDK script, custom part is dppo_loss): import os, tinker; sc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL']); tc = sc.create_lora_training_client(base_model='Qwen/Qwen3.5-0.8B', rank=32); for step in range(100): sampler = tc.save_weights_and_get_sampling_client(name=f's{step}'); data, old_logprobs, advantages = rollout_gsm8k(sampler, groups=32, group_size=8, max_tokens=512) # same rollout and advantage code as the cookbook loop; tc.forward_backward_custom(data, lambda d, logprobs: dppo_loss(logprobs, old_logprobs, advantages)).result() # dppo_loss implements the arXiv 2602.04879 objective; tc.optim_step(tinker.AdamParams(learning_rate=4e-5)).result(); then tc.save_weights_for_sampler(name='dppo-final') and evaluate both LoRAs on GSM8K test.",
"dataset": {
"hf_path": "openai/gsm8k",
"n_examples": 7473,
"name": "GSM8K",
"split": "train"
},
"eval": {
"benchmark": "GSM8K test",
"metric": "exact-match accuracy, greedy decoding",
"n_examples": 1319
},
"hypothesis": "With the same rollouts, rewards and group-normalized advantages, the DPPO loss raises Qwen3.5-0.8B GSM8K test exact-match accuracy by at least 2 points over the cookbook GRPO-style loss after 100 steps.",
"lora_rank": 32,
"max_seq_len": 1024,
"method": "Train a rank-32 LoRA on Qwen/Qwen3.5-0.8B with RL on the GSM8K train split, using a binary correct-answer reward. Both arms share the same sampling setup: group size 8, 32 groups per batch, max 512 tokens, temperature 1.0, and group-mean advantages. The baseline uses the stock cookbook loss. The treatment changes only the policy loss. It implements the DPPO objective exactly as written in arXiv 2602.04879, as a custom per-token loss through forward_backward_custom, and uses the paper's default hyperparameters. The custom part is the loss function only. Rollouts and advantages reuse the same code as the baseline. Both final LoRAs are evaluated with greedy decoding on the full GSM8K test set. Training reward curves and policy entropy are also logged as a stability check.",
"optimizer": {
"lr": 4e-05,
"name": "adam"
},
"steps": 100,
"success_criterion": "Validated if DPPO accuracy is at least 2.0 points above the baseline on the 1,319 GSM8K test examples, which is beyond the roughly 1.3-point standard error. Failed if DPPO is below the baseline by more than 2.0 points. Anything in between counts as a tie: no accuracy gain, so first-class support would rest on stability or customer demand alone.",
"title": "DPPO loss vs GRPO loss for LoRA RL on GSM8K",
"training_type": "rl"
}Runs
Source
raw text
Abhi Nambiar will be in the office today. he's already used spindle a bit to do some depoliticizing experiments for qwen, could pick his brains on how it's been so far [jason.mancuso] oh wat, I'm drafting up something about this rn [micah] Notes from quick sync: He tried Spindle and found that it was more expensive out of the box than Tinker, but this is because 1) was running sync RL and 2) he was only a single LoRA client at a time. Bugs/feature requests: sampler pools kept scaling down during training; He asked about DPPO loss support (https://arxiv.org/html/2602.04879v1) and wondered if we can add first-class support for it (kailash will own); He wondered whether we're using spec decode during inference rollouts; He wants the DS 4.1 Flash 1017 update.
links triage read (1)
triage output
{
"adjacent_experiment": "",
"reproducibility_reason": "DPPO replaces GRPO's ratio clipping with a divergence-based token mask. It is client-side loss code via forward_backward_custom in the cookbook RL loop on GSM8K. No fork is needed.",
"reproducible_on_spindle": true,
"sources": [
{
"brief": null,
"error": "digest failed: ask_json failed after 2 attempts: no JSON object in response: '{\"title\":\"Rethinking the Trust Region in LLM Reinforcement Learning\",\"claim\":\"PPO/GRPO ratio clipping is a noisy single-sample estimate of policy divergence. It over-constrains low-probability tokens,'",
"full_text": true,
"kind": "paper",
"title": "Rethinking the Trust Region in LLM Reinforcement Learning",
"url": "https://arxiv.org/html/2602.04879v1"
}
],
"spec": {
"activation_gate": {
"applicable": true,
"checks": [
{
"max": 0.3,
"measures": "fraction of loss tokens masked by the DPPO divergence rule on the second minibatch update, mean over pilot steps",
"metric": "dppo_masked_token_fraction",
"min": 0.001
},
{
"measures": "90th percentile of |p_current - p_sampler| over loss tokens at the second update",
"metric": "policy_drift_abs_p_q90",
"min": 0.005
},
{
"measures": "fraction of tokens clipped by the eps=0.2 ratio clip in the baseline loss on the same pilot batches (confirms the regime is off-policy for the baseline too)",
"metric": "grpo_clip_fraction",
"min": 0.0005
},
{
"max": 2,
"measures": "optim_step calls per rollout batch as logged",
"metric": "executed_updates_per_batch",
"min": 2
},
{
"measures": "model name reported at client creation",
"metric": "executed_base_model",
"one_of": [
"Qwen/Qwen3.5-2B"
]
}
],
"reason": "DPPO only differs from GRPO on tokens whose divergence exceeds the trust region. A 5-step pilot of the treatment arm, with 2 updates per batch, must show that the mask fires on a nontrivial but not dominant share of tokens and that the policy actually drifts. Executed config values must match the plan."
},
"arms": [
{
"loss": "grpo_clip_eps0.2",
"lr": 2e-05,
"name": "grpo_seed0",
"seed": 0,
"updates_per_batch": 2
},
{
"loss": "dppo_divergence_mask",
"lr": 2e-05,
"name": "dppo_seed0",
"seed": 0,
"updates_per_batch": 2
}
],
"base_model": "Qwen/Qwen3.5-2B",
"baseline": "GRPO clipped-ratio loss (eps=0.2) through the same custom loss path. Identical rollouts per step, 2 minibatch updates per batch, lr, rank, seed, data order and eval.",
"batch_size": 256,
"budget": {
"gpu_hours": 2.0,
"max_wall_minutes": 120
},
"command": "Custom tinker SDK loop; the loss function is the custom part. Sketch:\n\nimport os, tinker\nsc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\ntc = sc.create_lora_training_client(base_model='Qwen/Qwen3.5-2B', rank=32)\n\n# Rollouts: reuse tinker_cookbook.recipes.math_rl GSM8K env and renderer.\n# group_size=8, groups_per_batch=32, max_tokens=512, seed=0.\nfor step in range(50):\n sampler = tc.save_weights_and_get_sampling_client(f's{step}')\n batch = rollout(sampler, gsm8k_train_prompts[step], group_size=8) # stores sampler logprobs as old_lp\n adv = group_mean_center(batch.rewards)\n for mb in split(batch, 2):\n tc.forward_backward_custom(mb.data, make_loss(LOSS, old_lp=mb.old_lp, adv=mb.adv)).result()\n tc.optim_step(tinker.AdamParams(learning_rate=2e-5)).result()\n\n# LOSS='grpo': -min(r*A, clip(r, 0.8, 1.2)*A)\n# LOSS='dppo': -(r*A) * keep_mask(div(p, p_old) <= delta), with div and delta pinned from the paper code\n# Final eval: greedy decoding on openai/gsm8k test (1319 items), numeric exact match.",
"correctness_preflight": {
"paths": [
{
"baseline_revision": "UNRESOLVED: pin tinker-cookbook commit whose stock ppo/importance-sampling loss defines GRPO clipping",
"id": "grpo_dppo_custom_loss",
"kind": "loss",
"reference_revision": "UNRESOLVED: pin commit of the official DPPO implementation for arXiv 2602.04879",
"tolerances": {
"baseline_gradient": {
"atol": 1e-05,
"min_cosine": 0.9999,
"rtol": 0.001
},
"baseline_loss": {
"atol": 1e-05,
"rtol": 0.0001
},
"reference_gradient": {
"atol": 1e-05,
"min_cosine": 0.9999,
"rtol": 0.001
},
"reference_loss": {
"atol": 1e-05,
"rtol": 0.0001
}
}
}
],
"required": true,
"status": "unresolved: DPPO reference code revision and exact divergence/threshold not yet pinned (paper digest failed); no full sweep until resolved"
},
"dataset": {
"hf_path": "openai/gsm8k",
"n_examples": 1600,
"name": "gsm8k",
"split": "main/train"
},
"decision_rule": {
"alpha": 0.05,
"criteria": [
{
"bounds": [
0,
1
],
"id": "primary",
"margin": 0.0,
"metric": "accuracy",
"weights": {
"baseline": -1,
"treatment": 1
}
}
],
"estimand": "Difference in greedy GSM8K test exact-match accuracy, DPPO minus GRPO, for these fixed final checkpoints of Qwen3.5-2B LoRA; larger is better",
"method": "paired_hoeffding",
"seeds": [
0
],
"unit": "fixed_checkpoint_items",
"version": 1
},
"eval": {
"aggregation": null,
"benchmark": "GSM8K test (openai/gsm8k main/test)",
"metric": "greedy exact-match accuracy",
"n_examples": 1319
},
"experiment_features": {
"adaptive_selection": false,
"checkpoint_restore": false,
"custom_loss": true,
"custom_optimizer": false,
"generated_training_data": false,
"heldout_loss": false,
"learned_generator": false,
"paired_comparison": true,
"paired_training_arms": true,
"warmup_adaptation": false
},
"feasibility": null,
"feasibility_check": {
"gpu_hours": null,
"known_gpu_hours": 0,
"phases": {},
"problems": [
"Measured feasibility not recorded"
],
"status": "unknown",
"wall_seconds": null
},
"feasibility_required": true,
"generation_audit": {
"answer_comparison": "numeric_exact",
"max_eval_overlaps": 0,
"max_label_errors": 0,
"min_sample_size": 100,
"required": false
},
"hypothesis": "On Qwen3.5-2B LoRA RL on GSM8K with 2 policy updates per rollout batch, the DPPO loss reaches higher greedy exact-match accuracy on the 1319-item GSM8K test set than the GRPO clipped-ratio loss. The lower bound of the paired 95% CI on DPPO minus GRPO must be above 0.",
"lora_rank": 32,
"max_seq_len": 1024,
"method": "The requested Qwen3.5-0.8B is not served, so we use the closest served small model, Qwen/Qwen3.5-2B.\n\nBoth arms run the same client-side RL loop: the tinker-cookbook math_rl GSM8K env, renderer, prompts, reward and data order. Each step samples groups_per_batch=32 GSM8K train prompts with group_size=8 rollouts at temperature 1.0 and max_tokens=512. Advantages are group-mean-centered (GRPO).\n\nThe rollout batch is split into 2 minibatches. That gives 2 optim_steps per batch, so the policy drifts from the sampling policy and the trust region can act. With one update per fresh batch, neither mask nor clip fires. Both losses are computed in one forked custom loss path through forward_backward_custom, using the sampler logprobs as p_old.\n\nThe arms differ in exactly one switch:\n- Baseline (GRPO): token loss -min(r*A, clip(r, 1-0.2, 1+0.2)*A).\n- Treatment (DPPO, arXiv 2602.04879): token loss -r*A, with tokens masked out when the divergence between the current and sampling policy exceeds the threshold delta in the direction the advantage pushes. The divergence is estimated from the token probabilities, e.g. |p - p_old|, rather than the single-sample ratio.\n\nTriage only received a truncated digest of the paper. The exact divergence estimator, delta and masking direction must be pinned from the paper's released code before any training. Until then the correctness preflight is unresolved and no full sweep runs.\n\nTraining: LoRA rank 32, AdamW lr 2e-5, 50 rollout steps (100 optim steps) per arm, seed 0. Both arms start from the same base weights.\n\nEvaluation: the final checkpoint of each arm is scored with greedy decoding on all 1319 GSM8K test items, using the cookbook's numeric answer grader.",
"optimizer": {
"lr": 2e-05,
"name": "adam"
},
"primary_endpoint": "gsm8k_test_greedy_exact_match_accuracy@final_checkpoint (1319 items)",
"restoration_preflight": {
"applicable": true,
"fixed_batch_required": true,
"groups": [
{
"arms": [
{
"name": "grpo_seed0",
"optimizer": "adam",
"optimizer_config": {
"beta1": 0.9,
"beta2": 0.95,
"eps": 1e-08,
"learning_rate": 2e-05
}
},
{
"name": "dppo_seed0",
"optimizer": "adam",
"optimizer_config": {
"beta1": 0.9,
"beta2": 0.95,
"eps": 1e-08,
"learning_rate": 2e-05
}
}
]
}
],
"reason": "Both arms are LoRA-trained from the same base. Before the first update, verify identical initial state: compare the logprobs on a fixed tokenized GSM8K batch across both freshly created clients, which must be bitwise or within 1e-6. Also verify the active optimizer is AdamW with the planned config after creation. Server identity cannot be introspected and stays unverified."
},
"steps": 100,
"success_criterion": "On the 1319 GSM8K test items, the paired 95% Hoeffding CI for DPPO accuracy minus GRPO accuracy has a lower bound above 0.0, roughly a 7.5-point gap at this n. A smaller positive gap is reported as inconclusive, not validated.",
"title": "DPPO divergence-masked loss vs GRPO ratio-clipped loss, Qwen3.5-2B GSM8K LoRA RL",
"training_type": "rl"
},
"summary": "DPPO swaps PPO/GRPO ratio clipping for a trust region based on policy divergence; we compare GSM8K test accuracy against GRPO clipping at equal rollouts and updates.",
"tags": [
"rl",
"paper",
"custom-loss",
"lora",
"grpo"
],
"title": "Does DPPO's divergence mask beat GRPO clipping on GSM8K?",
"value_estimate": 4,
"value_rationale": "A paying user asked for first-class DPPO support, and a clean head-to-head against GRPO tells us whether to ship it as a built-in loss. Triage only saw a truncated digest of the paper, so the exact DPPO rule must be pinned from the source first."
}Comments
@reproducibility · 2d ago · 61/100 · Data is solid, but the treatment is a hand-written loss with no pinned seed and no reference impleme…
61/100 · Data is solid, but the treatment is a hand-written loss with no pinned seed and no reference implementation. Pin the seed and model revision, and cite the paper's reported GSM8K numbers and any official code so we can verify your dppo_loss against them.
5/5 data-available: Exact hf_path openai/gsm8k, train split (7,473) for training and GSM8K test (1,319) for eval; both public and precisely named.
3/5 stock-tooling: Baseline is a cookbook math_rl recipe, but the treatment needs a hand-written dppo_loss and a custom loop via forward_backward_custom, so custom code on top of the SDK.
3/5 rerunnable: Commands and hyperparameters are recorded, but no seed value is pinned, the treatment script is pseudocode, and the model version and paper hyperparameters aren't fixed.
2/5 claim-evidence: Request comes from a verbal customer ask in meeting notes; the arXiv paper exists, but no paper numbers or reference code are cited to compare against.
Reproducibility Auditor rubric v1
@prior-art · 2d ago · 31/100 · Talk to Kailash first: he owns DPPO support, and your custom-loss run may be his validation test. Th…
31/100 · Talk to Kailash first: he owns DPPO support, and your custom-loss run may be his validation test. Then state what DPPO changes versus GRPO's clipping and why that matters on 0.8B GSM8K. As written, a single seed can't resolve a 2-point gap.
3/5 novelty: Known paper's loss applied to Qwen3.5-0.8B LoRA on GSM8K; a small-model, small-task setting where a 2-point gain isn't obvious, but it is essentially a reproduction.
2/5 positioning: Cites arXiv 2602.04879 but never says what DPPO changes versus GRPO/PPO clipping, nor what the test adds beyond the paper's own results.
2/5 duplication-risk: Source says kailash will own first-class DPPO support; the idea doesn't link or coordinate with that, so a parallel implementation is likely.
2/5 insight: No mechanism argued for why DPPO should beat GRPO on GSM8K at 100 steps; hypothesis is just 'paper's loss is better' with a threshold.
Prior Art rubric v1
@reviewer-2 · 2d ago · 48/100 · A 2-point threshold on one seed per arm can't separate DPPO from RL seed noise. Run at least 3 seeds…
48/100 · A 2-point threshold on one seed per arm can't separate DPPO from RL seed noise. Run at least 3 seeds per arm, sweep lr for both losses, and send the baseline through your custom loop with a GRPO loss, so only the loss differs.
4/5 falsifiable-hypothesis: Clear claim: DPPO ≥2.0 points over GRPO on GSM8K test after 100 steps, with a pre-set failure and tie band. Slightly weakened by a single-seed point estimate.
2/5 fair-baseline: Same lr 4e-5 for both; DPPO uses paper defaults while the baseline is untuned. The stock cookbook loop and the custom SDK loop are also different code paths.
2/5 confounds: Baseline uses cookbook train.py, treatment a hand-written loop; hidden differences (on-policy vs async, advantage handling, KL, loss normalization, lr schedule, ratio clipping) are uncontrolled. Loss-only claim unverified.
2/5 noise: One seed per arm; 1.3-pt SE covers only test sampling. RL seed variance on a 0.8B model over 100 steps probably exceeds 2 points. No seed estimate.
4/5 eval-validity: Full GSM8K test (1,319), greedy exact match, held out from the train split. Missing: a pre-training base-model score, and the answer-extraction format is not specified.
Reviewer 2 rubric v1
@scope-cutter · 2d ago · 44/100 · Cut it down first. Run a 20-30 step pilot on both arms, checking reward and entropy curves plus a re…
44/100 · Cut it down first. Run a 20-30 step pilot on both arms, checking reward and entropy curves plus a reimplementation sanity check. Evaluate on a subset, and use the cookbook loop for both arms. Only then run 100 steps with 3 seeds, since one seed can't resolve a 2-point gap.
4/5 one-variable: Only the policy loss differs; rollouts, advantages, seed and LR are shared. But the baseline runs via the cookbook while treatment runs via a custom script, which adds an incidental implementation difference.
2/5 smallest-version: Batch of 256 rollouts (32×8) for 100 steps and a full 1,319-example eval on two runs is bigger than needed. A 20-30 step run with a subset eval would give a first signal.
2/5 incremental-path: Success and tie thresholds are defined, but no ladder is spelled out: no seeds, larger model, or paper-hyperparameter sweep follows. A single seed with a 2-point threshold can't separate signal from noise.
Scope Cutter rubric v1
@spindle-customer · 2d ago · 75/100 · This is a real customer ask, and the custom-loss path is worth testing. But one seed at 100 steps on…
75/100 · This is a real customer ask, and the custom-loss path is worth testing. But one seed at 100 steps on a 0.8B model will probably land in your tie band. Add seeds, or log stability and cost, so a tie still informs the ship decision.
4/5 customer-action: A named customer asked for DPPO; result decides first-class loss versus custom-loss recipe. Limited by one customer and a 0.8B model on GSM8K.
4/5 recipe-potential: A dppo_loss via forward_backward_custom beside the cookbook math_rl baseline is a copyable before/after, though the inline rollout script needs cleanup.
4/5 product-feedback: Exercises forward_backward_custom in a real RL loop with per-step sampler saves on Qwen3.5-0.8B; likely to surface custom-loss and sampler bugs.
4/5 story: 'DPPO vs GRPO on GSM8K: X vs Y points' is clean, but the ±1.3 SE and tie outcome add one caveat.
Spindle Customer Lens rubric v1
@compute-hawk · 2d ago · 44/100 · Feasible, but only just: run a 3-step timing test first, then run both arms in parallel. A single se…
44/100 · Feasible, but only just: run a 3-step timing test first, then run both arms in parallel. A single seed with a 2-point threshold is too noisy to be decisive, so log entropy and reward curves and plan a second seed or paired bootstrap.
3/5 budget-fit: About 7.7M sampled tokens per arm (256 rollouts × ~300 tokens × 100 steps), plus a custom-loss forward pass and per-step sampler sync. Two arms plus eval fit in 2 hours, with no room for a second seed.
3/5 signal-per-gpu-hour: Shared rollouts and advantages are sensible. But one seed per arm and a 2-point bar against about 1.3 SE mean seed noise may decide the outcome. A paired comparison or a stability metric would be cheaper.
2/5 small-scale-validity: DPPO's benefit is probably about stability and train/inference mismatch, which may not appear at 0.8B with LoRA over 100 steps. A tie would say little about larger scale.
3/5 throughput-risk: Sync RL needs a sampler refresh each step, and Abhi saw sampler pools scale down mid-training. Custom loss adds an extra forward pass. I need a per-step timing estimate from a 3-step smoke test before launching.
Compute Hawk rubric v1
anonymous · 2d ago
Started a session.
@instrument-check · 1d ago · 38/100 · I'd hold this until each arm gets its own LR grid at equal budget, with a rule to extend the grid wh…
38/100 · I'd hold this until each arm gets its own LR grid at equal budget, with a rule to extend the grid when the best LR sits on the edge. The activation gate and loss preflight are good, but run the baseline through the same pilot, and use more than one seed.
Dealbreaker, untuned-baseline: GRPO and DPPO share one untuned lr (2e-5) with a single seed, so a DPPO-vs-GRPO gap may just reflect which loss tolerates this lr.
3/5 capability-probe: The 5-step pilot checks base model and update count, but it covers only the DPPO arm. The baseline's exact client and config get no explicit one-step probe. 'adam' vs AdamW is inconsistent.
1/5 tuning-parity: Both arms share one fixed lr 2e-5 and seed 0, with no per-arm grid and no edge rule. 2e-5 is low for LoRA RL and unjustified.
4/5 riskiest-assumption: The activation_gate tests that the mask fires and the policy drifts, and unresolved preflight blocks the sweep. Stop-on-failure is implied, and the assumption is not labeled as riskiest.
Instrument Check rubric v1
@pre-mortem · 2d ago · 38/100 · Before the DPPO arm, I'd run the custom-loss path with the GRPO loss and confirm it matches the stoc…