🔬Modal Research Exchange

{
  "activation_gate": {
    "applicable": true,
    "checks": [
      {
        "measures": "model name reported by the created training client",
        "metric": "executed_base_model",
        "one_of": [
          "Qwen/Qwen3.5-4B"
        ]
      },
      {
        "measures": "loss_fn passed to forward_backward in both arms",
        "metric": "executed_loss_fn",
        "one_of": [
          "importance_sampling"
        ]
      },
      {
        "max": 0.95,
        "measures": "share of pilot rollouts where the judge flags at least one dimension, with non-empty critique text",
        "metric": "frac_rollouts_with_judge_failure_feedback",
        "min": 0.2
      },
      {
        "measures": "share of real assistant loss tokens with |log p_teacher - log p_student| > 0.05 nats",
        "metric": "frac_assistant_tokens_abs_sdpo_adv_gt_0.05",
        "min": 0.05
      },
      {
        "max": 10.0,
        "measures": "99th percentile |A_t| over assistant tokens before clipping; caps runaway teacher disagreement",
        "metric": "p99_abs_sdpo_adv_nats",
        "min": 0.1
      },
      {
        "measures": "share of GRPO groups whose 4 rubric rewards are not all equal",
        "metric": "frac_groups_nonzero_reward_variance",
        "min": 0.25
      }
    ],
    "reason": "SDPO only differs from GRPO if the judge produces failures with critiques, and if the teacher conditioned on that feedback actually shifts token probabilities. GRPO needs nonzero within-group reward variance. Executed model, loss and advantage source must match the spec."
  },
  "arms": [
    {
      "advantage": "group-centered scalar rubric reward",
      "lr": 4e-05,
      "name": "grpo",
      "seed": 0
    },
    {
      "advantage": "per-token log p_teacher - log p_student, frozen initial teacher",
      "lr": 4e-05,
      "name": "sdpo_token",
      "seed": 0
    }
  ],
  "base_model": "Qwen/Qwen3.5-4B",
  "baseline": "GRPO with the same env, user simulator, judge, tasks, task order, rollouts, LoRA config, lr, steps and eval. Only the advantage estimator differs.",
  "batch_size": 32,
  "budget": {
    "gpu_hours": 2.0,
    "max_wall_minutes": 120
  },
  "command": "Custom tinker SDK script (rx_sdpo_tau.py), written against tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL']).\n\nClients:\n- tc = create_lora_training_client(base_model='Qwen/Qwen3.5-4B', rank=32).\n- Frozen samplers for the user simulator and judge: Qwen/Qwen3.5-9B base sampling client.\n- Frozen teacher: Qwen3.5-4B base sampling client.\n\nEach of 20 steps:\n1. Take the next 8 train tasks (same fixed order for both arms).\n2. Sample 4 multi-turn rollouts per task via save_weights_for_sampler + sampling client (temperature 1.0, max 1024 tokens per turn, max 30 turns).\n3. Call the judge once per rollout, cached and shared logic for both arms.\n4. Compute advantages. GRPO uses reward minus group mean. SDPO (the custom part) calls teacher compute_logprobs on the feedback-conditioned prompt over the student's assistant tokens, then A_t = clip(lp_teacher - lp_student, -2, 2).\n5. tc.forward_backward(datums, loss_fn='importance_sampling') with the student sampling logprobs, then tc.optim_step(tinker.AdamParams(learning_rate=4e-5, beta1=0.9, beta2=0.95, eps=1e-8)).\n\nEval: at step 20, run 1 trial on each of the 115 retail test tasks, scored by the tau-bench ground-truth reward. Run the held-out gpt-oss-120b judge for per-dimension rates. Save per-task outcomes for the paired analysis.\n\nActivation-failure plan, in order:\n(a) Offline first. Forced activation: inject a known failure critique and check A_t is nonzero. Forced non-activation: give the teacher an empty-feedback, identical prompt and check A_t is about 0. Gradient check: the SDPO path with advantages set to GRPO values reproduces the stock loss and gradients within the correctness_preflight tolerances.\n(b) A 3-step treatment pilot is logged against activation_gate and preserved even if it fails.\n(c) If the gate fails, the only pre-specified calibration is one change to the judge rubric prompt's strictness, selected on 16 train tasks (never test tasks). Cap: 3 updates and 15 wall-minutes. Any change is recorded as a revised spec/deviation before the full run. Gates are never relaxed.",
  "correctness_preflight": {
    "paths": [
      {
        "baseline_revision": "UNRESOLVED: pin tinker-cookbook commit SHA (stock rl loop GRPO advantages + importance_sampling loss)",
        "id": "sdpo_token_advantage_into_importance_sampling",
        "kind": "loss",
        "reference_revision": "UNRESOLVED: pin SDPO authors' released code commit for arXiv 2601.20802 (token-level advantage); fallback is a float64 numpy reimplementation of the paper's equation, reviewed before running",
        "tolerances": {
          "baseline_gradient": {
            "atol": 0.0001,
            "min_cosine": 0.999,
            "rtol": 0.01
          },
          "baseline_loss": {
            "atol": 0.0001,
            "rtol": 0.001
          },
          "reference_gradient": {
            "atol": 0.001,
            "min_cosine": 0.995,
            "rtol": 0.02
          },
          "reference_loss": {
            "atol": 0.001,
            "rtol": 0.005
          }
        }
      }
    ],
    "required": true,
    "status": "unresolved: commit SHAs for tinker-cookbook and the SDPO reference code must be pinned before any full run; no full sweep until pinned"
  },
  "dataset": {
    "hf_path": null,
    "n_examples": 160,
    "name": "tau-bench retail tasks (github.com/sierra-research/tau-bench, pinned commit; not on HF)",
    "split": "train (training) / test (115 tasks, eval)"
  },
  "decision_rule": {
    "alpha": 0.05,
    "criteria": [
      {
        "bounds": [
          0,
          1
        ],
        "id": "primary",
        "margin": 0.03,
        "metric": "tau_bench_retail_pass1",
        "weights": {
          "baseline": -1,
          "treatment": 1
        }
      }
    ],
    "estimand": "Mean per-task difference in tau-bench retail test ground-truth success (pass^1), sdpo_token minus grpo, at the fixed step-20 checkpoints; larger is better",
    "method": "paired_hoeffding",
    "seeds": [
      0
    ],
    "unit": "fixed_checkpoint_items",
    "version": 1
  },
  "eval": {
    "aggregation": null,
    "benchmark": "tau-bench retail test",
    "metric": "tau_bench_retail_pass1 (ground-truth task success, 1 trial per task)",
    "n_examples": 115
  },
  "experiment_features": {
    "adaptive_selection": false,
    "checkpoint_restore": false,
    "custom_loss": true,
    "custom_optimizer": false,
    "generated_training_data": false,
    "heldout_loss": false,
    "learned_generator": false,
    "paired_comparison": true,
    "paired_training_arms": true,
    "warmup_adaptation": false
  },
  "feasibility": null,
  "feasibility_check": {
    "gpu_hours": null,
    "known_gpu_hours": 0,
    "phases": {},
    "problems": [
      "Measured feasibility not recorded"
    ],
    "status": "unknown",
    "wall_seconds": null
  },
  "feasibility_required": true,
  "generation_audit": {
    "answer_comparison": "categorical_exact",
    "max_eval_overlaps": 0,
    "max_label_errors": 0,
    "min_sample_size": 100,
    "required": false
  },
  "hypothesis": "With the same rollouts, judge calls and steps, token-level SDPO raises tau-bench retail test pass^1 over GRPO by at least 3 points. Its gain is largest on invalid-tool-call rate.",
  "lora_rank": 32,
  "max_seq_len": 8192,
  "method": "Both arms train a LoRA (rank 32) on Qwen/Qwen3.5-4B, thinking off, in a client-side multi-turn tau-bench retail environment built from the public tau-bench repo (pinned commit). The retail train tasks are used for training and the 115 retail test tasks are held out.\n\nShared rollout setup:\n- A frozen user simulator (Qwen/Qwen3.5-9B base sampling client) plays the customer.\n- A frozen rubric judge (Qwen/Qwen3.5-9B, fixed prompt) reads each trajectory and the policy doc. It returns text feedback plus 0/1 scores on three dimensions: invalid tool use, ungrounded policy/fact claims, and clarification-before-action when the request is ambiguous.\n- The scalar reward is the mean of the three dimension scores.\n- Each step uses 8 tasks x 4 rollouts, temperature 1.0, AdamW at lr 4e-5, for 20 steps.\n- Only assistant tokens get loss. User and tool tokens are masked.\n\nThe arms differ in exactly one thing, the advantage estimator fed to the stock importance_sampling loss:\n- Baseline: GRPO. Group-centered scalar judge reward with no std normalization, broadcast to all assistant tokens.\n- Treatment: token-level SDPO, the custom part. A teacher context is built from the conversation prefix, the judge feedback text, a successful sibling rollout from the same group if one exists, and 'Respond correctly to the customer.' The student's own trajectory is not shown to the teacher. The teacher is the frozen initial model (paper Table 4: frozen teacher 48.8 vs trust-region 50.6), scored via compute_logprobs on the student's sampled assistant tokens. The per-token advantage is A_t = log p_teacher(y_t) - log p_student(y_t), clipped to [-2, 2].\n\nDeviations from the paper:\n- Logit-level top-K KL is replaced by token-level, since only sampled-token logprobs are assumed. The paper ranks token-level below logit-level but above GRPO.\n- The model is 4B rather than 8B, to fit the 2 GPU-hour budget.\n\nThe primary eval is tau-bench's own ground-truth reward (database state plus required outputs), so it is independent of the training judge. Per-dimension rates are secondary and scored by a different held-out judge (openai/gpt-oss-120b) to expose judge gaming.",
  "optimizer": {
    "lr": 4e-05,
    "name": "adam"
  },
  "preflight": {
    "probe": "Create training clients for both arms with the exact adam config (lr 4e-5, rank 32). Run one forward_backward(importance_sampling) + optim_step on one recorded tau-bench rollout with GRPO advantages, then with SDPO advantages from teacher compute_logprobs. Confirm compute_logprobs returns per-token logprobs aligned to assistant tokens, and that the activation_gate token-advantage fraction is nonzero on that rollout.",
    "riskiest_assumption": "A 4B model conditioned on rubric-judge critique assigns meaningfully different token probabilities than the student, so SDPO advantages carry signal rather than noise. The paper shows SDPO weakens at small scale.",
    "tuning": "The arms share the optimizer and differ only in the advantage estimator, so both use the same fixed lr 4e-5 (a common LoRA scale-up of the paper's full-FT 1e-6/1e-5). No per-arm LR grid fits the 2 GPU-hour budget. A follow-up with more budget should give each arm a 3-point grid selected on held-out train tasks, with the grid-edge rule applied."
  },
  "primary_endpoint": "tau_bench_retail_pass1",
  "restoration_preflight": {
    "applicable": true,
    "fixed_batch_required": true,
    "groups": [
      {
        "arms": [
          {
            "name": "grpo",
            "optimizer": "adam",
            "optimizer_config": {
              "beta1": 0.9,
              "beta2": 0.95,
              "eps": 1e-08,
              "learning_rate": 4e-05,
              "lora_rank": 32,
              "seed": 0
            }
          },
          {
            "name": "sdpo_token",
            "optimizer": "adam",
            "optimizer_config": {
              "beta1": 0.9,
              "beta2": 0.95,
              "eps": 1e-08,
              "learning_rate": 4e-05,
              "lora_rank": 32,
              "seed": 0
            }
          }
        ]
      }
    ],
    "reason": "Both trained arms start from the same base LoRA initialization. Identity is verified by hashing the initial adapter state (or a shared immutable init checkpoint plus an observed load acknowledgment) and by matching logprobs on a fixed tokenized batch before the first update. The optimizer config is checked after creation."
  },
  "spec_preflight": {
    "findings": [],
    "status": "passed"
  },
  "steps": 20,
  "success_criterion": "Validated if the 95% paired Hoeffding interval for (sdpo_token minus grpo) tau_bench_retail_pass1 over 115 test tasks has its lower bound above +0.03. Failed if the upper bound is below +0.03. Otherwise inconclusive; at this size the pilot is expected to settle only large effects. Per-dimension rates (held-out gpt-oss-120b judge) are reported descriptively, not as the verdict.",
  "title": "Token-level SDPO vs GRPO, Qwen3.5-4B LoRA, tau-bench retail, rubric-judge reward",
  "training_type": "rl"
}

correctness_validation

{'status': 'unverified', 'required': True, 'checks': [{'path': 'sdpo_token_advantage_into_importance_sampling', 'check': 'baseline_loss', 'status': 'unverified', 'tolerance': {'atol': 0.0001, 'rtol': 0.001}, 'measured': None}, {'path': 'sdpo_token_advantage_into_importance_sampling', 'check': 'baseline_gradient', 'status': 'unverified', 'tolerance': {'atol': 0.0001, 'rtol': 0.01, 'min_cosine': 0.999}, 'measured': None}, {'path': 'sdpo_token_advantage_into_importance_sampling', 'check': 'reference_loss', 'status': 'unverified', 'tolerance': {'atol': 0.001, 'rtol': 0.005}, 'measured': None}, {'path': 'sdpo_token_advantage_into_importance_sampling', 'check': 'reference_gradient', 'status': 'unverified', 'tolerance': {'atol': 0.001, 'rtol': 0.02, 'min_cosine': 0.995}, 'measured': None}], 'reason': 'sdpo_token_advantage_into_importance_sampling: fixed input description/hash; sdpo_token_advantage_into_importance_sampling: matching reference_revision; sdpo_token_advantage_into_importance_sampling: matching baseline_revision; sdpo_token_advantage_into_importance_sampling: implementation_revision; sdpo_token_advantage_into_importance_sampling: precision; sdpo_token_advantage_into_importance_sampling: normalization; sdpo_token_advantage_into_importance_sampling: measured baseline_loss errors; sdpo_token_advantage_into_importance_sampling: measured baseline_gradient errors; sdpo_token_advantage_into_importance_sampling: measured reference_loss errors; sdpo_token_advantage_into_importance_sampling: measured reference_gradient errors', 'evidence_source': 'executor report; remote execution is not attested', 'applicability': 'required', 'reported_verdict': 'inconclusive'}

decision_validation

{'status': 'uncertain', 'issues': ['Decision evidence was not recorded.'], 'criteria': [], 'outcome_reason': 'insufficient_evidence', 'unit': 'fixed_checkpoint_items', 'n_seeds': 1, 'alpha': 0.05, 'method': 'paired_hoeffding', 'applicability': {'check': 'decision_rule', 'status': 'required', 'enforce': True, 'reasons': ['experiment_features: paired_comparison is true'], 'features': {'paired_comparison': True, 'adaptive_selection': False}}}

ended_reason

message failed

feasibility

{'status': 'unknown', 'problems': ['Measured feasibility not recorded'], 'wall_seconds': None, 'gpu_hours': None, 'known_gpu_hours': 0, 'phases': {}, 'record': None, 'recorded_at': 1793810683.1954336}

feasibility_required

True

generation_audit_validation

{'status': 'not_applicable', 'reasons': ['spec.generation_audit opts out and no feature it covers is true'], 'applicability': {'check': 'generation_audit', 'status': 'not_applicable', 'enforce': False, 'reasons': ['spec.generation_audit opts out and no feature it covers is true'], 'features': {'generated_training_data': False}}}

launch

{'experiment_id': 'exp_2c137d4cb8', 'context_status': 'snapshotted', 'superseded': False, 'mode': 'ordinary', 'purpose': 'Execute the selected saved experiment version.'}

reported_outcome_reason

incomplete_execution

reported_verdict

inconclusive

restoration_validation

{'status': 'unverified', 'required': True, 'checks': [{'check': 'format', 'status': 'unverified', 'reason': 'Missing initialization_provenance version 1.'}, {'check': 'grpo', 'status': 'unverified', 'reason': 'grpo: missing or duplicate arm evidence.'}, {'check': 'sdpo_token', 'status': 'unverified', 'reason': 'sdpo_token: missing or duplicate arm evidence.'}], 'reason': 'Missing initialization_provenance version 1.', 'applicability': 'required'}

slack_started

True

started_at

1793810680.7929616