🔬Modal Research Exchange

{
  "activation_gate": {
    "applicable": true,
    "checks": [
      {
        "measures": "fraction of real assistant loss tokens with |teacher_logprob - student_logprob| > 0.1 nats",
        "metric": "sdpo_frac_tokens_abs_adv_gt_0.1",
        "min": 0.05
      },
      {
        "max": 3.0,
        "measures": "mean |teacher - student| logprob over assistant tokens, in nats",
        "metric": "sdpo_mean_abs_token_adv",
        "min": 0.01
      },
      {
        "measures": "fraction of trajectories whose teacher prompt has nonempty judge rationale",
        "metric": "sdpo_frac_trajectories_with_feedback",
        "min": 0.9
      },
      {
        "measures": "fraction of rollout groups with nonzero composite variance",
        "metric": "grpo_frac_groups_nonzero_adv",
        "min": 0.25
      },
      {
        "measures": "fraction of judge calls returning 3 parsable scores plus rationale",
        "metric": "judge_parse_success_rate",
        "min": 0.95
      },
      {
        "measures": "model name reported by the created training client",
        "metric": "executed_base_model",
        "one_of": [
          "Qwen/Qwen3.5-4B"
        ]
      }
    ],
    "reason": "SDPO acts only if the feedback-conditioned teacher assigns materially different logprobs to the sampled assistant tokens. GRPO acts only if groups have nonzero reward variance. Both are measured in a 2-step pilot, which also confirms the executed config."
  },
  "arms": [
    {
      "advantage": "group-mean-centered composite, no std norm",
      "lr": 4e-05,
      "name": "grpo",
      "seed": 0
    },
    {
      "advantage": "teacher_logprob - student_logprob under feedback-conditioned prompt",
      "lr": 4e-05,
      "name": "sdpo_token",
      "seed": 0
    }
  ],
  "base_model": "Qwen/Qwen3.5-4B",
  "baseline": "GRPO-style RL on the scalar composite rubric score, using a group-mean baseline and the stock importance_sampling loss. Same model, LoRA, lr, tasks, task order, rollout budget, simulator, judge and eval as SDPO.",
  "batch_size": 16,
  "budget": {
    "gpu_hours": 2.0,
    "max_wall_minutes": 120
  },
  "command": "Custom python script against the tinker SDK. Custom parts: the multi-turn tau-bench env wrapper, judge calls, and SDPO advantage computation. Setup: svc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL']). tc = svc.create_lora_training_client(base_model='Qwen/Qwen3.5-4B', rank=32). user_sim = svc.create_sampling_client(base_model='Qwen/Qwen3-8B'). judge = svc.create_sampling_client(base_model='openai/gpt-oss-120b'). Data: train_tasks = tau-bench retail train tasks, shuffled with seed 0, first 80; the identical list and order is used for both arms. Training loop, for step in range(20): (1) sc = tc.save_weights_and_get_sampling_client(f'{arm}-{step}'). (2) For 4 tasks x 4 rollouts, run up to 12 agent turns with sc, executing tool calls in the tau-bench retail env and drawing user turns from user_sim. Record assistant token ids and sampler logprobs, with mask=1 on assistant tokens only. (3) The judge returns JSON {invalid_tool_use, ungrounded_claims, upfront_clarification, rationale}, and composite = mean. (4) Advantages: if arm == 'grpo', adv = composite - group_mean, broadcast over masked tokens. If arm == 'sdpo_token', build the teacher context (feedback block + best sibling if composite >= 0.8 and not self, never the self attempt). Then t_lp = sc.compute_logprobs(teacher_context + original trajectory tokens), aligned on assistant tokens, and adv = clip(t_lp - sampler_lp, -5, 5) * mask. (5) Training step: tc.forward_backward(datums with target_tokens, logprobs=sampler_lp, advantages=adv, loss_fn='importance_sampling'), then tc.optim_step(tinker.AdamParams(learning_rate=4e-5)). Logging: per-step activation metrics and the judge parse rate. Eval: on test tasks 0-59, one greedy rollout per arm checkpoint, scored by the same judge and native tau-bench reward. Per-item results are saved for paired Hoeffding; per-dimension means are reported as secondary. Activation-failure plan: first run offline forced checks. Feed a teacher context with the correct sibling inserted versus an empty feedback block, confirm advantage magnitude changes, and check that gradients are nonzero only on masked assistant tokens. Then run a 2-step pilot per arm; if a gate fails, preserve the pilot logs. Only source-justified fallbacks are allowed, chosen by a fixed rule (first that passes) and recorded as a spec deviation before any full run: first add a frozen-initial-teacher context, then raise max turns to 16. Calibration is capped at 4 steps and 15 wall minutes, uses train tasks only, and never touches the eval tasks. Gates are never relaxed. Full runs only after correctness_preflight is resolved and feasibility is published from a timing pilot.",
  "correctness_preflight": {
    "paths": [
      {
        "baseline_revision": "tinker-cookbook commit hash pinned at run time (stock rl loop importance_sampling + group-mean advantages)",
        "id": "grpo_via_new_advantage_path",
        "kind": "loss",
        "reference_revision": "same pinned tinker-cookbook commit",
        "tolerances": {
          "baseline_gradient": {
            "atol": 1e-05,
            "min_cosine": 0.9999,
            "rtol": 0.0001
          },
          "baseline_loss": {
            "atol": 1e-06,
            "rtol": 1e-05
          },
          "reference_gradient": {
            "atol": 1e-05,
            "min_cosine": 0.9999,
            "rtol": 0.0001
          },
          "reference_loss": {
            "atol": 1e-06,
            "rtol": 1e-05
          }
        }
      },
      {
        "baseline_revision": "tinker-cookbook commit hash pinned at run time, with SDPO disabled reproducing stock GRPO advantages",
        "id": "sdpo_token_advantage",
        "kind": "loss",
        "reference_revision": "float64 reimplementation of token-level SDPO advantage (arXiv 2601.20802v1 eq. for token-level KL gradient), committed hash pinned at run time; official SDPO repo commit if available",
        "tolerances": {
          "baseline_gradient": {
            "atol": 1e-05,
            "min_cosine": 0.9999,
            "rtol": 0.0001
          },
          "baseline_loss": {
            "atol": 1e-06,
            "rtol": 1e-05
          },
          "reference_gradient": {
            "atol": 0.0001,
            "min_cosine": 0.999,
            "rtol": 0.001
          },
          "reference_loss": {
            "atol": 0.0001,
            "rtol": 0.001
          }
        }
      }
    ],
    "required": true,
    "status": "unresolved until revisions are pinned; no full run before resolution"
  },
  "dataset": {
    "hf_path": null,
    "n_examples": 80,
    "name": "tau-bench retail (sierra-research/tau-bench GitHub, pinned commit; train tasks for training, test tasks for eval)",
    "split": "train (training) / test (eval, first 60 tasks by id)"
  },
  "decision_rule": {
    "alpha": 0.05,
    "criteria": [
      {
        "bounds": [
          0,
          1
        ],
        "id": "primary",
        "margin": 0.03,
        "metric": "composite_rubric_score",
        "weights": {
          "baseline": -1,
          "treatment": 1
        }
      },
      {
        "bounds": [
          0,
          1
        ],
        "id": "reward_guard",
        "margin": -0.1,
        "metric": "tau_bench_task_reward",
        "weights": {
          "baseline": -1,
          "treatment": 1
        }
      }
    ],
    "estimand": "On held-out tau-bench retail tasks, conditional on the trained checkpoints: the difference in mean composite rubric score between token-level SDPO and GRPO (larger is better), with a noninferiority guard on task reward.",
    "method": "paired_hoeffding",
    "seeds": [
      0
    ],
    "unit": "fixed_checkpoint_items",
    "version": 1
  },
  "eval": {
    "aggregation": null,
    "benchmark": "tau-bench retail test tasks, 60 tasks, 1 greedy rollout each, frozen Qwen3-8B simulator, frozen gpt-oss-120b judge",
    "metric": "mean composite rubric score (0-1); guard: mean tau-bench task reward (0/1 DB-state match)",
    "n_examples": 60
  },
  "experiment_features": {
    "adaptive_selection": false,
    "checkpoint_restore": false,
    "custom_loss": true,
    "custom_optimizer": false,
    "generated_training_data": false,
    "heldout_loss": false,
    "learned_generator": false,
    "paired_comparison": true,
    "paired_training_arms": true,
    "warmup_adaptation": false
  },
  "feasibility": null,
  "feasibility_check": {
    "gpu_hours": null,
    "known_gpu_hours": 0,
    "phases": {},
    "problems": [
      "Measured feasibility not recorded"
    ],
    "status": "unknown",
    "wall_seconds": null
  },
  "feasibility_required": true,
  "generation_audit": {
    "answer_comparison": "categorical_exact",
    "max_eval_overlaps": 0,
    "max_label_errors": 0,
    "min_sample_size": 100,
    "required": false
  },
  "hypothesis": "With a fixed frozen judge, token-level SDPO beats GRPO by at least 0.03 on mean composite rubric score (0-1) over 60 held-out tau-bench retail tasks at the same rollout budget, without losing more than 0.10 tau-bench task reward.",
  "lora_rank": 32,
  "max_seq_len": 8192,
  "method": "This extends arXiv 2601.20802, which never tested multi-turn agents or judge rewards; it is not a replication. Both arms train a rank-32 LoRA on Qwen/Qwen3.5-4B, non-thinking, from the same base. Training uses tau-bench retail train tasks; held-out tasks come from the retail test split. Source is sierra-research/tau-bench on GitHub, pinned to a commit hash at run time. The 4B model is a scale-down forced by the 2 GPU-hour budget. The paper reports SDPO gains shrink at small scale, so a null result here is weak evidence against SDPO at 8B+. The user simulator is a frozen served Qwen/Qwen3-8B, substituting for tau-bench's default GPT-4o; it only samples and is never trained. The rubric judge is a frozen served openai/gpt-oss-120b. For each trajectory it returns three 0-1 scores and a short rationale per dimension. The dimensions are invalid_tool_use (inverted, so higher is better), ungrounded_claims (inverted) and upfront_clarification. Composite reward = mean of the three. Rollouts: each step takes 4 tasks x 4 rollouts, 16 trajectories, for 20 steps. Max 12 agent turns per episode, 512 tokens per turn, temperature 1.0. Each arm takes one optim_step per batch, so the policy is on-policy at sampling time. Baseline arm (GRPO): per-token advantage = composite minus the group mean, without std normalization (Dr.GRPO-style, as the paper's improved GRPO). It is broadcast over assistant tokens and passed to the stock importance_sampling loss. Treatment arm (token-level SDPO, the paper's middle variant; logit-level top-K is out of scope): the teacher prompt is the original system prompt, prepended with three things. First, a 'Correct solution:' block with the highest-composite sibling trajectory from the same group, if its composite >= 0.8 and it is not the trajectory itself. Second, 'The following is feedback from your unsuccessful earlier attempt:' plus the judge rationales. Third, 'Correctly solve the original question.' The student's own attempt is never included in the teacher prompt, following paper Table 6. The original multi-turn trajectory is then rescored with sampling_client.compute_logprobs under this context. Teacher = current policy at sampling time, stop-grad. There is no EMA: 20 steps is short, and this deviation is recorded. Per-token advantage = teacher_logprob - student_logprob on each sampled assistant token. It is clipped to [-5,5], with user, tool and system tokens masked to 0, and fed to the same importance_sampling loss. The two arms differ only in the advantage computation. Shared settings: AdamW (stock tinker.AdamParams) at lr 4e-5, renderer, tasks, task order, simulator, judge and eval. Per-dimension judge scores are logged as secondary, descriptive answers to the idea's question.",
  "optimizer": {
    "lr": 4e-05,
    "name": "adam"
  },
  "preflight": {
    "probe": "Create both arms' training clients (Qwen/Qwen3.5-4B, rank 32, adam via tinker.AdamParams lr 4e-5). Run one retail train episode and one judge call, compute SDPO teacher logprobs with compute_logprobs, and take one forward_backward plus optim_step per arm on that single trajectory. Time each phase and confirm the same fixed-batch logprobs from both fresh clients before the update.",
    "riskiest_assumption": "Three conditions must hold in 2 GPU-hours. A 4B policy must complete multi-turn retail episodes with parsable tool calls. The frozen judge's rationales must give non-trivial teacher-student logprob gaps. Episodes, simulator and judge calls must fit 2 arms x 20 steps x 16 trajectories plus 2 x 60 eval episodes.",
    "tuning": "Both arms use the same AdamW optimizer, so there is no per-arm LR grid. Both use a fixed lr of 4e-5, a standard LoRA RL learning rate, set a priori. No hyperparameter is selected from results. If the pilot shows divergence (mean |adv| above the gate or NaN), stop and record a deviation rather than tuning on eval tasks."
  },
  "primary_endpoint": "heldout_composite_rubric_score_mean",
  "restoration_preflight": {
    "applicable": true,
    "fixed_batch_required": true,
    "groups": [
      {
        "arms": [
          {
            "name": "grpo",
            "optimizer": "adam",
            "optimizer_config": {
              "beta1": 0.9,
              "beta2": 0.95,
              "eps": 1e-08,
              "learning_rate": 4e-05,
              "lora_rank": 32
            }
          },
          {
            "name": "sdpo_token",
            "optimizer": "adam",
            "optimizer_config": {
              "beta1": 0.9,
              "beta2": 0.95,
              "eps": 1e-08,
              "learning_rate": 4e-05,
              "lora_rank": 32
            }
          }
        ]
      }
    ],
    "reason": "Both arms train from the same base LoRA initialization. Initial-state identity is checked by comparing per-token logprobs on a fixed tokenized trajectory batch for both fresh clients before any update. Optimizer config is read back after creation. Server identity is unverified beyond this."
  },
  "spec_preflight": {
    "findings": [],
    "status": "passed"
  },
  "steps": 20,
  "success_criterion": "Validated if the 95% simultaneous paired Hoeffding interval for SDPO minus GRPO mean composite rubric score is entirely above 0.03 on 60 held-out tasks. The tau-bench reward contrast interval must also lie entirely above -0.10. Failed if the composite interval lies entirely below 0.03. Otherwise inconclusive, which is likely at n=60; that outcome motivates a larger run.",
  "title": "Token-level SDPO vs GRPO on tau-bench retail with a frozen rubric judge, Qwen3.5-4B LoRA",
  "training_type": "rl"
}

[
  {
    "arm": "grpo",
    "step": 14,
    "uri": "tinker://a14956b6cd9d6e01ce4bdf0b832d6fd8:train:0/sampler_weights/grpo-s14"
  },
  {
    "arm": "sdpo_token",
    "step": 14,
    "uri": "tinker://4bdcbef25e675e7b6db12b015d1a2dcd:train:0/sampler_weights/sdpo_token-s14"
  }
]

{
  "base": {
    "composite_rubric_score": 0.9583333333333334,
    "tau_bench_task_reward": 0.05
  },
  "grpo": {
    "composite_rubric_score": 0.96,
    "tau_bench_task_reward": 0.05
  },
  "sdpo_token": {
    "composite_rubric_score": 0.99,
    "tau_bench_task_reward": 0.08333333333333333
  }
}

base_composite_rubric_score_mean

0.9583

base_tau_bench_task_reward_mean

0.05

base_invalid_tool_use_mean

1

base_ungrounded_claims_mean

0.9467

base_upfront_clarification_mean

0.9283

base_judge_parse_rate

1

composite_effect_sdpo_minus_grpo

0.03

composite_ci_lower

-0.3522

composite_ci_upper

0.4122

reward_effect_sdpo_minus_grpo

0.03333

reward_ci_lower

-0.3489

reward_ci_upper

0.4155

hoeffding_halfwidth

0.3822

paired_steps_completed

14

• checkpoint · grpo · step 14 · tinker://a14956b6cd9d6e01ce4bdf0b832d6fd8:train:0/sampler_weights/grpo-s14
• checkpoint · sdpo_token · step 14 · tinker://4bdcbef25e675e7b6db12b015d1a2dcd:train:0/sampler_weights/sdpo_token-s14
• effective_config · effective-config.json
• code_revision · code-revision.txt

1. Nov 4, 17:21 UTC · succeeded · done: grpo, sdpo_token · Both arms (GRPO and token-level SDPO, Qwen3.5-4B rank-32 LoRA, lr 4e-5, 16 trajectories per step) trained for 14 of the 20 planned paired steps on tau-bench retail train tasks before a wall-clock cut…
2. Nov 4, 16:49 UTC · running · Base eval done (composite 0.9583, reward 0.05). Both arms' training clients created with identical step-0 outputs on a fixed batch; 2-step activation pilot running. Spindle cold-starts a new sampler …
3. Nov 4, 16:17 UTC · running · Base Qwen3.5-4B held-out eval done (60 retail test tasks, greedy, frozen Qwen3-8B sim + gpt-oss-120b judge): composite 0.9583, task reward 0.05. Correctness preflight for both custom advantage paths …
4. Nov 4, 15:50 UTC · running · Setting up tau-bench retail env wrapper (pinned 59a200c), frozen Qwen3-8B user sim and gpt-oss-120b judge on shared Spindle; running preflight probe and correctness preflight before any arm trains.

correctness_preflight

{'required': True, 'paths': [{'id': 'grpo_via_new_advantage_path', 'fixed_input': {'sha256': 'ebc6ba087807acce2c5444cb650e8933ac0e36d1128706e10cfa969396a7c181', 'description': 'out/preflight/fixed_input.json: 4 rollouts of retail train task 0 from a fresh rank-32 LoRA (step 0) with judge composites, assistant masks and sampler logprobs; current logprobs held fixed (ratio=1 and seeded N(0,0.05) perturbation); no optimizer state'}, 'baseline_revision': 'tinker-cookbook f79fdb2ba4cd967ef2807ba43097c52985f9f17f rl/data_processing.compute_advantages + stock importance_sampling', 'reference_revision': 'tinker-cookbook f79fdb2ba4cd967ef2807ba43097c52985f9f17f', 'implementation_revision': 'sha256(tb.py+adv.py)=a685fa87ea08db6029a4756c8b6985c2059d2e4ddbac115ed224f9dfb6b486cb', 'precision': 'advantages torch float32 (as stock cookbook) -> float32 TensorData; loss/grad float64', 'normalization': 'importance_sampling sum over tokens; advantage broadcast over assistant (mask=1) tokens; no std norm', 'errors': {'baseline_loss': {'max_abs_error': 0.0, 'reference_max_abs': 1.4126953860667513, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}, 'baseline_gradient': {'max_abs_error': 0.0, 'reference_max_abs': 0.04804028897242256, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}, 'reference_loss': {'max_abs_error': 0.0, 'reference_max_abs': 1.4126953860667513, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}, 'reference_gradient': {'max_abs_error': 0.0, 'reference_max_abs': 0.04804028897242256, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}}, 'lr_scaling': []}, {'id': 'sdpo_token_advantage', 'fixed_input': {'sha256': 'ebc6ba087807acce2c5444cb650e8933ac0e36d1128706e10cfa969396a7c181', 'description': 'same fixed input; teacher logprobs from one compute_logprobs call per trajectory; current logprobs from one shared server forward'}, 'baseline_revision': 'tinker-cookbook f79fdb2 with SDPO disabled reproducing stock GRPO advantages', 'reference_revision': 'sdpo_ref.py float64 reimplementation sha256=77a5c4ae1fc031a717086659919baa69194499f8a9e25135e8988bb12191a9c1 (no official SDPO repo used)', 'implementation_revision': 'sha256(tb.py+adv.py)=a685fa87ea08db6029a4756c8b6985c2059d2e4ddbac115ed224f9dfb6b486cb', 'precision': 'teacher/current logprobs float32 from Spindle; impl float64->float32 datum; reference pure-python float64', 'normalization': 'importance_sampling sum over tokens; per-token adv clipped to [-5,5]; mask = sampled assistant tokens only', 'errors': {'baseline_loss': {'max_abs_error': 0.0, 'reference_max_abs': 1.4126953860667513, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}, 'baseline_gradient': {'max_abs_error': 0.0, 'reference_max_abs': 0.04804028897242256, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}, 'reference_loss': {'max_abs_error': 1.2314609421082423e-06, 'reference_max_abs': 401.35964074273113, 'relative_l2_error': 3.068223152256608e-09, 'cosine_similarity': 1.0}, 'reference_gradient': {'max_abs_error': 2.3824965733609815e-07, 'reference_max_abs': 5.925059615518825, 'relative_l2_error': 2.0490967212979427e-08, 'cosine_similarity': 0.9999999999999998}}, 'lr_scaling': []}]}

correctness_validation

{'status': 'unverified', 'required': True, 'checks': [{'path': 'grpo_via_new_advantage_path', 'check': 'baseline_loss', 'status': 'passed', 'tolerance': {'atol': 1e-06, 'rtol': 1e-05}, 'measured': {'max_abs_error': 0.0, 'reference_max_abs': 1.4126953860667513, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}}, {'path': 'grpo_via_new_advantage_path', 'check': 'baseline_gradient', 'status': 'passed', 'tolerance': {'atol': 1e-05, 'rtol': 0.0001, 'min_cosine': 0.9999}, 'measured': {'max_abs_error': 0.0, 'reference_max_abs': 0.04804028897242256, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}}, {'path': 'grpo_via_new_advantage_path', 'check': 'reference_loss', 'status': 'passed', 'tolerance': {'atol': 1e-06, 'rtol': 1e-05}, 'measured': {'max_abs_error': 0.0, 'reference_max_abs': 1.4126953860667513, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}}, {'path': 'grpo_via_new_advantage_path', 'check': 'reference_gradient', 'status': 'passed', 'tolerance': {'atol': 1e-05, 'rtol': 0.0001, 'min_cosine': 0.9999}, 'measured': {'max_abs_error': 0.0, 'reference_max_abs': 0.04804028897242256, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}}, {'path': 'sdpo_token_advantage', 'check': 'baseline_loss', 'status': 'passed', 'tolerance': {'atol': 1e-06, 'rtol': 1e-05}, 'measured': {'max_abs_error': 0.0, 'reference_max_abs': 1.4126953860667513, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}}, {'path': 'sdpo_token_advantage', 'check': 'baseline_gradient', 'status': 'passed', 'tolerance': {'atol': 1e-05, 'rtol': 0.0001, 'min_cosine': 0.9999}, 'measured': {'max_abs_error': 0.0, 'reference_max_abs': 0.04804028897242256, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}}, {'path': 'sdpo_token_advantage', 'check': 'reference_loss', 'status': 'passed', 'tolerance': {'atol': 0.0001, 'rtol': 0.001}, 'measured': {'max_abs_error': 1.2314609421082423e-06, 'reference_max_abs': 401.35964074273113, 'relative_l2_error': 3.068223152256608e-09, 'cosine_similarity': 1.0}}, {'path': 'sdpo_token_advantage', 'check': 'reference_gradient', 'status': 'passed', 'tolerance': {'atol': 0.0001, 'rtol': 0.001, 'min_cosine': 0.999}, 'measured': {'max_abs_error': 2.3824965733609815e-07, 'reference_max_abs': 5.925059615518825, 'relative_l2_error': 2.0490967212979427e-08, 'cosine_similarity': 0.9999999999999998}}], 'reason': 'grpo_via_new_advantage_path: matching reference_revision; grpo_via_new_advantage_path: matching baseline_revision; sdpo_token_advantage: matching reference_revision; sdpo_token_advantage: matching baseline_revision', 'evidence_source': 'executor report; remote execution is not attested', 'applicability': 'required', 'reported_verdict': 'refuted'}

decision

{'version': 1, 'observations_file': {'name': 'scored-observations.jsonl', 'sha256': '2a16c724a429e73f1ea098f834729ec29f219fc8754f1cb8f61f4c29f5f228ef'}}

decision_validation

{'status': 'unverified', 'issues': ['Observation file name/hash is not retained for this run.'], 'criteria': [], 'outcome_reason': 'insufficient_evidence', 'applicability': {'check': 'decision_rule', 'status': 'required', 'enforce': True, 'reasons': ['experiment_features: paired_comparison is true'], 'features': {'paired_comparison': True, 'adaptive_selection': False}}, 'reported_verdict': 'refuted'}

evaluation

{'version': 1, 'evaluators': {'3d43a6a944b0fe703a8f9ea440156cbe734a1843a322d999057e9299996a32d0': {'code_sha256': "see build_report.py: sha256(tb.py+evallib.py+'tau-bench@59a200c;tinker==0.37.0'); tb.py fb320dee..., evallib.py 22b03f73... (frozen before base eval)", 'prompt_template': 'Policy Qwen3.5-4B HF chat template, tau-bench retail wiki system + 16 tools, enable_thinking=False; user sim Qwen3-8B with tau-bench LLMUserSimulationEnv prompt; judge gpt-oss-120b harmony reasoning_effort=low with rubric JUDGE_SYS in tb.py', 'decoding': {'temperature': 0, 'max_tokens': 512, 'stop': ['<|im_end|>']}, 'parser': 'tb.parse_judge + tb.task_reward', 'items': {'ids': ['retail_test_0 .. retail_test_59 (tau-bench retail TASKS_TEST indices 0-59)']}}}, 'scores': [{'arm': 'base', 'metric': 'composite_rubric_score', 'value': 0.9583333333333334, 'evaluator_id': '3d43a6a944b0fe703a8f9ea440156cbe734a1843a322d999057e9299996a32d0', 'seed': 0, 'predictions_ref': 'predictions-base.jsonl'}, {'arm': 'grpo', 'metric': 'composite_rubric_score', 'value': 0.96, 'evaluator_id': '3d43a6a944b0fe703a8f9ea440156cbe734a1843a322d999057e9299996a32d0', 'seed': 0, 'predictions_ref': 'predictions-grpo.jsonl'}, {'arm': 'sdpo_token', 'metric': 'composite_rubric_score', 'value': 0.99, 'evaluator_id': '3d43a6a944b0fe703a8f9ea440156cbe734a1843a322d999057e9299996a32d0', 'seed': 0, 'predictions_ref': 'predictions-sdpo_token.jsonl'}, {'arm': 'base', 'metric': 'tau_bench_task_reward', 'value': 0.05, 'evaluator_id': '3d43a6a944b0fe703a8f9ea440156cbe734a1843a322d999057e9299996a32d0', 'seed': 0, 'predictions_ref': 'predictions-base.jsonl'}, {'arm': 'grpo', 'metric': 'tau_bench_task_reward', 'value': 0.05, 'evaluator_id': '3d43a6a944b0fe703a8f9ea440156cbe734a1843a322d999057e9299996a32d0', 'seed': 0, 'predictions_ref': 'predictions-grpo.jsonl'}, {'arm': 'sdpo_token', 'metric': 'tau_bench_task_reward', 'value': 0.08333333333333333, 'evaluator_id': '3d43a6a944b0fe703a8f9ea440156cbe734a1843a322d999057e9299996a32d0', 'seed': 0, 'predictions_ref': 'predictions-sdpo_token.jsonl'}], 'comparisons': [{'metric': 'composite_rubric_score', 'arms': ['grpo', 'sdpo_token']}, {'metric': 'tau_bench_task_reward', 'arms': ['grpo', 'sdpo_token']}]}

evaluator_validation

{'status': 'invalid', 'issues': ['Evaluator 3d43a6a944b0fe703a8f9ea440156cbe734a1843a322d999057e9299996a32d0: code_sha256 is required.', 'Evaluator 3d43a6a944b0fe703a8f9ea440156cbe734a1843a322d999057e9299996a32d0: identity does not match provenance hash.'], 'comparisons': [{'metric': 'composite_rubric_score', 'status': 'comparable', 'arms': ['base', 'grpo', 'sdpo_token'], 'evaluator_ids': ['3d43a6a944b0fe703a8f9ea440156cbe734a1843a322d999057e9299996a32d0'], 'missing_arms': []}, {'metric': 'tau_bench_task_reward', 'status': 'comparable', 'arms': ['base', 'grpo', 'sdpo_token'], 'evaluator_ids': ['3d43a6a944b0fe703a8f9ea440156cbe734a1843a322d999057e9299996a32d0'], 'missing_arms': []}]}

feasibility

{'status': 'unknown', 'problems': ['Re-projection required: lora_rank changed or was not recorded', 'Re-projection required: dataset changed or was not recorded', 'Re-projection required: steps changed or was not recorded', 'Re-projection required: n_examples changed or was not recorded'], 'wall_seconds': None, 'gpu_hours': None, 'known_gpu_hours': 0, 'phases': {'setup': {'wall_seconds': 20.8, 'planned_units': 1, 'planned_samples': 0.0, 'gpu_hours': None}, 'training': {'wall_seconds': 3127.1428571428573, 'planned_units': 40, 'planned_samples': 640.0, 'gpu_hours': None}, 'evaluation': {'wall_seconds': 1188.8, 'planned_units': 180, 'planned_samples': 180.0, 'gpu_hours': None}, 'recovery': {'wall_seconds': 600, 'gpu_hours': None}}, 'generation_counts': None, 'generation_applicability': {'check': 'generation_audit', 'status': 'not_applicable', 'enforce': False, 'reasons': ['spec.generation_audit opts out and no feature it covers is true'], 'features': {'generated_training_data': False}}, 'record': {'config': {'base_model': 'Qwen/Qwen3.5-4B', 'arms': ['grpo', 'sdpo_token'], 'steps': 20, 'batch_size': 16, 'max_seq_len': 8192, 'max_tokens': 512, 'n_eval': 60}, 'effective_config': {'base_model': 'Qwen/Qwen3.5-4B', 'arms': ['grpo', 'sdpo_token'], 'steps': 14, 'batch_size': 16, 'max_seq_len': 8192, 'max_tokens': 512, 'n_eval': 60}, 'primary_endpoint': 'heldout_composite_rubric_score_mean', 'source': 'out/preflight/preflight.json, out/*/steps.jsonl, base_eval.log, eval_wait.log', 'measurements': {'setup': {'seconds': 20.8, 'units': 1, 'planned_units': 1, 'samples': 0, 'gpu_count': None, 'allocation_source': None}, 'training': {'seconds': 2189, 'units': 28, 'planned_units': 40, 'samples': 448, 'gpu_count': None, 'allocation_source': None}, 'evaluation': {'seconds': 1188.8, 'units': 180, 'planned_units': 180, 'samples': 180, 'gpu_count': None, 'allocation_source': None}}, 'generation': None, 'recovery': {'seconds': 600, 'gpu_count': None, 'allocation_source': None}}, 'recorded_at': 1793812863.610728}

feasibility_reported_verdict

refuted

feasibility_required

True

generation_audit_validation

{'status': 'not_applicable', 'reasons': ['spec.generation_audit opts out and no feature it covers is true'], 'applicability': {'check': 'generation_audit', 'status': 'not_applicable', 'enforce': False, 'reasons': ['spec.generation_audit opts out and no feature it covers is true'], 'features': {'generated_training_data': False}}}

initialization_provenance

{'version': 1, 'arms': {'grpo': {'initialization': {'base_checkpoint': 'Qwen/Qwen3.5-4B (fresh rank-32 LoRA, server seed unsupported)', 'adapter_sha256': None, 'checkpoint_uri': 'tinker://a14956b6cd9d6e01ce4bdf0b832d6fd8:train:0/sampler_weights/grpo-init', 'source': 'server_response', 'evidence_ref': 'out/grpo/init.json', 'before_first_update': True, 'note': 'save_weights_external returned 404 on Spindle, so no adapter hash; per-arm init checkpoints are separate runs (no shared identity)'}, 'restore_count': 0, 'optimizer_checks': [{'after_restore': 0, 'before_next_update': True, 'requested': {'optimizer': 'adam', 'learning_rate': 4e-05, 'beta1': 0.9, 'beta2': 0.95, 'eps': 1e-08}, 'actual_family': 'adamw', 'actual_config': {'type': 'adamw'}, 'source': 'server_response (get_info.optimizer_config); lr/betas/eps sent per optim_step via AdamParams', 'evidence_ref': 'out/grpo/init.json'}], 'initial_output': {'available': True, 'batch_sha256': '5055e3c79bed9f3d31abe81504807700fd9a5e0587edf2ac6bc15634f5761efa', 'output_sha256': 'afe7187ea6fdb94222d1d34dab3d7d43c662d55d6caff3a459360948e391633e', 'evidence_ref': 'out/grpo/init.json', 'before_first_update': True}}, 'sdpo_token': {'initialization': {'base_checkpoint': 'Qwen/Qwen3.5-4B (fresh rank-32 LoRA, server seed unsupported)', 'adapter_sha256': None, 'checkpoint_uri': 'tinker://4bdcbef25e675e7b6db12b015d1a2dcd:train:0/sampler_weights/sdpo_token-init', 'source': 'server_response', 'evidence_ref': 'out/sdpo_token/init.json', 'before_first_update': True}, 'restore_count': 0, 'optimizer_checks': [{'after_restore': 0, 'before_next_update': True, 'requested': {'optimizer': 'adam', 'learning_rate': 4e-05, 'beta1': 0.9, 'beta2': 0.95, 'eps': 1e-08}, 'actual_family': 'adamw', 'actual_config': {'type': 'adamw'}, 'source': 'server_response (get_info.optimizer_config)', 'evidence_ref': 'out/sdpo_token/init.json'}], 'initial_output': {'available': True, 'batch_sha256': '5055e3c79bed9f3d31abe81504807700fd9a5e0587edf2ac6bc15634f5761efa', 'output_sha256': 'afe7187ea6fdb94222d1d34dab3d7d43c662d55d6caff3a459360948e391633e', 'evidence_ref': 'out/sdpo_token/init.json', 'before_first_update': True}}}, 'matched': 'initial_output sha256 identical across arms (4460-token fixed batch); adapter identity unverified (LoRA B=0 makes outputs match regardless of A)'}

launch

{'experiment_id': 'exp_4281eff0ca', 'context_status': 'snapshotted', 'superseded': False, 'mode': 'ordinary', 'purpose': 'Execute the selected saved experiment version.'}

progress_updated_at

1793812800.7632895

reported_verdict

refuted

restoration_validation

{'status': 'unverified', 'required': True, 'checks': [{'check': 'grpo', 'status': 'unverified', 'reason': 'grpo: missing or duplicate arm evidence.'}, {'check': 'sdpo_token', 'status': 'unverified', 'reason': 'sdpo_token: missing or duplicate arm evidence.'}], 'reason': 'grpo: missing or duplicate arm evidence.', 'applicability': 'required'}

slack_started

True

started_at

1793806564.083942

verdict_report

{'effect': 0.03, 'margin': 0.03, 'lower': -0.3521878523568343, 'upper': 0.41218785235683425, 'seeds': 1, 'confidence': 'weak', 'reasoning': 'SDPO beat GRPO by +0.030 on held-out composite rubric score (0.990 vs 0.960), which only ties the 0.03 margin and does not exceed it. The 95% simultaneous paired Hoeffding interval [-0.35, 0.41] is far too wide at n=60 to settle either the primary criterion or the reward guard (+0.033, CI [-0.35, 0.42]). Confidence is limited by: one seed, training cut to 14 of 20 steps, a judge near its ceiling (the base model already scores 0.958), and matched initialization verified only through identical initial outputs, not adapter hashes.'}