🔬Modal Research Exchange

{
  "base_model": "Qwen/Qwen3.5-0.8B",
  "baseline": "Same script, model, seeds, data order, group size 8, 32 groups per batch, max_tokens 512, LoRA rank 32, AdamW lr 1e-4, and 100 steps, with CENTER=0 (standard group-mean-centered GRPO surrogate, no score centering).",
  "batch_size": 256,
  "budget": {
    "gpu_hours": 2.0,
    "max_wall_minutes": 120
  },
  "command": "CENTER=0 SEED=0 python rl_score_centering.py && CENTER=1 SEED=0 python rl_score_centering.py (repeat SEED=1)  # rl_score_centering.py: sc=tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL']); tc=sc.create_lora_training_client(base_model='Qwen/Qwen3.5-0.8B', rank=32); loop 100 steps: samp=tc.save_weights_and_get_sampling_client(name=f'step{i}'); sample 32 GSM8K prompts x8 (max_tokens=512); reward=exact match; adv=r-mean(group); tc.forward_backward_custom(data, loss_fn) where loss_fn is the IS policy-gradient loss plus the score-centering term when CENTER=1 (custom part); tc.optim_step(tinker.AdamParams(learning_rate=1e-4)); every 10 steps eval 500 GSM8K test greedy; log grad norm, logprob gap, entropy to /tmp/rx-sc-$CENTER-$SEED",
  "dataset": {
    "hf_path": "openai/gsm8k",
    "n_examples": 3200,
    "name": "GSM8K",
    "split": "main/train (train), main/test first 500 (eval)"
  },
  "eval": {
    "benchmark": "GSM8K test (first 500)",
    "metric": "greedy exact-match accuracy; peak-minus-final drop over the run",
    "n_examples": 500
  },
  "hypothesis": "With LoRA GRPO on Qwen3.5-0.8B at a deliberately high learning rate, score centering limits the drop from peak to final GSM8K test accuracy to at most 2 points, while the uncentered baseline drops by at least 5 points.",
  "lora_rank": 32,
  "max_seq_len": 1024,
  "method": "Both arms use one tinker SDK GRPO script that differs only by a CENTER flag. Each step samples 32 GSM8K train problems with 8 rollouts each from the current LoRA and scores exact-match reward. Advantages are reward minus the group mean, and the importance-sampling policy-gradient loss goes through forward_backward_custom. With CENTER=1, the loss applies the score-centering correction exactly as written in Martin's paper. That correction is the only custom part, and it must be copied from the paper's equations before launch. With CENTER=0 the loss is the stock GRPO surrogate. Every 10 steps the script saves sampler weights and evaluates greedy exact match on 500 fixed GSM8K test problems. Every step it logs mean reward, gradient norm, sampler-vs-learner logprob gap and token entropy as stability signals. Each arm runs 2 seeds at learning rate 1e-4, chosen so the baseline is likely to destabilize.",
  "optimizer": {
    "lr": 0.0001,
    "name": "adam"
  },
  "steps": 100,
  "success_criterion": "Validated if, averaged over 2 seeds, the treatment's peak-minus-final accuracy drop is \u22642 points and is \u22653 points smaller than the baseline's, and final accuracy is not more than 1.5 points below baseline. Failed if the drops differ by <3 points, which is within seed noise on 500 examples.",
  "title": "Score centering vs plain GRPO on GSM8K at an aggressive learning rate",
  "training_type": "rl"
}

[
  {
    "arm": "pilot_sc_only_s0",
    "step": 2,
    "uri": "tinker://25e01ce3a6adb0fc2a45e3a9eb8bfa74:train:0/sampler_weights/s0c1final-ckpt2"
  },
  {
    "arm": "base",
    "step": 0,
    "uri": "tinker://b10b6b569002993796d968f558a5fa8f:train:0/sampler_weights/s0c0final-ckpt0"
  }
]

{
  "base": {
    "accuracy": 0.686
  }
}

base_accuracy_gsm8k500

0.686

base_no_answer_extracted

93

pilot_c0_step0_mean_reward

0.6094

pilot_c0_step0_grad_norm

0.02424

pilot_sconly_step0_mean_reward

0.5977

pilot_sconly_step1_mean_reward

0.5898

pilot_sconly_step1_grad_norm

0.02214

pilot_logprob_gap_absmean

0.007728

is_composed_vs_baseline_grad_max_abs_diff_reference

0

sc_only_vs_baseline_grad_cosine_reference

0.9525

• checkpoint · pilot_sc_only_s0 · step 2 · tinker://25e01ce3a6adb0fc2a45e3a9eb8bfa74:train:0/sampler_weights/s0c1final-ckpt2
• checkpoint · base · step 0 · tinker://b10b6b569002993796d968f558a5fa8f:train:0/sampler_weights/s0c0final-ckpt0
• code_revision · code-revision.txt
• effective_config · effective-config.json

1. Nov 3, 23:35 UTC · failed · Stopped before full training. On Qwen3.5-2B (0.8B is not served), I measured base GSM8K accuracy of 0.686 on 500 test items, ran 3 pilot steps, and checked the loss against the official score-centeri…
2. Nov 3, 23:16 UTC · running · Setting up: Qwen/Qwen3.5-0.8B is not served on shared Spindle; using Qwen/Qwen3.5-2B. Running optimizer preflight and correctness checks for the custom score-centering loss before any training.

adaptation_manifests

[]

correctness_preflight

{'required': True, 'paths': [{'id': 'score_centering_custom_loss', 'fixed_input': {'sha256': '46b68dbee02a8a2681ed420edf15f2daca0da7931c63f72a665c4827635757fa', 'description': 'parity_fixture.json: synthetic B=8,T=24,V=96,k=8 trainer logits, sampler logits with N(0,0.3) mismatch, sampled tokens, masks, group-centered advantages, sampler top-k (seed 1234); stateless loss, no optimizer state'}, 'baseline_revision': 'spec gives none; used martin-marek/score-centering@fdca22b losses.py loss_fn_rl(correction=ISCorrection(), score_center=False)', 'reference_revision': 'martin-marek/score-centering@fdca22b losses.py loss_fn_rl (top-k head branch), correction=ISCorrection() [is_composed] and correction=None [sc_only]', 'implementation_revision': 'sc_loss.py sha256 0e5553db193feedb15bc86f6617e856eeb3f17ea94974da54332f67d1a6e1a95', 'precision': 'float32 (torch CPU vs JAX CPU)', 'normalization': 'sum over masked completion tokens of A*score, divided by total masked tokens in batch', 'errors': {'baseline_loss': {'max_abs_error': 5.21540641784668e-08, 'reference_max_abs': 0.08615894615650177, 'relative_l2_error': 6.053238404719173e-07, 'cosine_similarity': 1.0}, 'baseline_grad': {'max_abs_error': 2.7939677238464355e-09, 'reference_max_abs': 0.009415442124009132, 'relative_l2_error': 1.1184862852024875e-07, 'cosine_similarity': 0.9999999999999938}, 'reference_is_composed_loss': {'max_abs_error': 1.4901161193847656e-08, 'reference_max_abs': 0.08463554084300995, 'relative_l2_error': 1.7606269240351104e-07, 'cosine_similarity': 1.0}, 'reference_is_composed_grad': {'max_abs_error': 2.7939677238464355e-09, 'reference_max_abs': 0.009415442124009132, 'relative_l2_error': 1.1184862852024875e-07, 'cosine_similarity': 0.9999999999999938}, 'reference_sc_only_loss': {'max_abs_error': 1.862645149230957e-09, 'reference_max_abs': 0.013056991621851921, 'relative_l2_error': 1.4265500072112103e-07, 'cosine_similarity': 1.0}, 'reference_sc_only_grad': {'max_abs_error': 4.656612873077393e-10, 'reference_max_abs': 0.005188265349715948, 'relative_l2_error': 4.518269532751553e-08, 'cosine_similarity': 0.9999999999999991}}, 'lr_scaling': []}]}

correctness_validation

{'status': 'unverified', 'required': True, 'checks': [], 'reason': 'Correctness preflight was not recorded.', 'evidence_source': 'executor report; remote execution is not attested', 'applicability': 'unknown', 'reported_verdict': 'inconclusive'}

evaluation

{'version': 1, 'evaluators': {'7d7a4ef4389082ab99584fb3140d8ca1c1aca38ea7571bdf7d03efcc272040aa': {'code_sha256': '82bf57c0e5c4067a7cbbfa3a6edb13000bd99970bb54a4b26a80e1764d54bf93', 'prompt_template': 'Qwen/Qwen3.5-2B HF chat template, apply_chat_template([{\'role\':\'user\',\'content\':question.strip()+" Reason step by step, then give the final answer as a number after \'####\'."}], add_generation_prompt=True, enable_thinking=False), no extra special tokens', 'decoding': {'temperature': 0, 'max_tokens': 512, 'stop': []}, 'parser': "common.py extract(): first number after last '####', else last \\boxed{}; norm(): strip commas/trailing '.', float->int string when integral; exact string match to gold after '####'", 'items': {'ids_rule': 'gsm8k-test-0 .. gsm8k-test-499 (openai/gsm8k main test, first 500 rows)'}}}, 'scores': [{'arm': 'base', 'metric': 'accuracy', 'value': 0.686, 'evaluator_id': '7d7a4ef4389082ab99584fb3140d8ca1c1aca38ea7571bdf7d03efcc272040aa', 'predictions_ref': 'base-eval/predictions_step0.jsonl (sha256 a0682ab8ed1dc00f9da90b2d7a276338bd96e7ed5ec59b3d8a24343c640dc3c3, not attached)'}], 'comparisons': []}

evaluator_validation

{'status': 'invalid', 'issues': ['Evaluator 7d7a4ef4389082ab99584fb3140d8ca1c1aca38ea7571bdf7d03efcc272040aa: item IDs or a hashed item manifest are required.', 'Evaluator 7d7a4ef4389082ab99584fb3140d8ca1c1aca38ea7571bdf7d03efcc272040aa: identity does not match provenance hash.'], 'comparisons': [{'metric': 'accuracy', 'status': 'unknown', 'arms': ['base'], 'evaluator_ids': ['7d7a4ef4389082ab99584fb3140d8ca1c1aca38ea7571bdf7d03efcc272040aa'], 'missing_arms': []}]}

feasibility

{'status': 'unknown', 'problems': ['Re-projection required: lora_rank changed or was not recorded', 'Re-projection required: dataset changed or was not recorded', 'Re-projection required: steps changed or was not recorded', 'Re-projection required: batch_size changed or was not recorded', 'Re-projection required: max_seq_len changed or was not recorded', 'Re-projection required: arms changed or was not recorded', 'Re-projection required: n_examples changed or was not recorded', 'Invalid configuration steps/evaluation count'], 'wall_seconds': None, 'gpu_hours': None, 'known_gpu_hours': 0, 'phases': {'setup': {'wall_seconds': 21.0, 'planned_units': 1, 'planned_samples': 0.0, 'gpu_hours': None}, 'training': {'wall_seconds': 36546.66666666667, 'planned_units': 400, 'planned_samples': 102400.0, 'gpu_hours': None}, 'evaluation': {'wall_seconds': 844.8, 'planned_units': 22000, 'planned_samples': 22000.0, 'gpu_hours': None}, 'recovery': {'wall_seconds': 1200, 'gpu_hours': None}}, 'generation_counts': None, 'generation_applicability': {'check': 'generation_audit', 'status': 'unknown', 'enforce': False, 'reasons': ['spec has no experiment_features: generated_training_data'], 'features': {'generated_training_data': None}}, 'record': {'config': {'base_model': 'Qwen/Qwen3.5-2B', 'steps': 'pilot 1-2', 'groups': 32, 'group_size': 8, 'max_tokens': 512, 'topk': 32, 'n_eval': 500}, 'effective_config': {'base_model': 'Qwen/Qwen3.5-2B', 'arms': ['baseline_s0', 'baseline_s1', 'center_s0', 'center_s1'], 'steps': 100, 'batch_size': 256, 'max_tokens': 512, 'n_eval': 500}, 'primary_endpoint': 'greedy exact-match accuracy; peak-minus-final drop over the run', 'source': '/workspace/rx/pilot-c0, pilot-c1-sconly, base-eval metrics.jsonl', 'measurements': {'setup': {'seconds': 21, 'units': 1, 'planned_units': 1, 'samples': 0, 'gpu_count': None, 'allocation_source': None}, 'training': {'seconds': 274.1, 'units': 3, 'planned_units': 400, 'samples': 768, 'gpu_count': None, 'allocation_source': None}, 'evaluation': {'seconds': 19.2, 'units': 500, 'planned_units': 22000, 'samples': 500, 'gpu_count': None, 'allocation_source': None}}, 'generation': None, 'recovery': {'seconds': 1200, 'gpu_count': None, 'allocation_source': None}}, 'recorded_at': 1793748933.6070862}

feasibility_required

True

initialization_provenance

{'version': 1, 'arms': {}, 'note': 'no paired training ran; initial_output available=false (not captured)'}

launch

{'experiment_id': 'exp_6188892aaf', 'context_status': 'legacy_unknown', 'superseded': False, 'mode': 'redo', 'purpose': "Repeat the selected saved experiment version to check reproducibility. This repetition tests that version's hypothesis and methods."}

progress_updated_at

1793748910.273956

reported_outcome_reason

incomplete_execution

reported_verdict

inconclusive

restoration_validation

{'status': 'unverified', 'required': False, 'checks': [], 'reason': 'Initialization and optimizer evidence was not required by this saved spec.', 'applicability': 'unknown'}

started_at

1793747567.206896