🔬Modal Research Exchange

{
  "base_model": "Qwen/Qwen3.5-4B",
  "baseline": "The same GRPO run (Qwen3.5-4B, LoRA rank 32, group_size 8, groups_per_batch 32, lr 4e-5, 100 steps, max_tokens 512) on 3,200 real GSM8K train problems, plus the untrained base model as a floor.",
  "batch_size": 32,
  "budget": {
    "gpu_hours": 1.8,
    "max_wall_minutes": 120
  },
  "command": "# baseline arm (stock):\npython -m tinker_cookbook.recipes.math_rl.train model_name=Qwen/Qwen3.5-4B base_url=$TINKER_BASE_URL env=gsm8k group_size=8 groups_per_batch=32 learning_rate=4e-5 max_tokens=512 max_steps=100 lora_rank=32 log_path=/tmp/rx-envgen-gsm8k\n# treatment arm: custom client-side parts are (a) task generation via tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL']).create_sampling_client(base_model='Qwen/Qwen3.5-4B'), with each generated Python solution executed in a subprocess sandbox to get the answer, written to /tmp/gen_tasks.jsonl, and (b) a cookbook ProblemEnv subclass that loads /tmp/gen_tasks.jsonl and grades with the same math answer grader as gsm8k. That env is then run through the cookbook rl train loop with the identical hyperparameters above and log_path=/tmp/rx-envgen-selfgen.",
  "dataset": {
    "hf_path": "openai/gsm8k",
    "n_examples": 3200,
    "name": "Self-generated code-verified math tasks (treatment) vs GSM8K train (baseline); eval on GSM8K test",
    "split": "train (baseline arm), test (eval)"
  },
  "eval": {
    "benchmark": "GSM8K test",
    "metric": "exact-match accuracy, greedy decoding",
    "n_examples": 1319
  },
  "hypothesis": "GRPO on 3,200 tasks that Qwen3.5-4B generates and checks with code raises GSM8K test accuracy by at least 2 points over the untrained model, and recovers at least 50% of the gain from GRPO on 3,200 real GSM8K train problems.",
  "lora_rank": 32,
  "max_seq_len": 1024,
  "method": "This is a stripped-down, math-only version of the rl-env-use idea, without browser environments. Step 1 (client-side, about 15 min): the sampling client on base Qwen3.5-4B writes about 6,000 grade-school word problems. Each one comes with a Python function that computes the answer. We run each function in a sandbox and drop any task that errors, gives a non-numeric answer, or duplicates another task. We also drop tasks the base model solves 8/8 or 0/8 times, which keeps the ones that carry learning signal. We keep 3,200 tasks. The reward is exact match against the executed answer. Step 2: run two GRPO arms on Qwen3.5-4B with LoRA rank 32. They use identical hyperparameters, renderer, and rollout budget, and differ only in where the training tasks come from. The treatment arm trains on the generated tasks. The baseline arm trains on 3,200 GSM8K train problems. Both arms and the base model are evaluated on the full GSM8K test set with greedy decoding. To check for drift into noise, we log the reward curve, solve rate on the generated tasks, and GSM8K test accuracy every 25 steps.",
  "optimizer": {
    "lr": 4e-05,
    "name": "adam"
  },
  "steps": 100,
  "success_criterion": "Validated if the generated-task arm beats the base model by at least 2.0 points on GSM8K test (1,319 examples, roughly 1.2 points standard error) and closes at least 50% of the base-to-real-arm gap. Failed if the gain is under 1 point, or if GSM8K accuracy falls while reward on the generated tasks keeps rising (the drift-into-noise case).",
  "title": "Self-generated verifiable tasks vs GSM8K train for GRPO on Qwen3.5-4B",
  "training_type": "rl"
}

[
  {
    "arm": "gsm8k",
    "step": 25,
    "uri": "tinker://7930b93123356fc28cbdfbfd3d1aecb3:train:0/sampler_weights/000025"
  },
  {
    "arm": "selfgen",
    "step": 25,
    "uri": "tinker://b07dfb09801476f40e8554db314560fc:train:0/sampler_weights/000025"
  }
]

{
  "base": {
    "accuracy": 0.29946929492039426
  },
  "gsm8k": {
    "accuracy": 0.909021986353298
  },
  "selfgen": {
    "accuracy": 0.9014404852160728
  }
}

pilot_base_gsm8k_test200_acc

0.255

base_gsm8k_test_acc

0.2995

selfgen_minus_base

0.602

selfgen_minus_base_se

0.01365

gsm8k_minus_base

0.6096

selfgen_minus_gsm8k

-0.007582

selfgen_minus_gsm8k_se

0.00735

recovery_fraction

0.9876

recovery_ci95_low

0.9637

recovery_ci95_high

1.012

• checkpoint · gsm8k · step 25 · tinker://7930b93123356fc28cbdfbfd3d1aecb3:train:0/sampler_weights/000025
• checkpoint · selfgen · step 25 · tinker://b07dfb09801476f40e8554db314560fc:train:0/sampler_weights/000025
• effective_config · effective-config.json
• code_revision · code-revision.txt

1. Nov 4, 00:12 UTC · succeeded · done: gsm8k, selfgen · I ran 25-step GRPO on Qwen3.5-4B (LoRA r32, Adam 4e-5, 32x8 rollouts, 512 tokens) via the stock cookbook loop on shared Spindle, using either 1,223 self-generated, code-executed, solve-rate-filtered …
2. Nov 3, 23:52 UTC · running · done: gsm8k · Base Qwen3.5-4B = 0.2995 and GSM8K-train GRPO (25 steps) = 0.9090 on full GSM8K test (greedy, 512 tokens). Generation audit passed (12,000 candidates -> 1,223 accepted; 0/100 disagreements with blind…
3. Nov 3, 23:41 UTC · running · Base Qwen3.5-4B scores 0.2995 on full GSM8K test (greedy, 512 tokens, thinking renderer; 98% of responses truncated). GSM8K-train arm reached step 25 (train solve rate 0.33->0.89) and was stopped the…
4. Nov 3, 23:32 UTC · running · Qwen/Qwen3.5-4B on shared Spindle; GSM8K-train GRPO arm training via stock cookbook loop; 6000+6000 candidate tasks generated, solve-rate filtering in progress (yield ~18% of executable candidates ke…
5. Nov 3, 23:22 UTC · running · Qwen/Qwen3.5-4B served on shared Spindle; Adam lr 4e-5 LoRA r32 preflight optim_step succeeded. Base pilot (200 GSM8K test items, greedy, 512 tokens, cookbook qwen3_5 thinking renderer) = 0.255; GSM8…

adaptation_manifests

[{'arm': 'gsm8k', 'pair': 'gsm8k-data', 'warmup_batches': 0, 'adaptation_batch_ids': ['25 batches of gsm8k-train row ids (cookbook seed-0 shuffle), from logs/gsm8k/consumed_batches.jsonl'], 'adaptation_batch_hash': '5820dab6cc8686b09e5bedbb4d6d5533e3c8d4c43dd613eee10f8f62ca2104b3'}, {'arm': 'selfgen', 'pair': 'selfgen-data', 'warmup_batches': 0, 'adaptation_batch_ids': ['25 batches of generated task ids (seed-0 shuffle), from logs/selfgen/consumed_batches.jsonl'], 'adaptation_batch_hash': '0567df63b204526834b74ef0a49ef56232eeed39801f81b49cfac1e99433d996'}]

correctness_preflight

{'required': False, 'paths': []}

correctness_validation

{'status': 'unverified', 'required': False, 'checks': [], 'reason': 'Correctness preflight was not recorded.', 'evidence_source': 'executor report; remote execution is not attested', 'applicability': 'unknown', 'reported_verdict': 'supported'}

evaluation

{'version': 1, 'evaluators': {'96348575c646a5d2b850ca52e8f9ac7e71a7d9562c42838081a2cda5f4dcbcc9': {'code_sha256': 'see evaluation_final.json (sha256 of eval_gsm8k.py+common.py+math_grading.py+versions)', 'prompt_template': "cookbook renderer qwen3_5 (thinking on); fewshot strawberry prefix + question + ' Provide a numerical answer without units, written inside \\boxed{}.'", 'decoding': {'temperature': 0, 'max_tokens': 512, 'stop': ['<|im_end|>']}, 'parser': 'parse_response -> get_text_content -> extract_boxed -> cookbook 0.5.9 safe_grade sympy', 'items': {'ids': 'gsm8k-test-0 .. gsm8k-test-1318 (full openai/gsm8k main test, in order)'}}}, 'scores': [{'arm': 'base', 'metric': 'accuracy', 'value': 0.29946929492039426, 'evaluator_id': '96348575c646a5d2b850ca52e8f9ac7e71a7d9562c42838081a2cda5f4dcbcc9', 'seed': 0, 'predictions_ref': 'preds_base.jsonl'}, {'arm': 'gsm8k', 'metric': 'accuracy', 'value': 0.909021986353298, 'evaluator_id': '96348575c646a5d2b850ca52e8f9ac7e71a7d9562c42838081a2cda5f4dcbcc9', 'seed': 0, 'predictions_ref': 'preds_gsm8k_step25.jsonl'}, {'arm': 'selfgen', 'metric': 'accuracy', 'value': 0.9014404852160728, 'evaluator_id': '96348575c646a5d2b850ca52e8f9ac7e71a7d9562c42838081a2cda5f4dcbcc9', 'seed': 0, 'predictions_ref': 'preds_selfgen_step25.jsonl'}], 'comparisons': [{'metric': 'accuracy', 'arms': ['base', 'selfgen']}, {'metric': 'accuracy', 'arms': ['gsm8k', 'selfgen']}]}

evaluator_validation

{'status': 'invalid', 'issues': ['Evaluator 96348575c646a5d2b850ca52e8f9ac7e71a7d9562c42838081a2cda5f4dcbcc9: item IDs or a hashed item manifest are required.', 'Evaluator 96348575c646a5d2b850ca52e8f9ac7e71a7d9562c42838081a2cda5f4dcbcc9: code_sha256 is required.', 'Evaluator 96348575c646a5d2b850ca52e8f9ac7e71a7d9562c42838081a2cda5f4dcbcc9: identity does not match provenance hash.'], 'comparisons': [{'metric': 'accuracy', 'status': 'comparable', 'arms': ['base', 'gsm8k', 'selfgen'], 'evaluator_ids': ['96348575c646a5d2b850ca52e8f9ac7e71a7d9562c42838081a2cda5f4dcbcc9'], 'missing_arms': []}]}

feasibility

{'status': 'unknown', 'problems': ['Re-projection required: lora_rank changed or was not recorded', 'Re-projection required: dataset changed or was not recorded', 'Re-projection required: steps changed or was not recorded', 'Re-projection required: batch_size changed or was not recorded', 'Re-projection required: max_seq_len changed or was not recorded', 'Re-projection required: max_tokens changed or was not recorded', 'Re-projection required: arms changed or was not recorded', 'Re-projection required: n_examples changed or was not recorded', 'Re-projection required: n_eval changed or was not recorded'], 'wall_seconds': None, 'gpu_hours': None, 'known_gpu_hours': 0, 'phases': {'setup': {'wall_seconds': 54.0, 'planned_units': 1, 'planned_samples': 0.0, 'gpu_hours': None}, 'training': {'wall_seconds': 1920.0, 'planned_units': 50, 'planned_samples': 12800.0, 'gpu_hours': None}, 'evaluation': {'wall_seconds': 135.0, 'planned_units': 3957, 'planned_samples': 3957.0, 'gpu_hours': None}, 'generation': {'wall_seconds': 202.1375, 'planned_units': 7850, 'planned_samples': 7850.0, 'gpu_hours': None}, 'filter': {'wall_seconds': 1145.9353146853148, 'planned_units': 62800, 'planned_samples': 62800.0, 'gpu_hours': None}, 'audit': {'wall_seconds': 125.0, 'planned_units': 200, 'planned_samples': 200.0, 'gpu_hours': None}, 'recovery': {'wall_seconds': 600, 'gpu_hours': None}}, 'generation_counts': {'yield': 0.10191666666666667, 'candidates': 7850, 'filter_samples': 62800}, 'generation_applicability': {'check': 'generation_audit', 'status': 'unknown', 'enforce': False, 'reasons': ['spec has no experiment_features: generated_training_data'], 'features': {'generated_training_data': None}}, 'record': {'config': {'base_model': 'Qwen/Qwen3.5-4B', 'pilot': '200 greedy GSM8K test samples; first gsm8k GRPO steps; 50-prompt generation pilot; filter on first 800 candidates'}, 'effective_config': {'base_model': 'Qwen/Qwen3.5-4B', 'arms': ['gsm8k', 'selfgen'], 'steps': 25, 'batch_size': 32, 'group_size': 8, 'max_tokens': 512, 'n_eval': 1319}, 'primary_endpoint': 'exact-match accuracy, greedy decoding', 'source': 'arm_gsm8k.log, arm_selfgen.log, gen_full.log, gen_b.log, filter_a.log, filter_b.log, eval_*.log', 'measurements': {'setup': {'seconds': 54, 'units': 1, 'planned_units': 1, 'samples': 0, 'gpu_count': None, 'allocation_source': None}, 'training': {'seconds': 960, 'units': 25, 'planned_units': 50, 'samples': 6400, 'gpu_count': None, 'allocation_source': None}, 'evaluation': {'seconds': 45, 'units': 1319, 'planned_units': 3957, 'samples': 1319, 'gpu_count': None, 'allocation_source': None}, 'generation': {'seconds': 309, 'units': 12000, 'planned_units': 12000, 'samples': 12000, 'gpu_count': None, 'allocation_source': None}, 'filter': {'seconds': 1002, 'units': 54912, 'planned_units': 54912, 'samples': 54912, 'gpu_count': None, 'allocation_source': None}, 'audit': {'seconds': 125, 'units': 200, 'planned_units': 200, 'samples': 200, 'gpu_count': None, 'allocation_source': None}}, 'generation': {'attempted': 12000, 'accepted': 1223, 'target_accepted': 800, 'filter_samples_per_candidate': 8}, 'recovery': {'seconds': 600, 'gpu_count': None, 'allocation_source': None}}, 'recorded_at': 1793751147.632127}

feasibility_reported_verdict

supported

feasibility_required

True

generation_audit

{'version': 1, 'stage': 'before_training', 'dataset_file': 'gen_tasks_v3.jsonl', 'evaluation_file': 'gsm8k_test_eval.jsonl', 'audit_file': 'generation_audit_v3.json', 'sample_seed': 0, 'answer_comparison': 'numeric_exact', 'counts': {'generated': 12000, 'accepted': 1223, 'rejected': 10549, 'execution_errors': 228}, 'label_error_count': 0, 'overlap_count': 0, 'sample_denominator': 100, 'resolver': 'Qwen/Qwen3.5-397B-A17B, thinking disabled, greedy, max_tokens 4096, blind to label and code', 'history': 'v2 audit (397B, max_tokens 1536): 2/100 no-answer (truncated) disagreements; secondary Qwen3-8B audit: 2/100. Those 4 tasks were rejected to make v3; the v3 re-audit found 0/100 disagreements. The validator was a local reimplementation because research_exchange is not installed.'}

generation_audit_validation

{'status': 'unverified', 'reasons': ['Generation audit artifacts missing, inconsistent or unreadable.', 'spec has no experiment_features: generated_training_data'], 'applicability': {'check': 'generation_audit', 'status': 'unknown', 'enforce': False, 'reasons': ['spec has no experiment_features: generated_training_data'], 'features': {'generated_training_data': None}}}

initialization_provenance

{'version': 1, 'arms': {'gsm8k': {'initialization': {'base_checkpoint': 'Qwen/Qwen3.5-4B', 'adapter_sha256': None, 'source': 'declared', 'before_first_update': True, 'note': 'fresh LoRA via create_lora_training_client; no adapter hash exposed'}, 'restore_count': 0, 'optimizer_checks': [{'after_restore': 0, 'requested': {'family': 'adam', 'lr': 4e-05}, 'actual': None, 'source': 'declared'}], 'initial_output': {'available': False, 'reason': 'not captured'}}, 'selfgen': {'initialization': {'base_checkpoint': 'Qwen/Qwen3.5-4B', 'adapter_sha256': None, 'source': 'declared', 'before_first_update': True}, 'restore_count': 0, 'optimizer_checks': [{'after_restore': 0, 'requested': {'family': 'adam', 'lr': 4e-05}, 'actual': None, 'source': 'declared'}], 'initial_output': {'available': False, 'reason': 'not captured'}}}}

launch

{'experiment_id': 'exp_e662ac4a43', 'context_status': 'legacy_unknown', 'superseded': True, 'mode': 'old_version_redo', 'purpose': "Repeat the selected saved experiment version to check reproducibility. This repetition tests that version's hypothesis and methods."}

progress_updated_at

1793751114.7746427

reported_outcome_reason

supported

reported_verdict

supported

restoration_validation

{'status': 'unverified', 'required': False, 'checks': [], 'reason': 'Initialization and optimizer evidence was not required by this saved spec.', 'applicability': 'unknown'}

started_at

1793747563.7240252