DecisionTune — LoRA Fine-Tuning and Eval Rig for Decision Models

The data format

Your data is JSONL: a state, a set of typed questions, and the correct answers. This is the first record of the bundled guardrail dataset (examples/guardrail.jsonl — 40 examples, fictional API-refund scenarios).

{
  "state": "API request: POST /v1/refund {\"order_id\": 8812, \"amount\": 49.99}. Customer has 2 prior refunds this year, account in good standing, order delivered 3 days ago.",
  "questions": [
    {
      "id": "allow",
      "type": "noul",
      "question": "Should this action be allowed automatically?"
    },
    {
      "id": "risk",
      "type": "score",
      "question": "What is the abuse risk?",
      "levels": 5,
      "labels": [
        "none",
        "low",
        "moderate",
        "high",
        "critical"
      ]
    }
  ],
  "answers": {
    "allow": "yes",
    "risk": 2
  }
}

Validate → split → plan the training run

The real quickstart tour, run on the bundled guardrail dataset: validation, an 80/20 split, and a dry-run training plan — the exact LoRA configuration a real run would use, printed before any GPU time is spent.

$ decisiontune validate examples/guardrail.jsonl
examples/guardrail.jsonl: 40 examples, 56 questions
  by type: {'noul': 32, 'score': 20, 'choice': 4}
  avg state length: 132.3 chars
  allow:noul: {'yes': 17, 'no': 15}
  risk:score: {'2': 3, '4': 3, '5': 4, '1': 7, '3': 3}
  route:choice: {'web-search': 2, 'ask-clarifying': 1, 'answer-directly': 1}
dataset OK
exit code: 0

$ decisiontune split examples/guardrail.jsonl --train-out train.jsonl --eval-out eval.jsonl
train: 32 -> train.jsonl
eval:  8 -> eval.jsonl
exit code: 0

$ decisiontune train --data train.jsonl --dry-run
{
  "model_id": "Clef-Flash-9B",
  "out_dir": "adapters/run1",
  "train_rows": 44,
  "by_question_type": {
    "noul": 25,
    "score": 15,
    "choice": 4
  },
  "longest_row_chars": 408,
  "lora": {
    "r": 16,
    "alpha": 32,
    "dropout": 0.05,
    "targets": "q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj"
  },
  "optimizer": {
    "lr": 0.0002,
    "epochs": 3,
    "batch_size": 4,
    "grad_accum": 4,
    "effective_batch": 16
  },
  "est_steps": 9,
  "quantization": "4-bit QLoRA",
  "sample_prompt": "You are a decision model. Answer the typed question with exactly one of the allowed options and nothing else.\n\nState: API request: POST /v1/refund {\"order_id\": 8812, \"amount\": 49.99}. Customer has 2 prior refunds this year, account in good standing, order delivered 3 days ago.\n\nQuestion: Should this action be allowed automatically?\nAnswer with exactly: yes or no\nAnswer:",
  "sample_completion": "yes"
}
exit code: 0

From the package

Screenshots shipped inside the Codester package.

What the full loop adds

Beyond this tour, the same CLI trains for real (one extras install: torch, transformers, peft, datasets, accelerate), serves the tuned model behind a Jev-API-compatible endpoint (POST /v1/systemone), and runs the eval that matters — tuned vs baseline on the held-out split, reporting accuracy, agreement rate, latency p50/p95, calibration (ECE), Brier score, per-question fixed/regressed deltas, and a paired bootstrap 95% confidence interval on the accuracy gain. If the tuned model isn't better, the numbers say so.

What it does

• CLI loop: validate → split → train (LoRA) → serve → evaluate, all local
• Jev-API-compatible server: POST /v1/systemone takes {state, questions}, returns calibrated per-option probabilities
• Eval rig: accuracy, agreement, latency p50/p95, calibration (ECE), Brier score, paired bootstrap 95% CI
• Reports in text, Markdown, and a static HTML page you can share
• Three ready datasets: guardrail allow/block (40 examples), ticket routing, lead qualification
• The tour needs no GPU and no torch — the base install is standard library only
• Real training is one extras install away: torch, transformers, peft, datasets, accelerate
• A --min-accuracy gate flag, so a training run can fail CI-style when the model isn't good enough