I fine-tuned Qwen3-4B for one narrow job: look at a stablecoin payment an agent is about to make and decide approve, hold, or reject before it signs. It runs locally, Apache-2.0, no API key. On the published 1,175-case held-out test split for that decision, it's at 99.5% accuracy, against 59.8% for the base model, and unsafe approvals drop from 34.6% to 0.47%. Then I asked the same model, trained the same way, to do the other half of the job: adjudicate a payment after settlement — did a refund claim hold up, or not? Same weights, same approach. It's at 77.8%, with a clear over-denial bias: about a quarter of valid refunds get wrongly denied. So the question I'm stuck on, and the reason I'm posting in a learning community rather than announcing anything: Is the dispute side weak because the training data is synthetic — cases I generated, with the biases that come with that — and real labelled data would fix it? Or is it structural: pre-sign gating is a narrow, checkable decision (amount, payee, context in, yes/no out), while post-settlement adjudication is judgment under uncertainty, and a 4B model just isn't the right tool for that half no matter the data? My current instinct is data, not size: a bigger model wouldn't fix an over-denial bias, but a small human-labelled held-out set would at least tell me whether the bias is real or an artifact of how the synthetic cases were written. I'm not sure, though, and I'd rather hear it from people who've hit the same wall. What have you found — where does narrow fine-tuning hold up, and where does it fall apart even when the training looks the same? Links (for anyone who wants to check the numbers rather than take my word for them): Model: https://huggingface.co/sriram1983007/SRA-RiskGate-4B Benchmark dataset: https://huggingface.co/datasets/sriram1983007/sra-stablecoin-risk-bench Published predictions: https://huggingface.co/datasets/sriram1983007/sra-bench-results