Fine-tuning a Language Model for Text-to-SQL with Constrained Decoding
Problem
A query can run successfully and still select the wrong columns or join the wrong tables. I compared the original and adapted models on the same questions, scoring executability and result correctness separately.
Methods
- I fine-tuned the model on 800 examples using QLoRA on one 16 GB V100. The reference SQL syntax trees supplied labels for tables, columns and join edges. A full-vocabulary KL penalty anchored training to the frozen base model.
- During generation, a token-level grammar and alias-scope check blocks invalid continuations. When static checks flag column ownership or a join path, the model gets one new attempt using the question, full schema and diagnostic feedback.
Results
On the same 100 BIRD development questions, the base model returned 36 correct results and 71 executable queries. Fine-tuning with constrained decoding brought these counts to 44 and 75. One repair attempt made two more queries executable, with 44 correct results overall.