OpenAI's GPT-2 Outperforms Custom Models on Instruction-Following Despite Worse Test Loss Scores

OpenAI's GPT-2 small weights are outperforming custom-trained models on instruction-following tasks despite worse test loss scores, sparking investigation into whether superior data quality, overtraining, or dropout are behind the surprising gap.

• OpenAI's GPT-2 small weights consistently outperform custom-trained models on instruction-following evaluations, despite some custom models achieving better cross-entropy loss scores on held-back test sets, pointing to a mismatch between pre-training loss landscapes and fine-tuning performance.
• Several hypotheses are being explored to explain the gap, including data quality differences between OpenAI's curated WebText corpus and the FineWeb dataset, the effects of dropout, and the possibility that OpenAI's models were overtrained well beyond the Chinchilla-optimal 20 tokens per parameter.
• An upcoming experiment will test whether overtraining custom models on significantly more data closes the performance gap with OpenAI's weights on instruction-following benchmarks, with weight tying and model size differences also noted as contributing factors to OpenAI's surprisingly strong results.