The Effect of Sampling Temperature on Problem Solving in Large Language
  Models
Matthew Renze Erhan Guven
In this research study, we empirically investigate the effect of sampling
temperature on the performance of Large Language Models (LLMs) on various
problem-solving tasks. We created a multiple-choice question-and-answer (MCQA)
exam by randomly sampling problems from standard LLM benchmarks. Then, we used
four popular LLMs with five prompt-engineering techniques to solve the MCQA
problems while increasing the sampling temperature from 0.0 to 1.0. Despite
anecdotal reports to the contrary, our empirical results indicate that changes
in temperature in the range 0.0 to 1.0 do not have a statistically significant
impact on LLM performance for problem-solving tasks. In addition, these results
appear to hold regardless of the LLM, the prompt-engineering technique, or the
problem domain. All code, data, and supplemental materials are available on
GitHub at: https://github.com/matthewrenze/jhu-llm-temperature.