A recent study reveals that self-consistency (SC) via majority vote may backfire for small large language models (LLMs). The researchers tested two instruction-tuned models from different families, Qwen2.5-7B and Llama-3-8B, on the GPQA Diamond benchmark (198 graduate-level science questions) and found that majority voting reduces per-problem accuracy on a majority of problems. Specifically, 56.6% of problems for Qwen2.5-7B and 65.7% for Llama-3-8B saw decreased accuracy. Furthermore, the researchers found that even using a grid oracle to select the best N (from 1 to 64) cannot reach the theoretical upper bound. They also discovered that confidence does not always track correctness, especially in the highest-agreement bin. Blogger's Review: This study exposes a potential flaw in small LLMs on science questions, indicating that self-consistency via majority vote may backfire. This finding has significant implications for LLM development and application, particularly in the science domain. Future research should focus on improving LLM performance on science questions, especially in balancing self-consistency and diversity.