In 2019, OpenAI released two million GPT-2 outputs—ungrammatical and half broken—to aid the detection of machine-generated text. The alignment that produced their more fluent successors is usually regarded as an engineering achievement; we interpret it instead as the latest expression of optimization culture: the belief that measurable improvements along predefined axes exhaust the question of value.
Tracing that conviction through the stack—pretraining, decoding, preference tuning, benchmarking, interface—and back through its genealogy in the audit society, we arrive at the limit: an optimization procedure can measure how improbable a piece of generated text is; it cannot tell whether that unlikelihood is error or invention. A procedure that cannot make that distinction has nonetheless, within half a decade, assumed the authority to set the protocols of legitimate language. Held for centuries by academies and schoolrooms, grammars and examiners, this authority has been given over to loss functions, reward models, benchmarks, and system prompts: an apparatus that executes the office of judgment with no capacity for judging.
Blogger's Review: The article profoundly reveals the impact of optimization culture on language generation technologies, emphasizing that relying solely on quantitative metrics fails to truly grasp the value and creativity of text. We need to reassess the role of algorithms in language judgment and avoid the blind pursuit of optimization while neglecting the complexity of human judgment.