FlexFloor is a system that enables earlier release of compute resources during synchronous response generation while preserving the expert computation required by ongoing responses. It achieves this by jointly adapting response scheduling and the active set of workers. The system leverages historical response lengths and memory constraints to decide batch sizes and prompt placement, then progressively releases workers whose local generation has finished, all while keeping expert availability and communication correct. Compared with the baseline, FlexFloor generates about 65.40% more response tokens per unit of occupied compute time and boosts throughput by 52.74%, with negligible change in rollout reward. This substantially reduces the computational cost of reinforcement‑learning post‑training, speeding up the iterative development of large language models and AI agents.
Review