Creative image editing tools, such as Photoshop's Remove or Generative Fill buttons, are central to everyday customer use and account for a major share of traffic in Photoshop and Lightroom. However, current generative AI models face significant latency challenges, especially when transitioning from convolution-based U-Nets to Diffusion Transformers (DiTs). Our evaluation on hundreds of representative image editing samples shows that the DiT module alone accounts for an average of 73% of the total model latency, even after being distilled from 50 timesteps down to 8.
To address this challenge, we propose HiLo-Token, an input-adaptive token compression framework that allocates more token budget to high-frequency, rich-context regions while assigning fewer tokens to low-frequency areas. Specifically, for the editing region specified by the user mask, we retain all tokens within a dilated mask to preserve strong locality and contextual relevance. Outside the editing region, we introduce a simple yet effective high-frequency token selection strategy based on spatial frequency to capture important local details, while using tokens from a 16x downsampled image to represent low-frequency components and preserve the blurry but global structure. Extensive experiments on production-level evaluation data validate the effectiveness of our method, achieving 3.13x, 2.59x, and 1.67x DiT speedups on A100-80GB for image editing tasks across small, medium, and large mask ratio categories with average ratios of 6.38%, 15.92%, and 35.36%, respectively, without any regression in generation quality.
Blogger's Review: The introduction of HiLo-Token effectively addresses the latency issues in current generative models by optimizing token allocation for high and low-frequency information. The experimental results demonstrate significant performance improvements, marking a substantial advancement in image processing technology.