In deep neural network (DNN) inference, bit-serial accelerators leverage bit-level sparsity to reduce inference costs, but existing designs only exploit sparsity on one operand, limiting speedup.
By extending sparsity exploitation to both operands, significant reductions in partial products can be achieved, yet this introduces a critical new bottleneck: workload imbalance. The execution cost of each concurrent weight-activation pair depends on the product of two independently varying operand non-zero bit counts, leading to vast differences in completion times, leaving faster computations idle. This limits processing element (PE) utilization to 56-64% in existing dual-sided designs.
We present BRIM, a hardware-software co-designed dual-sided bit-serial sparse accelerator that directly targets this bottleneck. BRIM combines two integrated mechanisms:
- Cyclic-Balanced Pruning (CBP), a post-training weight optimization that reshapes weight representations based on profiled activation statistics to equalize expected workloads across concurrently processed pairs offline;
- Pairwise Slot Donation, a lightweight hardware mechanism that absorbs residual runtime imbalance with negligible area overhead.
Evaluated across CNNs, ViTs, and LLMs under iso-area constraints, BRIM achieves over 90% PE utilization, up to 2.37x speedup, and up to 1.63x energy efficiency improvement over prior dual-sided designs.