Abstract
Inference-time intervention (ITI) has emerged as a promising method for steering large language models (LLMs) in a particular direction (e.g., improving helpfulness) by intervening on token representations without costly updates to the model's parameters. However, existing ITI approaches struggle with multi-attribute settings with conflicts, such as enhancing helpfulness while reducing toxicity. To tackle this, we introduce Multi-Attribute Targeted Steering (MAT-Steer), a novel steering framework designed for selective token-level intervention across multiple attributes. MAT-Steer learns steering vectors using an alignment objective that shifts the model's internal representations of undesirable outputs closer to those of desirable ones while enforcing sparsity and orthogonality among vectors for different attributes, thereby reducing inter-attribute conflicts.
We evaluate MAT-Steer in two distinct settings: (i) on question answering (QA) tasks where we balance attributes like truthfulness, bias, and toxicity; (ii) on generative tasks where we simultaneously improve attributes like helpfulness, correctness, and coherence. MAT-Steer outperforms existing ITI and parameter-efficient fine-tuning approaches across both task types (e.g., 3% average accuracy gain across QA tasks and a 55.82% win rate against the best ITI baseline).
Blogger's Review: The MAT-Steer method proposed in this paper effectively addresses multi-attribute conflicts through the design of steering vectors, showcasing its potential to enhance language model performance in multi-task scenarios. This flexible intervention approach provides new insights for future LLM research and is worthy of further exploration.