Gene set analysis underpins functional genomics but remains labor‑intensive and reliant on expert curation. Most existing large language model (LLM) approaches only use symbolic gene names, ignoring protein sequence information that governs molecular activity and interactions. We introduce SoftGene, a framework that leverages the hierarchical structure of gene sets to enhance LLM‑based annotation.\ \ First, a hierarchical attention encoder built on ESM—a protein language model—represents each gene set with embeddings derived from amino‑acid sequences. Second, a hybrid prompting scheme combines soft prompts from these gene‑set embeddings with hard prompts containing auxiliary context generated by an LLM; the concatenated prompt is fed to a local LLM for annotation.\ \ We evaluate the method on two benchmarks, Gene Ontology (GO) and the Molecular Signatures Database (MSigDB). Results show that integrating protein‑sequence representations with textual context improves overall annotation performance, while domain‑specific analyses reveal that the impact of protein embeddings varies across biological domains.\ \ Review: SoftGene demonstrates that incorporating protein‑level sequence features into LLM prompting yields more interpretable and accurate gene set annotations, offering a promising cross‑modal direction for functional genomics.