Multimodal objects often require different partitions under different instructions—for example, an e‑commerce catalog can be grouped by color, material, or both. Retrieval‑oriented embeddings typically fail to recover such variable partitions, while generative clustering relies on intermediate descriptions or iterative decisions. To address this, we propose the CIM-Embed framework that learns instruction‑conditioned multimodal embeddings directly for clustering.\
During training we use supervised contrastive learning (SupCon). Positives are samples belonging to the same group under the same instruction, while negatives come from other groups; samples from other tasks can also be incorporated to improve generalization. To handle tasks with varying numbers of groups, we construct heterogeneous structured batches that mix samples from different tasks in a single batch. The contrastive loss can be written as:
$$ L = -\frac{1}{|P|}\sum_{i\in P}\log\frac{\exp(\text{sim}(z_i, zi^+)/\tau)}{\sum{j\in N\cup{i}}\exp(\text{sim}(z_i, z_j)/\tau)} $$
where $z_i$ is the embedding of sample $i$, $z_i^+$ its positive counterpart, $N$ the set of negatives, and $\tau$ a temperature hyper‑parameter.\
At inference time the model receives image, text, and instruction as three modalities and maps them directly to a single vector. We then apply HDBSCAN to cluster all vectors density‑based, without requiring the true number of clusters.\
To evaluate the approach we built ProductMIC, derived from Amazon Reviews 2023 product metadata. The benchmark contains 5,373 products and 449 single‑attribute or compositional grouping tasks spanning 50 leaf categories that are excluded from training. Results show that with full‑dimensional embeddings and HDBSCAN, direct within‑task SupCon training achieves a mean V‑measure of 95.01, far surpassing the strongest general‑purpose baseline WeMM 9B, which scores 53.06. Frozen product‑trained encoders also improve out‑of‑domain conditional grouping, and controlled transfer experiments disentangle partition transfer from instruction response abilities.\
The code and dataset will be released upon acceptance.\
Review