RaRe

Reasoning-Aware Relational Representation Learning for Open-Vocabulary Scene Graph Generation

Jiawei Xiao1,*, Boya Wang1,*, Longtian Qiu1, Shan Ning1,3, Xuming He1,2,†
1ShanghaiTech University 2Shanghai Engineering Research Center of Intelligent Vision and Imaging 3Lingang Laboratory *Equal contribution   †Corresponding author
NeurIPS 2026
OpenReview Code Coming soon
RaRe performance and qualitative comparison
RaRe improves category-balanced relation recognition and encourages object-pair-specific relational evidence instead of generic CoT descriptions.
Overview

Abstract

Open-Vocabulary Scene Graph Generation (OV-SGG) requires models to recognize visual relationships beyond the training vocabulary, yet existing methods often rely on dataset-specific object–predicate co-occurrence patterns. Under long-tailed distributions and noisy object localization, such reliance produces biased, entangled relation representations and limits both rare-relation recognition and generalization to unseen predicates.

We propose RaRe, a reasoning-aware relational representation learning framework that transforms MLLM-generated chain-of-thought rationales into structured relation embeddings. RaRe first aligns rationale embeddings with textualized relation descriptions through self-supervised contrastive learning, then refines rationale generation with a specificity-oriented reinforcement learning objective that favors object-pair-specific relational evidence. Experiments on VG150 and PSG demonstrate improved category-balanced recognition, while revealing that the additional gains from learning and calibration vary across datasets and proposal settings.

Why RaRe?

From free-form reasoning to structured relations

01

Transfer beyond labels

Instance-level rationales capture spatial layout, object attributes, action states, and contextual cues that transfer beyond a fixed predicate vocabulary.

02

Separate similar relations

Contrastive alignment reshapes linguistically plausible but entangled CoT outputs into a retrieval-compatible relational embedding space.

03

Reward specificity

Policy optimization penalizes responses shared across unrelated entity pairs and promotes distinctive, object-pair-specific evidence.

Approach

Method

RaRe training pipeline
Training pipeline. RaRe learns a relational embedding space from paired CoT rationales and predictions, then uses this space to optimize MLLM generation through a specificity reward.
1

Self-Supervised Relational Space Construction

Each MLLM response naturally couples a CoT rationale with a relation prediction. A symmetric InfoNCE objective aligns these two views without requiring novel-predicate annotations.

Lstage1 = ½ (LCoT→pred + Lpred→CoT)
2

Specificity-Driven Policy Optimization

The learned embedding model supplies a representation-aligned reward. Responses resembling outputs for other entity pairs receive lower rewards, steering GRPO toward specific relational evidence.

ri = rispec + λ rifmt
3

Embedding Retrieval

At inference, RaRe embeds the complete generated response and matches it against multi-template predicate prototypes, followed by lightweight score normalization and prior calibration.

RaRe inference pipeline
Inference. CoT-based visual description, multi-template predicate prototypes, and embedding retrieval produce the final structured relation.
Evaluation

Main Results

VG150 evaluates both Predicate Classification (PredCls) and Scene Graph Detection (SGDet); PSG provides an additional detector-based evaluation.

22.49 Novel mR@50 VG150 · PredCls
24.62 Total mR@50 VG150 · PredCls
10.38 Total mR@50 VG150 · SGDet
17.52 Total mR@50 PSG · SGDet

VG150 PredCls

MethodTotal mR@50/100Base mR@50/100Novel mR@50/100
SGTR+ + RAHP16.88 / 22.1820.51 / 25.1811.82 / 15.46
PE-NET + RAHP24.50 / 28.2524.99 / 30.1915.70 / 23.73
RaRe24.62 / 25.8025.53 / 26.4522.49 / 24.27

Detector-based SGDet

Dataset / MethodTotal mR@50/100Novel mR@50/100
VG150 · ACC4.30 / 5.112.48 / 3.17
VG150 · ACC + RaRe10.38 / 12.326.69 / 8.02
PSG · PGSG13.50 / 16.407.40 / 11.30
PSG · PGSG + RaRe17.52 / 20.5914.76 / 16.10

Under fixed ACC proposals on VG150, training-free MLLM retrieval explains most of the Total mR gain; adding SSL, RL, and prior calibration gives modest additional Total mR gains while reducing Novel mR. On PSG, the same-proposal ablation improves both Total and Novel mR. RaRe therefore reports these setting-dependent effects explicitly.

Representation Analysis

More discriminative relational reasoning

t-SNE visualization of CoT representations
Self-supervised alignment and GRPO improve relational separation over the off-the-shelf embedding baseline.

What changes?

  • Global macro mean ratio rises from 0.0944 to 0.3177.
  • Novel macro mean ratio rises from 0.9683 to 1.4717.
  • Average CoT length falls from 392.8 to 297.1 tokens.

The improvements therefore do not rely on longer rationales; they come from making the generated evidence more specific and retrieval-compatible.

Head predicate prediction shift analysis
Novel-predicate samples shift away from frequent head predicates and toward their novel ground-truth semantics.
Qualitative Results

Object-pair-specific evidence

Qualitative example of motorcycle and street relation
Parked on. RaRe resolves a more specific relation from visual state and contact evidence.
Qualitative example of person carrying surfboard
Carrying. The rationale focuses on how the person physically holds the surfboard.
Qualitative example of wheel and skateboard relation
Part of. Structural and functional evidence separates component relations from generic spatial ones.
Citation

BibTeX

@inproceedings{xiao2026rare,
  title={Reasoning-Aware Relational Representation Learning for Open-Vocabulary Scene Graph Generation},
  author={Xiao, Jiawei and Wang, Boya and Qiu, Longtian and Ning, Shan and He, Xuming},
  booktitle={Advances in Neural Information Processing Systems},
  year={2026}
}