HCGRec

Hint-Conditioned Generative Recommendation with Semantic IDs

HCGRec is a research codebase for semantic-ID generative recommendation. The repository covers the full workflow used in the project: raw Amazon-style filtering, semantic ID construction, supervised fine-tuning with LLaMA-Factory, hint-conditioned GRPO training, and constrained-decoding evaluation.

Overview

HCGRec focuses on a training failure mode in recommendation-oriented RL. In many rollout groups, none of the sampled continuations reaches the target item branch, so every sample receives the same reward and the update carries little or no useful learning signal. This repository adds hint-aware training on top of a standard SID-based generative recommendation stack.

• Reachability diagnosis checks whether the current model can reach the target SID branch under the rollout budget.
• Training-only hints expose a short target prefix for hard samples during RL.
• Hint-aware optimization separates hinted prefix tokens from sampled suffix tokens during the policy update.
• Constrained decoding keeps generation inside the valid SID set during evaluation.

Repository Overview

Path	Description
src/hcgrec/	HCGRec training, hint analysis, constrained evaluation, and utility modules
src/index/	embedding extraction, RQ-VAE index training, SID export, and index evaluation
src/rewards/	reward implementations for RL training
scripts/experiments/<Domain>/	paper-facing wrappers for data prep, SFT, and RL on Instruments, Games, and Arts
scripts/index/base/	base SID pipeline: text2emb, train, generate, and evaluate
scripts/eval/	single-checkpoint and multi-checkpoint evaluation helpers
scripts/data/	generic dataset conversion helpers
examples/train_full/	LLaMA-Factory YAML configs used by SFT launchers
config/	DeepSpeed configs from zero0 to zero3_offload

Quickstart

1. Environment setup

The supported install path is uv. This repo is a normal src/ layout package, and the installable Python package name is hcgrec.

uv venv --python 3.12
source .venv/bin/activate
uv sync

.env.example is only a template. Nothing in the current repo auto-loads it. If you want reusable local defaults, copy it to something like .env and source it yourself before launching the scripts.

2. Build a paper-facing dataset variant

If you already have raw category files and an index under data/Instruments/, the shortest path is the per-domain wrapper:

bash scripts/experiments/Instruments/prepare.sh check
bash scripts/experiments/Instruments/prepare.sh build-data

By default this writes data/Instruments_grec_index/ with:

• sft/train.json, sft/valid.json, sft/test.json
• rl/train.json, rl/valid.json, rl/test.json
• id2sid.json
• new_tokens.json

Use the corresponding wrappers under scripts/experiments/Games/ and scripts/experiments/Arts/ for those domains.

3. Launch SFT

bash scripts/experiments/Instruments/sft.sh --dry-run
bash scripts/experiments/Instruments/sft.sh

The default Instruments launcher points at examples/train_full/Instruments/instruments_rec_full_sft_3b_dsz3_qwen4b_4_256_grec_genrec_aligned_8gpu.yaml. To switch configs, override YAML_PATH.

4. Launch HCGRec RL

Run the preflight first:

python scripts/check_rl_env.py \
  --model-path /path/to/sft_checkpoint \
  --data-dir data/Instruments_grec_index/rl \
  --index-path data/Instruments_grec_index/id2sid.json \
  --output-dir rl_outputs/instruments_hcgrec_smoke

Then launch RL:

MODEL_PATH=/path/to/sft_checkpoint \
bash scripts/experiments/Instruments/rl_rule.sh --dry-run

MODEL_PATH=/path/to/sft_checkpoint \
bash scripts/experiments/Instruments/rl_rule.sh

Other RL variants live beside it; see the variant table below for what each script does.

5. Evaluate a checkpoint

bash scripts/eval/evaluate_checkpoint.sh \
  --checkpoint-path /path/to/checkpoint \
  --test-data-path data/Instruments_grec_index/sft/test.json \
  --index-path data/Instruments_grec_index/id2sid.json

For direct Python evaluation:

python -m hcgrec.evaluate \
  --model_name_or_path /path/to/checkpoint \
  --test_data_path data/Instruments_grec_index/sft/test.json \
  --index_path data/Instruments_grec_index/id2sid.json \
  --result_json_path temp/eval/instruments_result.json

Full Pipeline Walk-through

0. Prerequisites

• Python 3.11 to 3.13
• One or more CUDA GPUs for training
• A causal LLM checkpoint for SFT and RL
• An embedding model checkpoint for SID construction

1. Environment Setup

Use uv for the full workspace setup:

uv venv --python 3.12
source .venv/bin/activate
uv sync

Useful environment variables:

• HF_ENDPOINT if you use a Hugging Face mirror
• WANDB_API_KEY, WANDB_PROJECT, WANDB_MODE for experiment tracking
• HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1, HF_DATASETS_OFFLINE=1 for offline runs
• CUDA_VISIBLE_DEVICES, CUDA_LIST, RESULTS_ROOT, and TEMP_ROOT for launcher control

If you only want the external training CLI outside this repo, the equivalent standalone install is:

uv pip install "llamafactory==0.9.5"

2. Raw Data Filtering

The raw Amazon18-style preprocessing entrypoint is data/amazon18_data_process.py.

If your raw files follow the script defaults, run it inside data/:

cd data
python amazon18_data_process.py \
  --dataset Instruments \
  --user_k 5 \
  --item_k 5 \
  --st_year 1996 \
  --st_month 10 \
  --ed_year 2018 \
  --ed_month 10 \
  --output_path .

This mode expects files such as meta_Instruments.json and Instruments_5.json in the current directory.

If your raw filenames or locations differ, call the Python entrypoint from the repo root and pass them explicitly:

python data/amazon18_data_process.py \
  --dataset Instruments \
  --metadata_file /path/to/meta_Instruments.json \
  --reviews_file /path/to/Instruments_5.json \
  --output_path data

The script writes files such as:

• data/Instruments/Instruments.train.inter
• data/Instruments/Instruments.valid.inter
• data/Instruments/Instruments.test.inter
• data/Instruments/Instruments.item.json
• data/Instruments/Instruments.review.json
• data/Instruments/Instruments.inter.json

3. SID Construction

The base SID pipeline under scripts/index/base/ is:

1. text2emb.sh
2. train.sh
3. generate.sh
4. evaluate.sh

Extract item embeddings:

DATASETS="Instruments" \
PLM_NAME="qwen3-embedding-4B" \
MODEL_PATH="/path/to/embedding-model" \
NUM_PROCESSES=1 \
BATCH_SIZE=256 \
bash scripts/index/base/text2emb.sh

Train the index model:

USE_MULTI_DATASETS=false \
DATASET=Instruments \
MODEL_NAME=qwen3-embedding-4B \
NPROC_PER_NODE=1 \
BATCH_SIZE=2048 \
bash scripts/index/base/train.sh

Generate SID files from a trained checkpoint:

USE_MULTI_DATASETS=false \
DATASET=Instruments \
MODEL_NAME=qwen3-embedding-4B \
CKPT_PATH=/path/to/best_collision_model.pth \
bash scripts/index/base/generate.sh

Evaluate the index checkpoint:

CKPT_PATH=/path/to/best_collision_model.pth \
DEVICE=cuda:0 \
BATCH_SIZE=2048 \
bash scripts/index/base/evaluate.sh

4. Convert Raw Category Data into SFT and RL Datasets

For the current paper workflow, prefer the domain wrappers:

bash scripts/experiments/Instruments/prepare.sh check
bash scripts/experiments/Instruments/prepare.sh build-data

Equivalent wrappers exist for:

• scripts/experiments/Games/prepare.sh
• scripts/experiments/Arts/prepare.sh

These wrappers validate the raw category directory, pick the resolved index file, build the SFT and RL JSON files, and update data/dataset_info.json for LLaMA-Factory.

5. Supervised Fine-Tuning

The SFT wrappers are thin shells around scripts/experiments/_canonical_sft_launcher.sh.

bash scripts/experiments/Instruments/sft.sh --dry-run
bash scripts/experiments/Instruments/sft.sh

To switch domains:

• bash scripts/experiments/Games/sft.sh
• bash scripts/experiments/Arts/sft.sh

To switch the exact YAML, override YAML_PATH with a file from examples/train_full/.

6. Hint-Conditioned RL

The RL wrappers are thin shells around scripts/experiments/_canonical_rl_launcher.sh.

Start with a preflight:

python scripts/check_rl_env.py \
  --model-path /path/to/sft_checkpoint \
  --data-dir data/Instruments_grec_index/rl \
  --index-path data/Instruments_grec_index/id2sid.json \
  --output-dir rl_outputs/instruments_hcgrec_smoke

Then launch the paper-style rule-only run:

MODEL_PATH=/path/to/sft_checkpoint \
bash scripts/experiments/Instruments/rl_rule.sh --dry-run

MODEL_PATH=/path/to/sft_checkpoint \
bash scripts/experiments/Instruments/rl_rule.sh

RL variant scripts

Each domain wrapper directory ships a subset of these RL launchers. They differ only in the flags they pass to scripts/experiments/_canonical_rl_launcher.sh:

Script	Hint mode	Reward mode	Extra terms	Trainer class	Domains
rl_rule.sh	none	rule_only	—	trl.GRPOTrainer (vanilla GRPO)	all
rl_fixed.sh	fixed prefix, per-sample depth map	rule_only	—	FixedHintRuleOnlyGRPOTrainer	all
rl_fixed_ce.sh	fixed prefix, per-sample depth map	rule_only	prefix CE loss on hinted tokens (HINT_CE_LOSS_COEF, 0.005)	FixedHintRuleOnlyGRPOTrainer	all
rl_fixed_ce_dual.sh	fixed prefix, per-sample depth map	rule_only	prefix CE loss; trains on task1_sid_sft + task5_title_desc2sid, evaluates on task1_sid_sft	FixedHintRuleOnlyGRPOTrainer	Instruments only
rl_fixed_full_sequence_sft.sh	fixed prefix, per-sample depth map	rule_only	full-sequence SFT regularization (FULL_SEQUENCE_SFT_LOSS_COEF, 0.001)	FixedHintRuleOnlyGRPOTrainer	Instruments, Arts
rl_dynamic.sh	dynamic per-rollout hints (DYNAMIC_HINT_MAX_DEPTH, 3)	rule_only	—	DynamicHintRuleOnlyGRPOTrainer	Instruments, Arts
rl_ndcg.sh	none	ranking	—	trl.GRPOTrainer	all

Notes on how to read this:

• Fixed hint means an offline reachability diagnosis runs before training: hcgrec.analyze_rl_beam_hint beam-searches the SFT model and exports a per-sample hint-depth map, which the trainer then uses to reveal a target-SID prefix on hard samples. Diagnosis artifacts are cached under temp/rl_beam_hint/artifacts/; set FORCE_REANALYZE=true to rebuild them.
• Dynamic hint skips the offline pass and escalates the hint depth during rollout itself, up to DYNAMIC_HINT_MAX_DEPTH.
• hint_ce_loss_coef and full_sequence_sft_loss_coef are mutually exclusive (enforced in hcgrec.trl_trainer).

7. Evaluation

Use the Python module for direct constrained decoding:

python -m hcgrec.evaluate \
  --model_name_or_path /path/to/checkpoint \
  --test_data_path data/Instruments_grec_index/sft/test.json \
  --index_path data/Instruments_grec_index/id2sid.json \
  --result_json_path temp/eval/instruments_result.json

Use the shell wrapper for checkpoint-oriented evaluation:

bash scripts/eval/evaluate_checkpoint.sh \
  --checkpoint-path /path/to/checkpoint \
  --test-data-path data/Instruments_grec_index/sft/test.json \
  --index-path data/Instruments_grec_index/id2sid.json

For multi-checkpoint or watcher workflows, see:

• scripts/eval/evaluate_all_checkpoints.sh
• python -m hcgrec.evaluate_all_checkpoints_sidecar

8. Optional Ops Helpers

scripts/ops/ contains repo-local helpers for syncing results, uploader state, and evaluation maintenance. They are optional and not required for the main training pipeline.

Citation

@article{zhang2026hcgrec,
  title   = {Learning from Unreachable Rewards: Hint-Conditioned Reinforcement Learning for Generative Recommendation},
  author  = {Zhang, Kangning and Fang, Haotian and Luo, Xukun and Yin, Hao and Gao, Yang and Yan, Peng and Liu, Weiwen and Zhang, Weinan and Yu, Yong},
  journal = {arXiv preprint arXiv:2608.11980},
  year    = {2026}
}

Acknowledgements

• LLaMA-Factory for the SFT training stack
• TRL for GRPO and reward-model training utilities
• Open-source generative recommendation work that helped shape the SID-based training and evaluation workflow