HKUST (Guangzhou)
MPhil · Data Science and Analytics (expected)
MPhil student · HKUST (Guangzhou)
Jiangshan He何姜杉
I work on multimodal retrieval, generative models, and the evaluation of language models. Currently, I am exploring video-to-music generation.
Data Science and Analytics · Expected graduation: August 2027
jhe367@connect.hkust-gz.edu.cn
Education & Experience
HKUST (Guangzhou)
Research Assistant · LLM and multimodal evaluation
The Hong Kong Polytechnic University
Research Assistant · PolySmart Group
Projects
All projects ↗
Ongoing research
Video-to-music generation
Exploring how video content can guide music generation, with an interest in extending the approach to longer videos.
My workLeading baseline reproduction and experimental iteration for video-conditioned music generation.
1st · Both AVS tracks
TRECVID 2023 · Ad-hoc Video Search
Combining vision-language models, diffusion-generated image queries, and relevance feedback for ad-hoc video search.
My workLed the AVS project, contributing to the retrieval pipeline, experiments, and technical report.
VBS 2023 · Best Newcomer
QIVISE · Interactive Video Search
An interactive video search engine that brings multimodal embeddings and quantum-inspired relevance feedback into a working system.
My workContributed to interactive retrieval algorithms and system development; co-first author of the system paper.
Selected Publications
All publications ↗First- and co-first-author work
Paper Figure 1 · Matched-control evaluation protocol ↗
Beyond Direct Gains: Matched Controls for Evaluating Concept Scaffolds
Uses matched controls to separate the effects of concept content from prompt format.
Retrieval framework from the technical report ↗
WHU-NERCMS at TRECVID 2023: Ad-Hoc Video Search (AVS) and Deep Video Understanding (DVU) Tasks
Combines vision-language models, generated image queries, and relevance feedback for video search.
Interaction method from the official repository ↗
QIVISE: A Quantum-Inspired Interactive Video Search Engine in VBS2023
Refines multimodal video retrieval through user feedback on relevant and irrelevant clips.
Co-authored Papers
Data curation pipeline from the paper ↗
PhyAVBench: A Challenging Audio Physics-Sensitivity Benchmark for Physically Grounded Text-to-Audio-Video Generation
Evaluates whether generated sounds respond correctly to controlled changes in physical conditions.
Awards
1st Place · TRECVID 2023
Automatic & interactive AVS tracks · Team award
Best Newcomer · Video Browser Showdown
QIVISE · Team award
Meritorious Winner · Mathematical Contest in Modeling
Mathematical Contest in Modeling (MCM)