Yihuai Hong 洪奕槐

Yihuai Hong 洪奕槐

New York University

Hello, I am Yihuai Hong, a 2nd Year Ph.D. student in Computer Science at New York University (Courant Institute). My research interests lie in Natural Language Processing and Language Models.

My research story revolves around the Mechanistic Interpretability of LLMs, turning what we find inside models into action to make them more capable and safer:

  • Interpretability for Capability making models learn faster, reason better, and run more efficiently
    • Training & Architecture
      • Manifold Pretraining Engineering Preprint soon
      • LooM: Latent Looped Reasoning Model In progress
      • LLM Knowledge Distribution [NeurIPS 2025]
    • Post-hoc Intervention
  • Interpretability for Safety checking from the inside what models really know, compute, and do

Before my PhD career, I was very fortunate to work with Prof. Mor Geva Pipek from Tel Aviv University and Google Research on LLM Unlearning, and with Prof. Zhijing Jin from University of Toronto on Mechanistic understanding of Reasoning. I was also a researcher at Alibaba DAMO Academy working with Dr. Wenxuan Zhang and Dr. Yu Rong. More information in my CV. And I am actively looking for internship opportunities for 2027 Summer! Please let me know if you think I am a good fit!

🔥 News

  • 2026.09 🚀🚀 Please check my newest paper! Making LLMs Say What They Think: Measuring and Improving CoT-Interpretability Alignment We propose CIA, a metric that checks whether an LLM’s chain-of-thought matches the reasoning detected inside the model by interpretability tools, and show that post-training can substantially improve this alignment while maintaining or improving task accuracy!
  • 2025.09 🎉🎉 My first NeurIPS! One 1st-author paper has been accepted to NeurIPS 2025! “The Rise of Parameter Specialization for Knowledge Storage in Large Language Models”! And I received the NeurIPS 2025 Scholar Award! Thanks to the guidance from Prof. Wenxuan Zhang, Dr. Yu Rong, and the help of other collaborators! See you at San Diego!
  • 2025.08 🎉🎉 One 1st-author paper and one co-authored paper have been both accepted to EMNLP 2025 Main! “Intrinsic Test of Unlearning Using Parametric Knowledge Traces”! Thanks to the guidance from Prof. Mor Geva, Dr. Shauli Ravfogel, Dr. Lei Yu, and the help of other collaborators!
  • 2025.05 🎉🎉 My new 1st-author paper has been accepted to ACL 2025 Findings! “The Reasoning-Memorization Interplay in Language Models Is Mediated by a Single Direction”! Thanks to the guidance from Prof. Zhijing Jin, Dr. Lei Yu and the help of other collaborators. See you in Vienna!
  • 2024.09 🎉🎉 My two new 1st-author papers both have been accepted to EMNLP 2024 Main! “Dissecting Fine-Tuning Unlearning in Large Language Models” and “Interpretability-based Tailored Knowledge Editing in Transformers”! Feel so grateful to all my mentors and collaborators. See you in Miami!
  • 2024.06 🚀🚀 Please check my newest paper! Intrinsic Evaluation of Unlearning Using Parametric Knowledge Traces This is the first-ever parametric LLM Unlearning Benchmark! Thanks to the guidance from Prof. Mor Geva and the help of other collaborators.
  • 2023.12 🎉🎉 My first work is accepted to AAAI 2024 main track and I also won the AAAI-24 Student Scholarship! I am genuinely thankful to Prof. Zeng for guiding me and for her help along this path!

📝 Research

Overview of CoT-Interpretability Alignment (CIA)

Making LLMs Say What They Think: Measuring and Improving CoT-Interpretability Alignment

Yihuai Hong, Shauli Ravfogel, Chen Zhao†, Eunsol Choi†

Arxiv / GitHub

Preprint 2026

We propose CoT-Interpretability Alignment (CIA), which measures whether an LLM’s chain-of-thought matches the reasoning strategy that interpretability tools detect inside the model, and find limited alignment (44.8–75.9%) across three tasks and three LLMs. Using CIA as a post-training reward substantially improves this alignment while maintaining or improving task accuracy.

Parameter specialization overview

The Rise of Parameter Specialization for Knowledge Storage in Large Language Models

Yihuai Hong, Yiran Zhao, Wei Tang, Yu Rong, Wenxuan Zhang

Arxiv / GitHub

NeurIPS 2025

Analyzing 20 open-source LLMs, we find that stronger models store similar knowledge in more specialized MLP parameters, and causal training experiments show that this specialization helps models use their stored knowledge more efficiently.

ConceptVectors benchmark overview

Intrinsic Test of Unlearning Using Parametric Knowledge Traces

Yihuai Hong, Lei Yu, Haiqin Yang, Shauli Ravfogel, Mor Geva

Website / Arxiv / GitHub / Huggingface / Twitter

EMNLP 2025 Main

Current unlearning methods change model behavior without erasing the knowledge stored in the parameters. Our ConceptVectors benchmark (285 concept vectors in two open-source LLMs) tests unlearning at the parameter level, and ablating these vectors removes the knowledge and makes models less vulnerable to adversarial attacks.

Linear reasoning features overview

The Reasoning-Memorization Interplay in Language Models Is Mediated by a Single Direction

Yihuai Hong, Dian Zhou, Meng Cao, Lei Yu, Zhijing Jin

Arxiv / GitHub

ACL 2025 Findings

We identify linear features in the residual stream that govern whether an LLM reasons or recalls from memory, and show that intervening on them helps the model use the right problem-solving ability.

Dissecting fine-tuning unlearning overview

Dissecting Fine-Tuning Unlearning in Large Language Models

Yihuai Hong, Yuelin Zou, Lijie Hu, Ziqian Zeng, Di Wang, Haiqin Yang

Arxiv / GitHub

EMNLP 2024 Main Oral

Using activation patching and parameter restoration, we show that fine-tuning-based unlearning changes how models retrieve knowledge rather than erasing it, with MLP coefficients in the final layer driving most of the apparent forgetting.

TailoredKE overview

Interpretability-based Tailored Knowledge Editing in Transformers

Yihuai Hong, Aldo Lipani

Paper / Github

EMNLP 2024 Main

Interpretability shows that different samples recall their attributes at different layers, so TailoredKE edits each sample at its own depth, which mitigates over-editing.

ConsistentEE overview

ConsistentEE: A Consistent and Hardness-Guided Early Exiting Method for Accelerating Language Models Inference

Ziqian Zeng*, Yihuai Hong*, Huiping Zhuang, Cen Chen, HongLiang Dai

Arxiv / GitHub

AAAI 2024 Main Track

We formulate early exiting as a reinforcement learning problem so that training and inference stay consistent, and use a per-instance hardness measure (Memorized Layer) in the reward to balance accuracy and speed.

📖 Educations

  • 2025.09 - present, Ph.D. student in Computer Science, Courant Institute, New York University
  • 2020.09 - 2024.06, Bachelor of Engineering, School of Computer Science and Engineering, South China University of Technology

💻 Internships

🎖 Honors and Awards

  • 2025.10 NeurIPS 2025 Scholar Award
  • 2023.12 AAAI-24 Student Scholarship
  • 2023.11 Top Ten Excellent Students Nomination Award of South China University of Technology
  • 2023.09 China National Scholarship (top 0.1%)
  • 2023.05 Meritorious Winner of The Mathematical Contest in Modeling (MCM)
  • 2022.03 Kaggle Bronze medal (Top 6%) - Evaluating Student Writing: Analyze argumentative writing elements from students grades 6-12
  • 2021.07 Kaggle Silver medal (Top 5%) - CommonLit Readability Prize: Rate the complexity of literary passages for grades 3-12 classroom use

🙋‍♂️ Academic Services

  • Program Committee: ICLR (2025, 2026), NeurIPS (2025, 2026), ICML (2026), ACL ARR (Feb. 2024 - Aug. 2026)

💬 Teaching

  • Section Leader: Fall 2026 - DS-GA 1011 Fundamentals of Natural Language Processing - NYU Center for Data Science
  • Section Leader: Fall 2026 - DS-GA 1006 Capstone Project and Presentation - NYU Center for Data Science
  • Section Leader: Spring 2026 - DS-GA 1003 Machine Learning - NYU Center for Data Science
  • Section Leader: Fall 2025 - DS-GA 1006 Capstone Project and Presentation - NYU Center for Data Science