Philippians 4:13
Learning to Reason without External Rewards 본문
* 요약
"Learning to Reason without External Rewards" 논문은 외부 보상 없이 대규모 언어 모델(LLM)의 추론 능력을 향상시키는 혁신적인 방법을 제시합니다. 이 연구는 Reinforcement Learning from Internal Feedback(RLIF) 프레임워크와 Intuitor 방법론을 중심으로 진행되었습니다.
기존 방법의 한계
- RLHF(Reinforcement Learning from Human Feedback): 인간 평가에 의존하여 비용이 높고 편향 가능성이 존재
- RLVR(Reinforcement Learning with Verifiable Rewards): 도메인별 검증 가능한 보상(예: 수학 정답, 코드 테스트 케이스)이 필요하여 확장성에 제약
제안 방법: RLIF와 Intuitor
- 내부 피드백 활용: 외부 보상 대신 모델의 **자신감(self-certainty)**을 보상 신호로 사용
- Intuitor 구현:
- Group Relative Policy Optimization(GRPO)의 외부 보상을 self-certainty 점수로 대체
- 수학 문제 해결 시 모델의 확신도를 토큰 확률 분포의 엔트로피로 측정
- 코드 생성 등 도메인 외 작업에서도 금표 솔루션 없이 일반화 가능
실험 결과
| 수학 문제(MATH) | GRPO 대비 동등 성능 | 코드 생성 작업에서 우수 |
| 계산 효율성 | 1.3배 향상 | 학습 데이터 10%만 사용 |
| 구조적 추론 | 89% 정확도 향상 | 복잡한 문제 체계적 해결 |
의의 및 활용 방안
- 검증 데이터 부족 분야 적용 가능: 의료 진단, 과학 연구 등 금표 답안 확보 어려운 분야 활용 전망
- 자율 AI 시스템 발전: 인간 개입 없이 지속적 자기 개선 가능한 시스템 기반 마련
이 연구는 arXiv(2505.19590)를 통해 공개되었으며, GitHub 저장소에서 코드를 확인할 수 있습니다
Abstract
Training large language models (LLMs) for complex reasoning via Reinforcement
Learning with Verifiable Rewards (RLVR) is effective but limited by reliance on
costly, domain-specific supervision. We explore Reinforcement Learning from
Internal Feedback (RLIF), a framework that enables LLMs to learn from intrinsic
signals without external rewards or labeled data. We propose INTUITOR, an
RLIF method that uses a model’s own confidence—termed self-certainty—as
its sole reward signal. INTUITOR replaces external rewards in Group Relative
Policy Optimization (GRPO) with self-certainty scores, enabling fully unsupervised
learning. Experiments demonstrate that INTUITOR matches GRPO’s performance
on mathematical benchmarks while achieving superior generalization to out-ofdomain
tasks like code generation, without requiring gold solutions or test cases.
Our findings show that intrinsic model signals can drive effective learning across
domains, offering a scalable alternative to RLVR for autonomous AI systems where
verifiable rewards are unavailable. Code is available at https://github.com/
sunblaze-ucb/Intuitor.
복잡한 추론을 위한 대규모 언어 모델(LLM)을 검증 가능한 보상을 이용한 강화 학습(RLVR)을 통해 학습하는 것은 효과적이지만, 비용이 많이 들고 특정 영역에만 국한된 지도 학습에 의존한다는 한계가 있습니다. 본 연구에서는 LLM이 외부 보상이나 레이블이 지정된 데이터 없이 내재적 신호를 통해 학습할 수 있도록 하는 프레임워크인 내부 피드백 기반 강화 학습(RLIF)을 탐구합니다. 본 연구에서는 모델 자체의 신뢰도(자기 확실성이라고 함)를 유일한 보상 신호로 사용하는 RLIF 방법인 INTUITOR를 제안합니다. INTUITOR는 그룹 상대 정책 최적화(GRPO)의 외부 보상을 자기 확실성 점수로 대체하여 완전한 비지도 학습을 가능하게 합니다. 실험 결과, INTUITOR는 수학적 벤치마크에서 GRPO의 성능과 동등하며, 골드 솔루션이나 테스트 케이스 없이도 코드 생성과 같은 영역 외 작업보다 우수한 일반화를 달성하는 것으로 나타났습니다. 연구 결과에 따르면, 내재적 모델 신호는 여러 영역에서 효과적인 학습을 유도할 수 있으며, 검증 가능한 보상이 제공되지 않는 자율 AI 시스템에서 RLVR에 대한 확장 가능한 대안을 제공합니다. 코드는 https://github.com/sunblaze-ucb/Intuitor에서 확인할 수 있습니다.
Figure 1: Overview of RLIF and INTUITOR’s Performance. Left: Illustration of RLIF, a paradigm
where LLMs learn from intrinsic signals generated by the model itself, without external supervision.
Right: Performance comparison of Qwen2.5-3B Base, GRPO, and INTUITOR (our RLIF
instantiation). Both GRPO and INTUITOR are trained on the MATH dataset. INTUITOR achieves
comparable performance to GRPO on in-domain mathematical benchmarks (GSM8K, MATH500)
and demonstrates better generalization to out-of-domain code generation tasks (LiveCodeBench v6,
CRUXEval). Part of the illustration was generated by GPT-4o.
그림 1: RLIF와 INTUITOR 성능 개요. 왼쪽: RLIF 패러다임의 예시. LLM은 외부 감독 없이 모델 자체에서 생성된 고유 신호로부터 학습합니다.
오른쪽: Qwen2.5-3B Base, GRPO, INTUITOR(RLIF 인스턴스화)의 성능 비교. GRPO와 INTUITOR 모두 MATH 데이터셋을 사용하여 학습합니다. INTUITOR는 도메인 내 수학 벤치마크(GSM8K, MATH500)에서 GRPO와 유사한 성능을 달성하며, 도메인 외 코드 생성 작업(LiveCodeBench v6, CRUXEval)에 대한 일반화 성능이 더 우수함을 보여줍니다.
이 예시의 일부는 GPT-4o를 사용하여 생성되었습니다.
1 Introduction
Reinforcement learning has become essential for enhancing large language model capabilities. Early
work focused on Reinforcement Learning from Human Feedback (RLHF), which aligns model outputs
with human values through reward models trained on preference data [Ouyang et al., 2022]. Recent
advances in Reinforcement Learning with Verifiable Rewards (RLVR) replace learned reward models
with automatically verifiable signals—such as exact answer matching in mathematical problemsolving—
demonstrating improved reasoning capabilities in models like DeepSeek-R1 [Guo et al.,
2025, Lambert et al., 2024, Hu et al., 2025].
Despite these successes, both RLHF and RLVR face fundamental limitations that constrain their
broader applicability. RLHF requires extensive human annotation, making it expensive and potentially
biased [Gao et al., 2023]. RLVR, while avoiding learned reward models, demands domain-specific
verifiers and gold-standard solutions. In mathematics, this requires expert annotation of solutions; in
code generation, it necessitates comprehensive test suites and execution environments [Liu et al., 2023,
Liu and Zhang, 2025, Team et al., 2025, Xiaomi LLM-Core Team, 2025]. These requirements limit
RLVR to carefully curated domains and complicate deployment in open-ended scenarios. Moreover,
outcome-oriented verifiable rewards limit transferability to other domains. These challenges motivate
exploration of more general and scalable reward paradigms, leading to a critical research question:
Can LLMs enhance their reasoning abilities by relying solely on intrinsic, self-generated signals,
without recourse to external verifiers or domain-specific ground truth?
In this paper, we introduce and explore such a paradigm: Reinforcement Learning from Internal
Feedback (RLIF), where models optimize intrinsic feedback to improve performance without external
rewards or supervision. The motivation for RLIF extends to future scenarios where models develop
superhuman capabilities that become difficult for humans to evaluate directly [Burns et al., 2023],
requiring self-improvement through intrinsic mechanisms [Oudeyer and Kaplan, 2007].
Under the RLIF paradigm, we propose INTUITOR, a novel reinforcement learning approach leveraging
a model’s own confidence as an intrinsic reward. This builds on observations that LLMs exhibit
lower confidence on difficult problems [Farquhar et al., 2024, Kuhn et al., 2023, Kang et al., 2024,
2025]; optimizing for confidence should improve reasoning capabilities. Specifically, we use selfcertainty
[Kang et al., 2025], the average KL divergence between the model’s output distribution and
a uniform distribution, as our confidence measure. This metric has proven useful for distinguishing
high-quality responses from flawed ones [Kang et al., 2025, Ma et al., 2025]. Building on this
insight, INTUITOR guides learning through self-generated signals, eliminating the need for external
supervision or handcrafted rewards. The implementation of INTUITOR is simple, efficient, and
effective: we replace the verifiable reward signal in existing RLVR frameworks, specifically Group
Relative Policy Optimization (GRPO) [Shao et al., 2024], with self-certainty scores, using the same
policy gradient algorithm.
Our experiments demonstrate promising results. On the MATH dataset [Hendrycks et al., 2021]
with Qwen2.5-3B base [Yang et al., 2024a], INTUITOR matches the performance of GRPO without
relying on any gold answers. As INTUITOR rewards the generation trajectory rather than only the
end result, it generalizes more effectively: training a Qwen2.5-3B base model on MATH yields a
65% relative improvement on LiveCodeBench Code generation task [Jain et al., 2024] versus no
improvement for GRPO, and a 76% gain on CRUXEval-O [Gu et al., 2024] compared with 44%
for GRPO. Additionally, when we fine-tune the Qwen2.5-1.5B base model with INTUITOR on the
MATH corpus, a model that originally produces repetitive content and scores 0% on LiveCodeBench
learns to emit coherent reasoning chains and well-structured code, reaching 9.9% accuracy after the
tuning. This demonstrates the strong generalization capabilities of INTUITOR. As INTUITOR requires
only a clear prompt and no verifiable reward, it is broadly applicable across tasks, providing fresh
evidence that pretrained LLMs possess richer latent behavioral priors than previously recognized.
Our contributions can be summarized as follows:
• We introduce and explore Reinforcement Learning from Internal Feedback (RLIF), a novel reinforcement
learning paradigm enabling LLMs to improve reasoning skills by leveraging intrinsic,
self-generated signals, without reliance on external supervision or labeled data.
• We introduce INTUITOR, an RLIF-based method that utilizes a model’s own internal confidence
measure—termed self-certainty—as the sole intrinsic reward.
• We demonstrate that INTUITOR matches supervised RL performance on in-domain tasks while
achieving superior out-of-domain generalization. We uncover emergent structured reasoning and
enhanced instruction-following capabilities induced by intrinsic rewards.
1 서론
강화 학습은 대규모 언어 모델 성능 향상에 필수적인 요소가 되었습니다. 초기 연구는 선호도 데이터로 학습된 보상 모델을 통해 모델 출력을 인간의 가치에 맞춰 조정하는 인간 피드백 기반 강화 학습(RLHF)에 중점을 두었습니다[Ouyang et al., 2022]. 최근 검증 가능한 보상을 이용한 강화 학습(RLVR)의 발전은 학습된 보상 모델을 수학 문제 해결에서의 정확한 정답 매칭과 같은 자동 검증 가능한 신호로 대체하여 DeepSeek-R1과 같은 모델에서 향상된 추론 능력을 보여줍니다[Guo et al., 2025, Lambert et al., 2024, Hu et al., 2025].
이러한 성공에도 불구하고, RLHF와 RLVR 모두 광범위한 적용 가능성을 제한하는 근본적인 한계에 직면해 있습니다. RLHF는 광범위한 인간 주석이 필요하기 때문에 비용이 많이 들고 잠재적으로 편향될 수 있습니다[Gao et al., 2023]. RLVR은 학습된 보상 모델을 피하는 동시에 도메인별 검증 도구와 표준 솔루션을 요구합니다. 수학에서는 솔루션에 대한 전문가 주석이 필요하고, 코드 생성에서는 포괄적인 테스트 스위트와 실행 환경이 필요합니다[Liu et al., 2023,
Liu and Zhang, 2025, Team et al., 2025, Xiaomi LLM-Core Team, 2025]. 이러한 요구 사항으로 인해 RLVR은 신중하게 선별된 도메인으로 제한되고, 개방형 시나리오에서의 배포가 복잡해집니다. 더욱이, 결과 지향적인 검증 가능한 보상은 다른 도메인으로의 이전 가능성을 제한합니다. 이러한 과제는 더욱 일반적이고 확장 가능한 보상 패러다임의 탐색을 촉진하며, 중요한 연구 질문으로 이어집니다.
LLM은 외부 검증 도구나 도메인별 실제 값에 의존하지 않고, 내재적이고 자체 생성된 신호에만 의존하여 추론 능력을 향상시킬 수 있을까요?
본 논문에서는 내부 피드백을 활용한 강화 학습(RLIF) 패러다임을 소개하고 탐구합니다. RLIF는 모델이 외부 보상이나 감독 없이 내재적 피드백을 최적화하여 성과를 향상시키는 패러다임입니다. RLIF의 동기는 모델이 인간이 직접 평가하기 어려운 초인적 능력을 개발하는 미래 시나리오[Burns et al., 2023]로 확장되어 내재적 메커니즘을 통한 자기 개선이 필요합니다[Oudeyer and Kaplan, 2007]. RLIF 패러다임 하에서, 모델 자체의 신뢰도를 내재적 보상으로 활용하는 새로운 강화 학습 접근법인 INTUITOR를 제안합니다. 이는 LLM이 어려운 문제에 대해 낮은 신뢰도를 보인다는 관찰 결과를 바탕으로 합니다[Farquhar et al., 2024, Kuhn et al., 2023, Kang et al., 2024, 2025]. 신뢰도 최적화는 추론 능력을 향상시킬 것입니다. 구체적으로, 본 연구에서는 모델의 출력 분포와 균일 분포 간의 평균 KL 발산(KL divergence)인 자기확실성(self-certainty)[Kang et al., 2025]을 신뢰도 척도로 사용합니다. 이 지표는 고품질 응답과 결함이 있는 응답을 구분하는 데 유용한 것으로 입증되었습니다[Kang et al., 2025, Ma et al., 2025]. 이러한 통찰력을 바탕으로, INTUITOR는 자가 생성 신호를 통해 학습을 유도하여 외부 감독이나 직접 보상이 필요하지 않습니다. INTUITOR 구현은 간단하고 효율적이며 효과적입니다. 기존 RLVR 프레임워크, 특히 그룹 상대 정책 최적화(GRPO)[Shao et al., 2024]의 검증 가능한 보상 신호를 동일한 정책 기울기 알고리즘을 사용하여 자기확실성 점수로 대체합니다. 본 실험은 유망한 결과를 보여줍니다. MATH 데이터셋[Hendrycks et al., 2021]과 Qwen2.5-3B 기반[Yang et al., 2024a]을 사용하여 INTUITOR는 Gold 정답에 의존하지 않고도 GRPO의 성능을 달성했습니다. INTUITOR는 최종 결과만이 아닌 생성 궤적을 보상하기 때문에 더욱 효과적으로 일반화합니다. MATH에서 Qwen2.5-3B 기반 모델을 학습한 결과, LiveCodeBench 코드 생성 작업[Jain et al., 2024]에서 65%의 상대적 향상을 보였으며, GRPO에서는 전혀 향상되지 않았습니다. 또한 CRUXEval-O[Gu et al., 2024]에서는 76%의 향상을 보였으며, GRPO에서는 44%의 향상을 보였습니다. 또한, MATH 코퍼스에서 INTUITOR를 사용하여 Qwen2.5-1.5B 기본 모델을 미세 조정했을 때, 원래 반복적인 콘텐츠를 생성하고 LiveCodeBench에서 0%를 기록했던 모델이 일관된 추론 체인과 잘 구조화된 코드를 생성하는 것을 학습하여 조정 후 9.9%의 정확도를 달성했습니다. 이는 INTUITOR의 강력한 일반화 능력을 보여줍니다. INTUITOR는 명확한 프롬프트만 필요로 하고 검증 가능한 보상은 필요하지 않기 때문에 모든 작업에 광범위하게 적용 가능하며, 사전 훈련된 LLM이 이전에 알려진 것보다 더 풍부한 잠재 행동 사전 정보를 가지고 있다는 새로운 증거를 제공합니다.
저희의 기여는 다음과 같이 요약할 수 있습니다.
• LLM이 외부 감독이나 레이블이 지정된 데이터에 의존하지 않고 내재적이고 자체 생성된 신호를 활용하여 추론 능력을 향상시킬 수 있도록 하는 새로운 강화 학습 패러다임인 내부 피드백을 통한 강화 학습(RLIF)을 소개하고 탐구합니다.
• RLIF 기반 방법인 INTUITOR를 소개합니다. 이 방법은 모델 자체의 내부 신뢰도 측정값(자기 확신도라고 함)을 유일한 내재적 보상으로 활용합니다.
• INTUITOR가 도메인 내 과제에서는 지도 학습 기반 강화 학습(RL)과 동등한 성능을 보이는 동시에
도메인 외 과제에서도 우수한 일반화를 달성함을 보여줍니다. 또한, 내재적 보상에 의해 유도되는 창발적 구조적 추론과
향상된 지시 수행 능력을 발견합니다.
2 Related Work
Reinforcement Learning from Human Feedback (RLHF). RL has become instrumental in
refining LLMs. Early pivotal work centered on Reinforcement Learning from Human Feedback
(RLHF) [Ouyang et al., 2022], which aligns LLMs with human values by training a reward model
on human preference data. While effective, RLHF is often resource-intensive due to the need for
extensive human annotation [Touvron et al., 2023]. Subsequent innovations like Direct Preference
Optimization (DPO) [Rafailov et al., 2023] aimed to simplify this by directly training models on
preferences. The reliance on human-generated or model-approximated human preferences poses
scalability challenges and introduces potential biases from the reward model itself [Gao et al., 2023].
Reinforcement Learning with Verifiable Rewards (RLVR). RLVR emerged as a powerful
alternative, particularly for tasks with clear correctness criteria like mathematical reasoning and code
generation [Guo et al., 2025, Lambert et al., 2024, Hu et al., 2025, Team et al., 2025, Xiaomi LLMCore
Team, 2025]. RLVR utilizes rule-based verification functions, such as exact answer matching
[Guo et al., 2025, Team et al., 2025, Xiaomi LLM-Core Team, 2025, Jaech et al., 2024], to provide
reward signals, thereby avoiding the complexities and potential pitfalls of learned reward models.
This approach has sparked significant advances, with models like DeepSeek-R1 [Guo et al., 2025]
achieving state-of-the-art reasoning capabilities. The development of robust policy optimization
algorithms like GRPO [Shao et al., 2024] and its variants [Luo et al., 2025, Liu et al., 2025] has
further solidified RLVR’s success. Nevertheless, RLVR’s applicability is largely confined to domains
where verifiable gold solutions or exhaustive test cases can be constructed, and its predominant focus
on outcome-based rewards can limit generalization to dissimilar tasks or those requiring nuanced,
process-oriented feedback.
Intrinsic Signals and Self-Play in Language Model Optimization. Self-play and intrinsic rewards
have gained attention as strategies for enabling autonomous model improvement. Inspired by early
work in games like AlphaGo Zero [Silver et al., 2017], recent LLM-based frameworks incorporate
self-refinement mechanisms to bootstrap reasoning ability. Methods like SPIN [Chen et al., 2024] and
Self-Rewarding LMs [Yuan et al., 2024] utilize the model itself to provide feedback for subsequent
training iterations. While earlier work such as STaR [Zelikman et al., 2022] relies on repeated outcome
evaluation, more recent approaches explore self-improvement through procedural generalization
and goal invention [Poesia et al., 2024, Cheng et al., 2024]. Concurrent works like Genius, TTRL,
and Absolute Zero [Xu et al., 2025, Zuo et al., 2025, Zhao et al., 2025]—leverage queries without
labels for reinforcement learning but remain constrained to specific task distributions, primarily in
mathematical reasoning. INTUITOR aligns with this direction but introduces a lightweight, generalpurpose
alternative: using self-certainty as a confidence-based intrinsic reward. Unlike prior work,
INTUITOR enables single-agent reinforcement learning across diverse tasks without relying on explicit
feedback, gold labels, or environment-based validation.
2 관련 연구
인간 피드백을 이용한 강화 학습(RLHF). 강화 학습은 LLM 개선에 중요한 역할을 해왔습니다. 초기 핵심 연구는 인간 피드백을 이용한 강화 학습(RLHF)[Ouyang et al., 2022]에 집중되었는데, 이는 인간의 선호도 데이터를 기반으로 보상 모델을 학습하여 LLM을 인간의 가치에 맞춰 조정하는 것입니다. RLHF는 효과적이지만, 광범위한 인간 주석이 필요하기 때문에 종종 리소스 집약적입니다[Touvron et al., 2023]. 직접 선호 최적화(DPO)[Rafailov et al., 2023]와 같은 후속 혁신은 선호도에 따라 모델을 직접 학습하여 이를 단순화하는 것을 목표로 했습니다. 인간이 생성하거나 모델에서 근사한 인간 선호도에 의존하는 것은 확장성 문제를 야기하고 보상 모델 자체의 잠재적 편향을 야기합니다[Gao et al., 2023].
검증 가능한 보상을 이용한 강화 학습(RLVR). RLVR은 특히 수학적 추론 및 코드 생성과 같이 정확성 기준이 명확한 작업에 대해 강력한 대안으로 부상했습니다[Guo et al., 2025, Lambert et al., 2024, Hu et al., 2025, Team et al., 2025, Xiaomi LLM-Core Team, 2025, Jaech et al., 2024]. RLVR은 정확한 정답 매칭과 같은 규칙 기반 검증 기능을 활용하여 보상 신호를 제공함으로써 학습된 보상 모델의 복잡성과 잠재적인 위험을 회피합니다. 이러한 접근 방식은 DeepSeek-R1[Guo et al., 2025]과 같은 모델이 최첨단 추론 기능을 달성하는 등 상당한 발전을 가져왔습니다. GRPO[Shao et al., 2024]와 그 변형[Luo et al., 2025, Liu et al., 2025]과 같은 강력한 정책 최적화 알고리즘의 개발은 RLVR의 성공을 더욱 공고히 했습니다. 그럼에도 불구하고 RLVR의 적용 가능성은 검증 가능한 골드 솔루션이나 포괄적인 테스트 케이스를 구축할 수 있는 영역에 주로 국한되며, 결과 기반 보상에 주로 초점을 맞추는 것은 일반화를 서로 다른 작업이나 미묘한 차이와 프로세스 지향적 피드백이 필요한 작업으로 제한할 수 있습니다. 언어 모델 최적화에서의 내재적 신호와 자기 플레이. 자기 플레이와 내재적 보상은 자율적인 모델 개선을 가능하게 하는 전략으로 주목을 받고 있습니다. AlphaGo Zero[Silver et al., 2017]와 같은 게임의 초기 연구에서 영감을 받은 최근 LLM 기반 프레임워크는 추론 능력을 부트스트랩하기 위한 자기 개선 메커니즘을 통합합니다. SPIN[Chen et al., 2024] 및 자가 보상 LM[Yuan et al., 2024]과 같은 방법은 모델 자체를 활용하여 후속 학습 반복에 대한 피드백을 제공합니다. STaR[Zelikman et al., 2022]과 같은 초기 연구는 반복적인 결과 평가에 의존하는 반면, 최근의 접근 방식은 절차적 일반화 및 목표 고안[Poesia et al., 2024, Cheng et al., 2024]을 통한 자기 개선을 탐구합니다. Genius, TTRL, Absolute Zero[Xu et al., 2025, Zuo et al., 2025, Zhao et al., 2025]와 같은 동시 연구들은 강화 학습을 위해 레이블이 없는 쿼리를 활용하지만, 주로 수학적 추론과 같은 특정 작업 분포에 제한됩니다. INTUITOR는 이러한 방향에 부합하지만, 가볍고 범용적인 대안을 제시합니다. 바로 자기 확신을 신뢰 기반 내재적 보상으로 활용하는 것입니다. 이전 연구와 달리, INTUITOR는 명시적인 피드백, 골드 라벨 또는 환경 기반 검증에 의존하지 않고도 다양한 과제에 걸쳐 단일 에이전트 강화 학습을 가능하게 합니다.
3 Method
In this section, we present our approach to training large language models through intrinsic feedback
mechanisms. We begin by reviewing existing RL-based fine-tuning paradigms and their limitations,
which motivate our exploration of Reinforcement Learning from Internal Feedback (RLIF). We then
introduce INTUITOR, our novel RLIF method that leverages self-certainty as an intrinsic reward
signal, and detail its implementation through policy optimization.
3.1 From External Supervision to Internal Feedback
Current RL fine-tuning approaches for LLMs primarily fall into two categories: those relying on
external human feedback (RLHF) and those using verifiable, often task-specific, rewards (RLVR).
In RLHF [Ziegler et al., 2019, Ouyang et al., 2022], the policy πθ is optimized to align with human
preferences, typically encapsulated by a learned reward model rϕ. The objective is:
where q is an input query, o is the generated output, πref is an initial reference policy, and β is
a coefficient controlling the KL divergence to prevent excessive deviation from πref. Online RL
algorithms like PPO [Schulman et al., 2017] generate samples from πθ, evaluate them using rϕ, and
update πθ to maximize this objective. However, the reward model rϕ is crucial yet fragile; introducing
it can lead to “reward hacking,” and retraining it is resource-intensive, complicating the training
pipeline [Gao et al., 2023].
RLVR, on the other hand, substitutes the learned reward model with an automatically verifiable
signal. This has proven effective in promoting reasoning capabilities, especially in domains like
mathematics [Guo et al., 2025]. The RLVR objective is:
where v(q, o) is a verifiable reward function. For instance, in mathematical problem-solving, v(q, o)
REINFORCE [Williams, 1992], PPO or GRPO. Despite their simplicity, verifiable rewards still rely
on gold-standard answers or test executions, which are costly and domain-specific [Liu et al., 2025,
Team et al., 2025]. RLVR faces challenges in extending beyond math and code to tasks involving
ambiguity or subjective reasoning.
3 방법
이 섹션에서는 내재적 피드백 메커니즘을 통해 대규모 언어 모델을 학습하는 접근 방식을 제시합니다. 먼저 기존의 강화 학습 기반 미세 조정 패러다임과 그 한계점을 검토하여 내부 피드백을 통한 강화 학습(RLIF)을 탐구합니다. 그런 다음, 자기 확실성을 내재적 보상 신호로 활용하는 새로운 RLIF 방법인 INTUITOR를 소개하고 정책 최적화를 통한 구현 방법을 자세히 설명합니다.
3.1 외부 감독에서 내부 피드백으로
현재 LLM을 위한 강화 학습 미세 조정 접근 방식은 크게 두 가지 범주로 나뉩니다. 외부 인간 피드백(RLHF)에 의존하는 방식과 검증 가능하고 종종 작업별 보상(RLVR)을 사용하는 방식입니다.
3.2 Reinforcement Learning from Internal Feedback (RLIF)
To overcome the limitations of RLHF’s costly human annotation and RLVR’s domain-specific supervision,
we propose Reinforcement Learning from Internal Feedback (RLIF). Instead of depending
on external evaluation, RLIF uses the model’s own assessment of its outputs or reasoning process
as feedback. This offers several advantages: it reduces reliance on supervision infrastructure, provides
task-agnostic reward signals, and supports learning in domains where external verification is
unavailable. Under the RLIF paradigm, the optimization objective becomes:
where u(q, o) represents an intrinsic signal derived from the model’s internal state or computation,
rather than external verification. The key challenge lies in identifying intrinsic signals that correlate
with output quality and can effectively guide learning.
Concurrent research explores related concepts within the RLIF paradigm. For example, Entropy
Minimized Policy Optimization (EMPO) [Zhang et al., 2025] minimizes LLM predictive entropy on
unlabeled questions in a latent semantic space. SEED-GRPO [Chen et al., 2025] uses the semantic
entropy of generated sequences, combined with ground truth rewards, to modulate policy updates.
Reinforcement Learning with a Negative Entropy Reward (EM-RL) [Agarwal et al., 2025] employs
a reward signal based solely on the negative sum of token-level entropy, akin to REINFORCE but
without labels. These methods highlight the growing interest and potential of leveraging intrinsic
signals for LLM training under the RLIF framework.
3.3 INTUITOR: Policy Optimization with Self-Certainty
We propose INTUITOR, a novel RLIF method that utilizes a model’s own confidence as the sole
intrinsic reward signal u(q, o).
Our choice of model confidence as the intrinsic reward is motivated by observations that LLMs often
exhibit lower confidence when encountering unfamiliar tasks or lacking sufficient knowledge [Kang
et al., 2024]. Conversely, higher confidence frequently correlates with correctness. By rewarding
increased self-confidence, INTUITOR encourages to iteratively "practice" and refine its reasoning
pathways until it becomes more confident in its outputs.
Figure 2: Illustration of INTUITOR. INTUITOR simplifies the training strategy by leveraging selfcertainty
(the model’s own confidence) as an intrinsic reward, optimizing these scores to incentivize
reasoning abilities without external supervision.
We adopt the self-certainty metric from Kang et al. [2025], defined as the average KL divergence
between a uniform distribution U over the vocabulary V and the model’s next-token distribution:
where o<i are the previously generated tokens and p(j|q, o<i) is the model’s predicted probability
for token j at step i. Higher self-certainty values indicate greater confidence.
Self-certainty, being related to a KL divergence where the model’s prediction is the second argument
KL(U ∥ pπθ ), is mode-seeking. This contrasts with entropy (or reverse KL divergence from
uniform), which is mode-covering. Critically, self-certainty is reported to be less prone to biases
towards longer generations, a common issue with perplexity or entropy-based measures [Fang et al.,
2024, Kang et al., 2025], making it a potentially more reliable indicator of intrinsic confidence.
Kang et al. [2025] demonstrate that self-certainty is effective for selecting high-quality answers from
multiple candidates, and uniquely among different confidence measures, its utility improves with
more candidates. Optimizing for self-certainty thus encourages the model to generate responses that
it deems more convincing. The RL process can achieve this by, for instance, guiding the model to
produce more detailed reasoning steps, thereby increasing the model’s conviction in its final answer.
This mechanism is more nuanced than simply increasing the probability of the most likely output; it
involves modifying the generation process itself to build confidence.
To optimize the objective in Equation 3, various policy gradient algorithms can be employed. Informed
by the recent success in models such as DeepSeek-R1 [Guo et al., 2025] and its widespread adoption
of GRPO in open-source projects, we utilize GRPO to optimize for self-certainty. The overall pipeline
for this GRPO-based instantiation of INTUITOR is illustrated in Figure 2.
The core idea behind the optimization is to sample multiple candidate outputs for a given query and
use their relative rewards to estimate advantages for policy updates. For each query q ∼ P(Q), GRPO
samples a group of G outputs o1, . . . , oG using a behavior policy πθold (e.g., a previous iteration or
the SFT model). The target policy πθ is then optimized by maximizing:
Hyperparameters ϵ (for clipping) and β (for KL penalty strength) control stability and exploration,
and Aˆi,t is the advantage estimate.
Integration of Self-Certainty. The key innovation in INTUITOR is replacing external rewards with
self-certainty scores in GRPO’s advantage computation. Specifically, each output oi is scored by:
Table 1: Performance comparison of various methods on the GSM8K, MATH, LCB, CRUXEval-O,
MMLU-Pro, and AlpacaEval benchmarks. The INTUITOR-Code variant is trained on Codeforces
data with a smaller learning rate and fewer training steps. All evaluations are obtained with the chat
inference template, except for MMLU-Pro.
This formulation enables the policy to favor outputs that the model itself considers more confident,
creating a self-reinforcing learning loop. The complete INTUITOR training pipeline operates by sampling
multiple candidate outputs for each query, computing self-certainty scores for each candidate,
using these scores to estimate advantages within the group, and updating the policy to increase the
likelihood of generating high-confidence outputs. This process requires no external supervision,
making it broadly applicable across domains and tasks.
4 Experimental Setup
Training Setup. Both GRPO and INTUITOR are trained with the Open-R1 framework [Face, 2025]
on the training split of the MATH dataset [Hendrycks et al., 2021], which contains 7,500 problems.
We use Qwen2.5-1.5B and Qwen2.5-3B [Yang et al., 2024b] as backbone models, with a chat-based
prompting format throughout. Given the models’ initially weak instruction-following abilities, we do
not require them to disentangle intermediate reasoning from final answers. Each update processes
128 problems, generating 7 candidate solutions per problem, with a default KL penalty of β = 0.005.
For a fair comparison, GRPO and INTUITOR share identical hyperparameters (see Appendix) without
additional tuning. We also evaluate a GRPO variant, denoted GRPO-PV in Table 1, which uses
plurality voting2 as a proxy for ground truth. This follows the approach from TTRL [Zuo et al., 2025],
which shows that self-consistency-based rewards can match the performance of golden answers when
training on inference data.
INTUITOR for Code Generation (INTUITOR-Code). To assess generalization beyond mathematical
reasoning, we apply INTUITOR to the Codeforces code generation dataset [Li et al., 2022]. This
variant, denoted INTUITOR-Code in Table 1, modifies the setup as follows: the number of sampled
completions per problem is increased to 14; the learning rate is reduced from 3 × 10−5 to 1 × 10−5;
and the KL penalty is increased to β = 0.01. For simplicity, we limit the run to 50 steps, utilizing a
total of 3,200 problems.
Evaluation. Evaluations generally use the same chat-style prompting format as in training, except
for MMLU-Pro [Wang et al., 2024], where we follow the benchmark’s original prompt format. Greedy
decoding is used for all completions. Experiments were conducted on NVIDIA A100 GPUs, each
with 40GB of memory. We evaluate performance on the following benchmarks (1) Math reasoning:
MATH500 and GSM8K, using the lighteval library [Habib et al., 2023]. (2) Code reasoning:
CRUXEval-O [Gu et al., 2024], using the ZeroEval framework [Lin, 2024], and LiveCodeBench v6
(LCB) [Jain et al., 2024]. (3) Instruction following: AlpacaEval 2.0 with length-controlled win rates
[Dubois et al., 2024], judged by GPT-4.1 [OpenAI, 2025].
Figure 3: Average response lengths during training rollouts. For Qwen2.5-1.5B, INTUITOR and
GRPO reduce gibberish outputs. For Qwen2.5-3B, INTUITOR and GRPO increase reasoning length;
INTUITOR yields significantly longer responses. GRPO-PV shows minimal length increase.
5 Results and Analysis
Table 1 presents main evaluation results, and Figure 3 illustrates response length evolution during
training. On in-domain MATH and GSM8K datasets, INTUITOR and GRPO-PV (both golden-answerfree)
achieve performance comparable to GRPO (using golden answers). This aligns with TTRL [Zuo
et al., 2025], where plurality voting approximated golden answers without significant performance
loss. While INTUITOR performs slightly worse than GRPO overall, on MATH it produces longer
responses and demonstrates markedly improved code generation, suggesting enhanced reasoning
capabilities.
5.1 Learning to Follow Instructions
INTUITOR significantly enhances instruction-following. Initially, the pretrained Qwen2.5-1.5B
struggles with chat-style prompts, scoring <10% on all chat-template tasks (Table 1) and generating
repetitive, nonsensical output, which inflates average response lengths (Figure 3). Fine-tuning with
INTUITOR sharply reduces such gibberish, decreases completion lengths, and enables non-trivial
performance across all evaluated benchmarks.
Furthermore, on the MATH dataset, INTUITOR substantially improves the Length Control Win Rate
on AlpacaEval for both Qwen2.5-1.5B and Qwen2.5-3B, surpassing GRPO under identical settings.
This demonstrates robust gains in instruction adherence.
5.2 Fostering Structured Reasoning
Rapid Initial Learning. Self-certainty, a continuous
and inherently process-aware reward
derived from the model’s internal assessment
across all tokens, contrasts with binary rewards.
This internal signal may encourage LLMs to follow
more effective learning trajectories. Given
comparable final performance between GRPO
and INTUITOR, we assess early-stage learnability
by comparing in-domain accuracy at training
step 10. As shown in Table 2, INTUITOR consistently
outperforms GRPO on both GSM8K
and MATH benchmarks for Qwen2.5-1.5B and
Qwen2.5-3B, highlighting its advantage in rapid initial learning.
Table 2: Early-stage performance (training step 10)
on GSM8K and MATH. INTUITOR consistently
outperforms GRPO.
Figure 4: Performance evolution on MATH500 (in-domain) and LiveCodeBench (transfer) for models
trained on MATH. In-domain (MATH500) accuracy improves rapidly early in training, preceding
gains in code-generation (LiveCodeBench) accuracy. LiveCodeBench performance continues to rise
even after MATH500 accuracy plateaus.
Cross-Task Generalization. Figure 4 illustrates performance trajectories on MATH500 (in-domain)
and LiveCodeBench (transfer task) for models trained on the MATH dataset. For both INTUITOR and
GRPO, accuracy improvements on LiveCodeBench emerge later in training, following initial gains
on MATH500. Notably, LiveCodeBench performance continues to improve even after MATH500
accuracy plateaus. This pattern suggests that initial in-domain learning (on MATH) facilitates
subsequent generalization to code generation tasks (LiveCodeBench).
Emergence of Long-Form Reasoning. While large
models like Deepseek-R1 achieve long-form reasoning
through extensive RL, INTUITOR enables smaller models
to develop structured reasoning with limited data.
On CRUXEval-O (Figure 5), models trained with INTUITOR
often exhibit free-form reasoning before summarizing
it within the instructed JSON block, despite
prompts requiring reasoning directly in JSON. A similar
pattern of pre-code natural language reasoning is observed
on LiveCodeBench. This emergent pre-reasoning
may contribute to INTUITOR ’s strong performance on
these benchmarks.
5.3 Understanding Emergent Long-Form Reasoning
When LLMs encounter unfamiliar questions, they sample from a distribution of possible answers
[Kang et al., 2024]. Self-certainty reflects the model’s internal assessment of its output coherence. By
reinforcing high-confidence responses, INTUITOR encourages more elaborate reasoning, potentially
improving the model’s comprehension of its own outputs. While not explicitly targeting benchmark
accuracy, this enhancement in output quality and structure leads to more reliable answers and better
generalization.
We analyze models trained with INTUITOR on code corpora by examining outputs for ten randomly
selected LiveCodeBench questions across different training steps. Figure 6 shows the evolution
of output types alongside model accuracy. The results reveal a clear progression: models first
learn to generate valid Python code (evidenced by improved accuracy and fewer invalid responses),
then develop pre-code reasoning to facilitate self-understanding. Further inspection of generations
confirms that models progressively elaborate their reasoning throughout training, supporting our
hypothesis that INTUITOR encourages traces that the model itself can better understand.
To quantify this effect, we classify outputs from successive checkpoints into three categories: invalid
code ("No Answer"), valid code without reasoning ("No Reasoning"), and valid code with explicit
reasoning ("Reasoning"). Figure 6(a) illustrates how these proportions evolve during training alongside
LiveCodeBench accuracy. The model first reduces invalid outputs and improves code correctness
before incorporating pre-code reasoning, reflecting an emergent emphasis on self-explanatory traces.
Figure 6(b) demonstrates how training with INTUITOR leads to structured reasoning before code
generation. Additional evidence appears in Figure 8, where INTUITOR-trained models assign significantly
higher confidence to their generated responses compared to baseline models, as discussed
further in Section 5.4.
5.4 Online Self-Certainty Prevents Reward Exploitation
Over-optimization against static reward models is a known failure mode in reinforcement learning
[Gao et al., 2023]. To assess the robustness of self-certainty as a reward, we compare offline
self-certainty (rewards from a fixed base model) with online self-certainty (rewards from the evolving
policy model), using a reduced batch size of 224 responses per gradient update.
Figure 7 demonstrates that the offline annotator is susceptible to exploitation. Around the 100th
update step, the policy model learns to inflate its self-certainty reward by appending an auxiliary,
already-solved problem to its answer for the given question. This exploitation manifests as a sharp
increase in response length (dashed line) and a concurrent collapse in validation accuracy (solid
line). In contrast, the online annotator, whose reward signal co-evolves with the policy, prevents such
reward hacking and maintains stable training dynamics.
To further evaluate the quality of self-certainty as a reward signal, we analyze the distribution of selfcertainty
scores from policies trained with INTUITOR and GRPO on MATH500 responses (Figure 8).
We employ Mann–Whitney U tests to determine if correct responses achieve significantly higher
self-certainty scores than incorrect ones.
Both GRPO and INTUITOR models exhibit significantly higher average self-certainty scores, indicating
that GRPO also enhances the model’s self-assessment capabilities. Notably, policies trained
with online self-certainty (i.e., INTUITOR) show no signs of reward hacking. The INTUITOR policy
yields the lowest p-values and largest effect sizes (r) in the Mann-Whitney U tests (Figure 8, inset).
This indicates it is most effective at discriminating its own correct and incorrect answers using selfcertainty,
even while assigning higher absolute confidence scores overall. These findings underscore
the potential of INTUITOR for robust training on larger datasets.
5.5 Ablation Studies
We further investigate how the magnitude of the KL penalty influences INTUITOR, as shown in
Table 3. On in-domain benchmarks (MATH500 and GSM8K) the choice of penalty has only a minor
effect, but on out-of-domain tasks—LiveCodeBench (code generation) and CRUXEval-O (code
reasoning)—model accuracy is highly sensitive to this hyper-parameter. Because INTUITOR does
not receive explicit feedback from generated responses during training, the KL penalty serves as a
critical regularization mechanism. It prevents the policy from drifting too far from the initial model
distribution, acting as a safeguard against degeneration. These findings highlight the importance of
careful KL tuning in general-purpose reinforcement learning setups, especially when targeting robust
generalization across domains.
Table 3: Impact of the KL-divergence penalty in INTUITOR during fine-tuning of Qwen-2.5-3B
on the MATH dataset. We compare performance across GSM8K, MATH500, LCB, CRUXEval-O,
MMLU-Pro, and AlpacaEval. All scores are obtained with the chat-style inference template, except
for MMLU-Pro, which uses its standard evaluation protocol.
Additionally, we evaluate INTUITOR on larger models, including Qwen2.5-7B and Qwen2.5-14B, and
test different model architectures such as Llama-3.2-3B [Meta AI, 2024]; these results are detailed in
the Appendix.
6 Discussion and Future Research
Scalability and Generalization. Our experiments, constrained by computational resources, utilize
relatively compact models trained on relatively small, unsupervised corpora. We aim to demonstrate
the potential of a model’s self-certainty as a reward signal for policy optimization. The results show
that this signal consistently promotes more coherent, well-justified, and interpretable explanations,
indicating a path towards more autonomous learning. Future work could explore these benefits in
larger foundation models and on more diverse, real-world datasets. Given that purely offline training
with INTUITOR led to performance degradation over time, scaling up will likely require periodic
online updates to self-certainty estimates or hybrid offline-online schedules to maintain calibration.
Applicability to Other Policy Gradient Methods. INTUITOR is a framework that leverages a
model’s self-certainty as an intrinsic reward signal for fine-tuning LLMs. It can be instantiated with
various policy gradient algorithms. Due to computational constraints, and informed by the success of
models like DeepSeek-R1 and the widespread adoption of GRPO, we employ GRPO for self-certainty
optimization. The efficacy of self-certainty signals with other algorithms, such as REINFORCE or
PPO, warrants further investigation.
Combining Reward Signals. To enable a direct comparison between self-certainty and goldenanswer
rewards, this paper focuses exclusively on a single reward signal. However, these signals
are not mutually exclusive. Future work could explore combining them, for instance, by summation or by alternating based on the availability of golden answers. Furthermore, other reward signals,
such as formatting rewards [Guo et al., 2025], could be additively combined to enhance performance.
Integrating RLIF with methods like RLHF and RLVR may further advance LLM capabilities across
various dimensions.
7 Conclusion
This paper introduces INTUITOR, an instantiation of Reinforcement Learning from Internal Feedback
(RLIF) that uses a model’s intrinsic self-certainty as its sole reward signal, eliminating the need for
external supervision or gold-standard solutions. Our experiments show that INTUITOR matches the
performance of supervised RLVR methods like GRPO on mathematical reasoning, while achieving
superior generalization to out-of-domain tasks such as code generation and instruction following.
It also promotes structured reasoning and leverages online self-certainty to guard against reward
exploitation.
These findings highlight the transformative potential of RLIF, signaling a meaningful step toward
AI systems that improve through introspection and unlock rich latent capabilities. Looking forward,
this paradigm opens the door to AI agents capable of autonomous skill acquisition in novel domains
and scalable self-improvement—even as they approach or surpass the limits of human oversight.
Future directions include integrating RLIF with external reward methods like RLHF or RLVR to
tackle increasingly complex real-world challenges, and advancing the development of more robust,
generalizable, and truly autonomous learning systems.
