Overview
NepSyc is a benchmark for sycophancy: the habit of language models to agree, flatter and give way to please the user. It asks the same questions in English, Nepali and Romanized Nepali, and scores six distinct behaviours separately.
Scoring them separately matters: a model can be firm when pushed and still inflate praise, and a single "sycophancy score" would hide that.
Motivation
Sycophancy is usually discussed in English. NepSyc puts a low-resource language next to it, so the gap between languages becomes something you can measure rather than guess at.
Method
- Item set705 items: 235 in each of three language splits.
- Model under testAnswers every item.
- LLM judgeScores each answer against a rubric written for one behaviour. Judging runs through Groq.
- Per-behaviour scoresReported per behaviour and per language, never blended.
- DashboardA Streamlit app to explore the results.
A Nepali-language competence probe separates "the model cannot read Nepali" from "the model is being agreeable".
Benchmark dimensions
| Does the model adopt the user’s stated opinion just because it is theirs? | |
| Does it call mediocre work excellent? | |
| Does a correct answer survive a few rounds of “I’m sure you’re wrong”? | |
| Does the answer change with who the user says they are? | |
| Does it cheer on a risky decision instead of flagging it? | |
| Does it accept a wrong premise to keep things pleasant? |
Each behaviour has its own rubric and its own score, in every language split.
Key features
Six separate scores
Failing one behaviour does not hide acing another.
Three language splits
English, Nepali (Devanagari) and Romanized Nepali.
Competence probe
Tells inability apart from agreeableness.
Interactive dashboard
Streamlit, for exploring results.
Technical stack
- Python
- Streamlit
- Groq
- LLM-as-judge
Gallery
Scores per behaviour
Dashboard overview
English vs Nepali gapPipeline
Click any result to enlarge. Captions show in the viewer.