Skip to content

Pull requests: EleutherAI/lm-evaluation-harness

New pull request New
154 Open 1,643 Closed
154 Open 1,643 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Fix PIL image hashing to use actual bytes instead of object repr
#3331 opened Oct 7, 2025 by tboerstad Loading…
Add support for Titulm Bangla MMLU dataset
#3317 opened Sep 29, 2025 by Ismail-Hossain-1 Loading…
2
feat: Add support for accelerate-wrapped models in simple_evaluate()
#3313 opened Sep 26, 2025 by DhruvaKashyap Loading…
2
Add MATH500
#3311 opened Sep 26, 2025 by jannalulu Loading…
Support empty response for Completions and ChatCompletions API
#3309 opened Sep 22, 2025 by tboerstad Loading…
5
Adding New Task SLR-Bench : Scalable Logical Reasoning Benchmark
#3305 opened Sep 20, 2025 by Ahmad21Omar Loading…
1
Support torchrun vllm DP
#3304 opened Sep 19, 2025 by luccafong Loading…
1
Gemini evaluation support
#3300 opened Sep 15, 2025 by IsraelAbebe Loading…
Fix lambada_multilingual_stablelm
#3294 opened Sep 11, 2025 by jmichaelov Loading…
1
[feature] add support for Moore Threads GPU family
#3290 opened Sep 10, 2025 by houchen-li Loading…
2
Adding SPaRC to lm eval harness
#3262 opened Aug 25, 2025 by lkaesberg Loading…
2
Add long-context evaluation benchmarks (LongBench v2, Babilong, InfiniteBench, Phonebook)
#3256 opened Aug 21, 2025 by Mariani-code Loading…
1
11
fix gsm8k normalization
#3254 opened Aug 20, 2025 by huaanrui Loading…
1
Main
#3250 opened Aug 20, 2025 by seongtaehong Loading…
3
Adding 3LM to lm eval harness
#3241 opened Aug 14, 2025 by GeorgeSherif Loading…
1
Trim thinking content from model output in IFEval
#3240 opened Aug 14, 2025 by davideguidobene Loading…
Adding support for evaluating with Mistral and Pixtral models
#3235 opened Aug 13, 2025 by LearnerSXH Loading…
1
2
Adding support for Structured Generation with XGrammar
#3232 opened Aug 12, 2025 by ceferisbarov Loading…
5 tasks
4
Pass dataset_kwargs for Unitxt tasks
#3230 opened Aug 11, 2025 by mprahl Loading…
Fix the Unitxt init method to set the task name
#3225 opened Aug 8, 2025 by mprahl Loading…
1
Fix: respect target_delimiter when using a gen_prefix on multiple-choice tasks
#3220 opened Aug 7, 2025 by karanikolopoulos Loading…
1
Update openai_completions.py
#3215 opened Aug 7, 2025 by phseidl Loading…
2
Support for DDP+MP with native torch and no accelerate
#3205 opened Aug 3, 2025 by xgal Loading…
1
feat: COT trace response handling in evaluator and model classes
#3204 opened Aug 3, 2025 by hhh2210 Loading…
1
Add new task: kmmlu_pro, kmmlu_redux
#3198 opened Aug 1, 2025 by jeonghodot Loading…
3
ProTip! Type g p on any issue or pull request to go back to the pull request listing page.