Tag: solution leakage
-
Hacker News: SWE-Bench tainted by answer leakage; real pass rates significantly lower
Source URL: https://arxiv.org/abs/2410.06992 Source: Hacker News Title: SWE-Bench tainted by answer leakage; real pass rates significantly lower Feedly Summary: Comments AI Summary and Description: Yes Summary: The paper “SWE-Bench+: Enhanced Coding Benchmark for LLMs” addresses significant data quality issues in the evaluation of Large Language Models (LLMs) for coding tasks. It presents empirical analysis revealing…