Skip to content

#Reviews/benchmarks

0 items today
10/6Tue
  1. DEV Community · MCP78

    FP8 pitfall: GPU bill dropped 47%, but the model outputs “!!!!!!”

    The author ran Qwen2.5 7B/32B/72B on a single AMD MI300X with vLLM ROCm, priced at $2.99/GPU-hr. The BF16 baseline was 7B at $0.227/M, 32B at $0.77/M, and 72B at $1.67/M output tokens.

    Why it matters: The author benchmarked FP8 quantization on the MI300X and found that per-token billing can hide the model's output degrading into gibberish, then gave a reusable way to verify it.

9/24Thu
  1. Hacker News · Prompt Injection78

    Can Open-Source Prompt Injection Detectors Stop Real AI Agent Attacks? Testing 629 AgentDojo Attacks in Practice

    The author tested 10 open-source prompt injection detectors against 629 AgentDojo injection attacks—each buried in real tool output—plus 97 benign samples.

    Why it matters: The author tested 10 open-source detectors against 629 real injection attacks, with full comparison data at both default thresholds and after calibration.

9/9Tue
  1. Terminal-Bench · News38

    Terminal-Bench 排行榜回应超时争议：OB-1 重新提交成绩后重回榜首

    OpenBlock Labs 的智能体 OB-1 在按正确超时限制重新提交成绩后，重新成为 Terminal-Bench 排行榜得分最高的智能体。此前该提交将数据集中每个任务的超时统一改为固定 30 分钟，而多数任务原本限时 5 分钟，导致 75 个任务超时被放宽、2 个不变、3 个被缩短。

    Awaiting translation