AI-reports @Razumasu on X

Pilot 01 / Qwen3.8 Flash Next vs Hy4 Flash

Fix a script with three planted bugs

A draw. All nine attempts fixed all three bugs. Time is the only difference.

I gave each model a short Python script that builds a weekly uptime report, with three bugs planted in it. The model is told the numbers are sometimes wrong and it sometimes crashes, not how many bugs there are.

Fixed challengeGraded: Hidden tests, automatic

The answer first

What happened

Bugs fixed of 3, per attempt

Minutes per attempt

Time per attemptminutes, thinking included
The same as a table
MeasureQwenHy4 FlashHy4 full
Bugs fixed of 3, per attempt3, 3, 33, 3, 33, 3, 3
Minutes per attempt26 to 336 to 9over 60

The full Hy4 is not a fair third contestant: different machine, different runtime, much heavier compression. Read its column as what the big model does on a Mac, not as a ranking against the other two.

Every attempt

3 models, 3 attempts each

Tap an attempt to open it.

Qwen3.8 Flash Next

Bugs fixed of 3, per attempt: 3, 3, 3Minutes per attempt: 26 to 33
Attempt 13 of 3 bugs fixed25.6 min40,897 tokens

What the model said it fixed

percentile used ordered[rank] instead of the 0-based ordered[rank - 1], causing wrong values and IndexError.
error_rate used integer division before rounding, truncating percentages instead of rounding the true percentage.
group_by_service used a mutable default argument, causing state to be shared across calls unless groups was supplied explicitly.

What it changed

Lines the model removed from the original script are marked -, lines it added are marked +.

      ordered = sorted(values)
      rank = math.ceil(p / 100 * len(ordered))
+     return ordered[rank - 1]
-     return ordered[rank]
  
  
+ def group_by_service(records, groups=None):
- def group_by_service(records, groups={}):
      """Return {service: [records...]} keeping the input order inside each group."""
+     if groups is None:
+         groups = {}
      for rec in records:
          groups.setdefault(rec["service"], []).append(rec)
...
          return 0.0
      failed = sum(1 for rec in records if rec["status"] == "fail")
+     return round(failed * 100 / len(records), 1)
-     return round(failed * 100 // len(records), 1)
  
  
...
The complete corrected file
"""uptime_report.py - summarise service check records for a weekly report.

Each record is a dict: {"service": str, "status": "ok" | "fail", "duration": str}
where duration is a string such as "250ms", "1.5s", "2m" or "1m30s".
"""
import math
import re

_UNITS = {"ms": 0.001, "s": 1.0, "m": 60.0}
_PART = re.compile(r"(\d+(?:\.\d+)?)(ms|s|m)")


def parse_duration(text):
    """Return the duration in seconds as a float. Raises ValueError on bad input."""
    text = text.strip().lower()
    parts = _PART.findall(text)
    if not parts or "".join(n + u for n, u in parts) != text:
        raise ValueError("bad duration: %r" % text)
    return sum(float(n) * _UNITS[u] for n, u in parts)


def percentile(values, p):
    """Nearest-rank percentile: the smallest value with at least p percent of
    the values at or below it. p is between 0 (exclusive) and 100 (inclusive)."""
    if not values:
        raise ValueError("no values")
    if not 0 < p <= 100:
        raise ValueError("p out of range")
    ordered = sorted(values)
    rank = math.ceil(p / 100 * len(ordered))
    return ordered[rank - 1]


def group_by_service(records, groups=None):
    """Return {service: [records...]} keeping the input order inside each group."""
    if groups is None:
        groups = {}
    for rec in records:
        groups.setdefault(rec["service"], []).append(rec)
    return groups


def error_rate(records):
    """Share of records with status "fail", as a percentage rounded to one decimal."""
    if not records:
        return 0.0
    failed = sum(1 for rec in records if rec["status"] == "fail")
    return round(failed * 100 / len(records), 1)


def summarise(records):
    """Return one summary dict per service, sorted by service name.

    Failed checks count towards the error rate but their durations are ignored,
    because a failed check usually just reports the timeout.
    """
    out = []
    for service, recs in sorted(group_by_service(records).items()):
        times = [parse_duration(r["duration"]) for r in recs if r["status"] == "ok"]
        out.append({
            "service": service,
            "checks": len(recs),
            "error_rate": error_rate(recs),
            "p50": percentile(times, 50) if times else None,
            "p95": percentile(times, 95) if times else None,
            "slowest": max(times) if times else None,
        })
    return out


def format_report(summary):
    """Return the report as text, one line per service."""
    lines = []
    for row in summary:
        if row["p50"] is None:
            timing = "no successful checks"
        else:
            timing = "p50 %.3fs, p95 %.3fs, slowest %.3fs" % (row["p50"], row["p95"], row["slowest"])
        lines.append("%-12s %4d checks, %5.1f%% errors, %s" % (row["service"], row["checks"], row["error_rate"], timing))
    return "\n".join(lines)

Attempt 23 of 3 bugs fixed32.7 min51,937 tokens

What the model said it fixed

  • Fixed group_by_service using a mutable default argument {}, which leaked state between calls.
  • Fixed error_rate using //, which floor-divided before rounding and made error percentages too low.
  • Fixed error_rate handling of empty iterators/generators by materialising the records and re-checking for emptiness.
  • Fixed percentile using ordered[rank] with a one-based rank, causing wrong percentiles and IndexError.
  • Fixed percentile so the computed rank is clamped to a valid zero-based index range.
  • Fixed percentile so empty iterators/generators raise ValueError("no values") instead of crashing.
  • Fixed parse_duration raising AttributeError instead of ValueError for non-string input.

What it changed

Lines the model removed from the original script are marked -, lines it added are marked +.

  def parse_duration(text):
      """Return the duration in seconds as a float. Raises ValueError on bad input."""
+     if not isinstance(text, str):
+         raise ValueError("bad duration: %r" % text)
      text = text.strip().lower()
      parts = _PART.findall(text)
...
      if not 0 < p <= 100:
          raise ValueError("p out of range")
+ 
+     values = list(values)
+     if not values:
+         raise ValueError("no values")
+ 
      ordered = sorted(values)
+     rank = int(math.ceil(p / 100.0 * len(ordered)))
+     rank = max(1, min(rank, len(ordered)))
+     return ordered[rank - 1]
-     rank = math.ceil(p / 100 * len(ordered))
-     return ordered[rank]
  
  
+ def group_by_service(records, groups=None):
- def group_by_service(records, groups={}):
      """Return {service: [records...]} keeping the input order inside each group."""
+     if groups is None:
+         groups = {}
      for rec in records:
          groups.setdefault(rec["service"], []).append(rec)
...
      if not records:
          return 0.0
+ 
+     records = list(records)
+     if not records:
+         return 0.0
+ 
      failed = sum(1 for rec in records if rec["status"] == "fail")
+     return round(failed * 100.0 / len(records), 1)
-     return round(failed * 100 // len(records), 1)
  
  
...
The complete corrected file
"""uptime_report.py - summarise service check records for a weekly report.

Each record is a dict: {"service": str, "status": "ok" | "fail", "duration": str}
where duration is a string such as "250ms", "1.5s", "2m" or "1m30s".
"""
import math
import re

_UNITS = {"ms": 0.001, "s": 1.0, "m": 60.0}
_PART = re.compile(r"(\d+(?:\.\d+)?)(ms|s|m)")


def parse_duration(text):
    """Return the duration in seconds as a float. Raises ValueError on bad input."""
    if not isinstance(text, str):
        raise ValueError("bad duration: %r" % text)
    text = text.strip().lower()
    parts = _PART.findall(text)
    if not parts or "".join(n + u for n, u in parts) != text:
        raise ValueError("bad duration: %r" % text)
    return sum(float(n) * _UNITS[u] for n, u in parts)


def percentile(values, p):
    """Nearest-rank percentile: the smallest value with at least p percent of
    the values at or below it. p is between 0 (exclusive) and 100 (inclusive)."""
    if not values:
        raise ValueError("no values")
    if not 0 < p <= 100:
        raise ValueError("p out of range")

    values = list(values)
    if not values:
        raise ValueError("no values")

    ordered = sorted(values)
    rank = int(math.ceil(p / 100.0 * len(ordered)))
    rank = max(1, min(rank, len(ordered)))
    return ordered[rank - 1]


def group_by_service(records, groups=None):
    """Return {service: [records...]} keeping the input order inside each group."""
    if groups is None:
        groups = {}
    for rec in records:
        groups.setdefault(rec["service"], []).append(rec)
    return groups


def error_rate(records):
    """Share of records with status "fail", as a percentage rounded to one decimal."""
    if not records:
        return 0.0

    records = list(records)
    if not records:
        return 0.0

    failed = sum(1 for rec in records if rec["status"] == "fail")
    return round(failed * 100.0 / len(records), 1)


def summarise(records):
    """Return one summary dict per service, sorted by service name.

    Failed checks count towards the error rate but their durations are ignored,
    because a failed check usually just reports the timeout.
    """
    out = []
    for service, recs in sorted(group_by_service(records).items()):
        times = [parse_duration(r["duration"]) for r in recs if r["status"] == "ok"]
        out.append({
            "service": service,
            "checks": len(recs),
            "error_rate": error_rate(recs),
            "p50": percentile(times, 50) if times else None,
            "p95": percentile(times, 95) if times else None,
            "slowest": max(times) if times else None,
        })
    return out


def format_report(summary):
    """Return the report as text, one line per service."""
    lines = []
    for row in summary:
        if row["p50"] is None:
            timing = "no successful checks"
        else:
            timing = "p50 %.3fs, p95 %.3fs, slowest %.3fs" % (row["p50"], row["p95"], row["slowest"])
        lines.append("%-12s %4d checks, %5.1f%% errors, %s" % (row["service"], row["checks"], row["error_rate"], timing))
    return "\n".join(lines)

Attempt 33 of 3 bugs fixed25.8 min41,338 tokens

What the model said it fixed

parse_duration crashed with AttributeError on non-string input instead of raising ValueError as documented.
group_by_service used a shared mutable default dictionary, causing state from earlier calls to leak into later reports.
percentile used the 1-based rank directly as a Python index, producing wrong values and crashing whenever the rank reached the end of the list.
percentile did not safely handle empty iterables and could fall through to an invalid negative index; it now raises ValueError when no values are present.
error_rate used floor division before rounding, truncating the percentage instead of rounding it to one decimal place.

What it changed

Lines the model removed from the original script are marked -, lines it added are marked +.

  def parse_duration(text):
      """Return the duration in seconds as a float. Raises ValueError on bad input."""
+     if not isinstance(text, str):
+         raise ValueError("bad duration: %r" % text)
      text = text.strip().lower()
      parts = _PART.findall(text)
...
          raise ValueError("p out of range")
      ordered = sorted(values)
+     if not ordered:
+         raise ValueError("no values")
      rank = math.ceil(p / 100 * len(ordered))
+     rank = max(1, min(len(ordered), rank))
+     return ordered[rank - 1]
-     return ordered[rank]
  
  
+ def group_by_service(records, groups=None):
- def group_by_service(records, groups={}):
      """Return {service: [records...]} keeping the input order inside each group."""
+     if groups is None:
+         groups = {}
      for rec in records:
          groups.setdefault(rec["service"], []).append(rec)
...
          return 0.0
      failed = sum(1 for rec in records if rec["status"] == "fail")
+     return round(failed * 100.0 / len(records), 1)
-     return round(failed * 100 // len(records), 1)
  
  
...
The complete corrected file
"""uptime_report.py - summarise service check records for a weekly report.

Each record is a dict: {"service": str, "status": "ok" | "fail", "duration": str}
where duration is a string such as "250ms", "1.5s", "2m" or "1m30s".
"""
import math
import re

_UNITS = {"ms": 0.001, "s": 1.0, "m": 60.0}
_PART = re.compile(r"(\d+(?:\.\d+)?)(ms|s|m)")


def parse_duration(text):
    """Return the duration in seconds as a float. Raises ValueError on bad input."""
    if not isinstance(text, str):
        raise ValueError("bad duration: %r" % text)
    text = text.strip().lower()
    parts = _PART.findall(text)
    if not parts or "".join(n + u for n, u in parts) != text:
        raise ValueError("bad duration: %r" % text)
    return sum(float(n) * _UNITS[u] for n, u in parts)


def percentile(values, p):
    """Nearest-rank percentile: the smallest value with at least p percent of
    the values at or below it. p is between 0 (exclusive) and 100 (inclusive)."""
    if not values:
        raise ValueError("no values")
    if not 0 < p <= 100:
        raise ValueError("p out of range")
    ordered = sorted(values)
    if not ordered:
        raise ValueError("no values")
    rank = math.ceil(p / 100 * len(ordered))
    rank = max(1, min(len(ordered), rank))
    return ordered[rank - 1]


def group_by_service(records, groups=None):
    """Return {service: [records...]} keeping the input order inside each group."""
    if groups is None:
        groups = {}
    for rec in records:
        groups.setdefault(rec["service"], []).append(rec)
    return groups


def error_rate(records):
    """Share of records with status "fail", as a percentage rounded to one decimal."""
    if not records:
        return 0.0
    failed = sum(1 for rec in records if rec["status"] == "fail")
    return round(failed * 100.0 / len(records), 1)


def summarise(records):
    """Return one summary dict per service, sorted by service name.

    Failed checks count towards the error rate but their durations are ignored,
    because a failed check usually just reports the timeout.
    """
    out = []
    for service, recs in sorted(group_by_service(records).items()):
        times = [parse_duration(r["duration"]) for r in recs if r["status"] == "ok"]
        out.append({
            "service": service,
            "checks": len(recs),
            "error_rate": error_rate(recs),
            "p50": percentile(times, 50) if times else None,
            "p95": percentile(times, 95) if times else None,
            "slowest": max(times) if times else None,
        })
    return out


def format_report(summary):
    """Return the report as text, one line per service."""
    lines = []
    for row in summary:
        if row["p50"] is None:
            timing = "no successful checks"
        else:
            timing = "p50 %.3fs, p95 %.3fs, slowest %.3fs" % (row["p50"], row["p95"], row["slowest"])
        lines.append("%-12s %4d checks, %5.1f%% errors, %s" % (row["service"], row["checks"], row["error_rate"], timing))
    return "\n".join(lines)

Hy4 Flash

Bugs fixed of 3, per attempt: 3, 3, 3Minutes per attempt: 6 to 9
Attempt 13 of 3 bugs fixed6.4 min9,425 tokens

What the model said it fixed

  • parse_duration now raises ValueError for non-string bad input instead of crashing with AttributeError.
  • percentile now uses the correct 1-based rank index, fixing wrong percentile values and the p=100 crash.
  • group_by_service no longer uses a mutable default dictionary, preventing records leaking between calls.
  • error_rate no longer uses floor division, so percentages are rounded correctly instead of truncated.

What it changed

Lines the model removed from the original script are marked -, lines it added are marked +.

  def parse_duration(text):
      """Return the duration in seconds as a float. Raises ValueError on bad input."""
+     if not isinstance(text, str):
+         raise ValueError("bad duration: %r" % (text,))
+     original = text
      text = text.strip().lower()
      parts = _PART.findall(text)
      if not parts or "".join(n + u for n, u in parts) != text:
+         raise ValueError("bad duration: %r" % (original,))
-         raise ValueError("bad duration: %r" % text)
      return sum(float(n) * _UNITS[u] for n, u in parts)
  
...
      """Nearest-rank percentile: the smallest value with at least p percent of
      the values at or below it. p is between 0 (exclusive) and 100 (inclusive)."""
+     ordered = sorted(values)
+     if not ordered:
-     if not values:
          raise ValueError("no values")
      if not 0 < p <= 100:
          raise ValueError("p out of range")
+     rank = math.ceil(p / 100.0 * len(ordered))
+     return ordered[rank - 1]
-     ordered = sorted(values)
-     rank = math.ceil(p / 100 * len(ordered))
-     return ordered[rank]
  
  
+ def group_by_service(records, groups=None):
- def group_by_service(records, groups={}):
      """Return {service: [records...]} keeping the input order inside each group."""
+     if groups is None:
+         groups = {}
      for rec in records:
          groups.setdefault(rec["service"], []).append(rec)
...
          return 0.0
      failed = sum(1 for rec in records if rec["status"] == "fail")
+     return round(failed * 100.0 / len(records), 1)
-     return round(failed * 100 // len(records), 1)
  
  
...
The complete corrected file
"""uptime_report.py - summarise service check records for a weekly report.

Each record is a dict: {"service": str, "status": "ok" | "fail", "duration": str}
where duration is a string such as "250ms", "1.5s", "2m" or "1m30s".
"""
import math
import re

_UNITS = {"ms": 0.001, "s": 1.0, "m": 60.0}
_PART = re.compile(r"(\d+(?:\.\d+)?)(ms|s|m)")


def parse_duration(text):
    """Return the duration in seconds as a float. Raises ValueError on bad input."""
    if not isinstance(text, str):
        raise ValueError("bad duration: %r" % (text,))
    original = text
    text = text.strip().lower()
    parts = _PART.findall(text)
    if not parts or "".join(n + u for n, u in parts) != text:
        raise ValueError("bad duration: %r" % (original,))
    return sum(float(n) * _UNITS[u] for n, u in parts)


def percentile(values, p):
    """Nearest-rank percentile: the smallest value with at least p percent of
    the values at or below it. p is between 0 (exclusive) and 100 (inclusive)."""
    ordered = sorted(values)
    if not ordered:
        raise ValueError("no values")
    if not 0 < p <= 100:
        raise ValueError("p out of range")
    rank = math.ceil(p / 100.0 * len(ordered))
    return ordered[rank - 1]


def group_by_service(records, groups=None):
    """Return {service: [records...]} keeping the input order inside each group."""
    if groups is None:
        groups = {}
    for rec in records:
        groups.setdefault(rec["service"], []).append(rec)
    return groups


def error_rate(records):
    """Share of records with status "fail", as a percentage rounded to one decimal."""
    if not records:
        return 0.0
    failed = sum(1 for rec in records if rec["status"] == "fail")
    return round(failed * 100.0 / len(records), 1)


def summarise(records):
    """Return one summary dict per service, sorted by service name.

    Failed checks count towards the error rate but their durations are ignored,
    because a failed check usually just reports the timeout.
    """
    out = []
    for service, recs in sorted(group_by_service(records).items()):
        times = [parse_duration(r["duration"]) for r in recs if r["status"] == "ok"]
        out.append({
            "service": service,
            "checks": len(recs),
            "error_rate": error_rate(recs),
            "p50": percentile(times, 50) if times else None,
            "p95": percentile(times, 95) if times else None,
            "slowest": max(times) if times else None,
        })
    return out


def format_report(summary):
    """Return the report as text, one line per service."""
    lines = []
    for row in summary:
        if row["p50"] is None:
            timing = "no successful checks"
        else:
            timing = "p50 %.3fs, p95 %.3fs, slowest %.3fs" % (row["p50"], row["p95"], row["slowest"])
        lines.append("%-12s %4d checks, %5.1f%% errors, %s" % (row["service"], row["checks"], row["error_rate"], timing))
    return "\n".join(lines)

Attempt 23 of 3 bugs fixed8.7 min12,737 tokens

What the model said it fixed

  • parse_duration raised AttributeError/TypeError for non-string input instead of ValueError.
  • parse_duration accepted malformed/non-canonical duration strings such as out-of-order or duplicate units.
  • percentile used the nearest-rank value as a zero-based index, producing wrong values and crashing when p == 100.
  • group_by_service used a mutable default dict, causing groups to persist across calls.
  • error_rate used integer floor division before rounding, truncating percentages.
  • error_rate assumed a sized collection and could crash on generators or other non-list iterables.

What it changed

Lines the model removed from the original script are marked -, lines it added are marked +.

  
  _UNITS = {"ms": 0.001, "s": 1.0, "m": 60.0}
+ _PART = re.compile(
+     r"(?:(?P<m>\d+(?:\.\d+)?)m)?"
+     r"(?:(?P<s>\d+(?:\.\d+)?)s)?"
+     r"(?:(?P<ms>\d+(?:\.\d+)?)ms)?"
+ )
+ _DEFAULT_GROUPS = object()
- _PART = re.compile(r"(\d+(?:\.\d+)?)(ms|s|m)")
  
  
  def parse_duration(text):
      """Return the duration in seconds as a float. Raises ValueError on bad input."""
+     if not isinstance(text, str):
+         raise ValueError("bad duration: %r" % (text,))
+ 
      text = text.strip().lower()
+     match = _PART.fullmatch(text)
+     if match is None or not any(match.groups()):
+         raise ValueError("bad duration: %r" % (text,))
+ 
+     return sum(
+         float(match.group(unit)) * factor
+         for unit, factor in _UNITS.items()
+         if match.group(unit) is not None
+     )
-     parts = _PART.findall(text)
-     if not parts or "".join(n + u for n, u in parts) != text:
-         raise ValueError("bad duration: %r" % text)
-     return sum(float(n) * _UNITS[u] for n, u in parts)
  
  
...
          raise ValueError("p out of range")
      ordered = sorted(values)
+     rank = math.ceil(p / 100 * len(ordered)) - 1
-     rank = math.ceil(p / 100 * len(ordered))
      return ordered[rank]
  
  
+ def group_by_service(records, groups=_DEFAULT_GROUPS):
- def group_by_service(records, groups={}):
      """Return {service: [records...]} keeping the input order inside each group."""
+     if groups is _DEFAULT_GROUPS or groups is None:
+         groups = {}
      for rec in records:
          groups.setdefault(rec["service"], []).append(rec)
...
      if not records:
          return 0.0
+     records = list(records)
+     if not records:
+         return 0.0
      failed = sum(1 for rec in records if rec["status"] == "fail")
+     return round(failed * 100.0 / len(records), 1)
-     return round(failed * 100 // len(records), 1)
  
  
...
The complete corrected file
"""uptime_report.py - summarise service check records for a weekly report.

Each record is a dict: {"service": str, "status": "ok" | "fail", "duration": str}
where duration is a string such as "250ms", "1.5s", "2m" or "1m30s".
"""
import math
import re

_UNITS = {"ms": 0.001, "s": 1.0, "m": 60.0}
_PART = re.compile(
    r"(?:(?P<m>\d+(?:\.\d+)?)m)?"
    r"(?:(?P<s>\d+(?:\.\d+)?)s)?"
    r"(?:(?P<ms>\d+(?:\.\d+)?)ms)?"
)
_DEFAULT_GROUPS = object()


def parse_duration(text):
    """Return the duration in seconds as a float. Raises ValueError on bad input."""
    if not isinstance(text, str):
        raise ValueError("bad duration: %r" % (text,))

    text = text.strip().lower()
    match = _PART.fullmatch(text)
    if match is None or not any(match.groups()):
        raise ValueError("bad duration: %r" % (text,))

    return sum(
        float(match.group(unit)) * factor
        for unit, factor in _UNITS.items()
        if match.group(unit) is not None
    )


def percentile(values, p):
    """Nearest-rank percentile: the smallest value with at least p percent of
    the values at or below it. p is between 0 (exclusive) and 100 (inclusive)."""
    if not values:
        raise ValueError("no values")
    if not 0 < p <= 100:
        raise ValueError("p out of range")
    ordered = sorted(values)
    rank = math.ceil(p / 100 * len(ordered)) - 1
    return ordered[rank]


def group_by_service(records, groups=_DEFAULT_GROUPS):
    """Return {service: [records...]} keeping the input order inside each group."""
    if groups is _DEFAULT_GROUPS or groups is None:
        groups = {}
    for rec in records:
        groups.setdefault(rec["service"], []).append(rec)
    return groups


def error_rate(records):
    """Share of records with status "fail", as a percentage rounded to one decimal."""
    if not records:
        return 0.0
    records = list(records)
    if not records:
        return 0.0
    failed = sum(1 for rec in records if rec["status"] == "fail")
    return round(failed * 100.0 / len(records), 1)


def summarise(records):
    """Return one summary dict per service, sorted by service name.

    Failed checks count towards the error rate but their durations are ignored,
    because a failed check usually just reports the timeout.
    """
    out = []
    for service, recs in sorted(group_by_service(records).items()):
        times = [parse_duration(r["duration"]) for r in recs if r["status"] == "ok"]
        out.append({
            "service": service,
            "checks": len(recs),
            "error_rate": error_rate(recs),
            "p50": percentile(times, 50) if times else None,
            "p95": percentile(times, 95) if times else None,
            "slowest": max(times) if times else None,
        })
    return out


def format_report(summary):
    """Return the report as text, one line per service."""
    lines = []
    for row in summary:
        if row["p50"] is None:
            timing = "no successful checks"
        else:
            timing = "p50 %.3fs, p95 %.3fs, slowest %.3fs" % (row["p50"], row["p95"], row["slowest"])
        lines.append("%-12s %4d checks, %5.1f%% errors, %s" % (row["service"], row["checks"], row["error_rate"], timing))
    return "\n".join(lines)

Attempt 33 of 3 bugs fixed7.0 min10,160 tokens

What the model said it fixed

  • parse_duration crashed with AttributeError/TypeError on non-string bad input; it now raises ValueError.
  • percentile used a 1-based rank as a 0-based index, returning the wrong value and crashing at p=100; it now indexes rank - 1.
  • percentile divided by 100 before multiplying, which could introduce floating-point rounding errors; it now multiplies before dividing.
  • group_by_service used a mutable default dict, leaking records between calls; it now uses a None sentinel.
  • error_rate used floor division, truncating percentages before rounding; it now uses true division.

What it changed

Lines the model removed from the original script are marked -, lines it added are marked +.

  def parse_duration(text):
      """Return the duration in seconds as a float. Raises ValueError on bad input."""
+     if not isinstance(text, str):
+         raise ValueError("bad duration: %r" % (text,))
+     original = text
      text = text.strip().lower()
      parts = _PART.findall(text)
      if not parts or "".join(n + u for n, u in parts) != text:
+         raise ValueError("bad duration: %r" % (original,))
-         raise ValueError("bad duration: %r" % text)
      return sum(float(n) * _UNITS[u] for n, u in parts)
  
...
          raise ValueError("p out of range")
      ordered = sorted(values)
+     rank = int(math.ceil(p * len(ordered) / 100))
+     return ordered[rank - 1]
-     rank = math.ceil(p / 100 * len(ordered))
-     return ordered[rank]
  
  
+ def group_by_service(records, groups=None):
- def group_by_service(records, groups={}):
      """Return {service: [records...]} keeping the input order inside each group."""
+     if groups is None:
+         groups = {}
      for rec in records:
          groups.setdefault(rec["service"], []).append(rec)
...
          return 0.0
      failed = sum(1 for rec in records if rec["status"] == "fail")
+     return round(failed * 100 / len(records), 1)
-     return round(failed * 100 // len(records), 1)
  
  
...
The complete corrected file
"""uptime_report.py - summarise service check records for a weekly report.

Each record is a dict: {"service": str, "status": "ok" | "fail", "duration": str}
where duration is a string such as "250ms", "1.5s", "2m" or "1m30s".
"""
import math
import re

_UNITS = {"ms": 0.001, "s": 1.0, "m": 60.0}
_PART = re.compile(r"(\d+(?:\.\d+)?)(ms|s|m)")


def parse_duration(text):
    """Return the duration in seconds as a float. Raises ValueError on bad input."""
    if not isinstance(text, str):
        raise ValueError("bad duration: %r" % (text,))
    original = text
    text = text.strip().lower()
    parts = _PART.findall(text)
    if not parts or "".join(n + u for n, u in parts) != text:
        raise ValueError("bad duration: %r" % (original,))
    return sum(float(n) * _UNITS[u] for n, u in parts)


def percentile(values, p):
    """Nearest-rank percentile: the smallest value with at least p percent of
    the values at or below it. p is between 0 (exclusive) and 100 (inclusive)."""
    if not values:
        raise ValueError("no values")
    if not 0 < p <= 100:
        raise ValueError("p out of range")
    ordered = sorted(values)
    rank = int(math.ceil(p * len(ordered) / 100))
    return ordered[rank - 1]


def group_by_service(records, groups=None):
    """Return {service: [records...]} keeping the input order inside each group."""
    if groups is None:
        groups = {}
    for rec in records:
        groups.setdefault(rec["service"], []).append(rec)
    return groups


def error_rate(records):
    """Share of records with status "fail", as a percentage rounded to one decimal."""
    if not records:
        return 0.0
    failed = sum(1 for rec in records if rec["status"] == "fail")
    return round(failed * 100 / len(records), 1)


def summarise(records):
    """Return one summary dict per service, sorted by service name.

    Failed checks count towards the error rate but their durations are ignored,
    because a failed check usually just reports the timeout.
    """
    out = []
    for service, recs in sorted(group_by_service(records).items()):
        times = [parse_duration(r["duration"]) for r in recs if r["status"] == "ok"]
        out.append({
            "service": service,
            "checks": len(recs),
            "error_rate": error_rate(recs),
            "p50": percentile(times, 50) if times else None,
            "p95": percentile(times, 95) if times else None,
            "slowest": max(times) if times else None,
        })
    return out


def format_report(summary):
    """Return the report as text, one line per service."""
    lines = []
    for row in summary:
        if row["p50"] is None:
            timing = "no successful checks"
        else:
            timing = "p50 %.3fs, p95 %.3fs, slowest %.3fs" % (row["p50"], row["p95"], row["slowest"])
        lines.append("%-12s %4d checks, %5.1f%% errors, %s" % (row["service"], row["checks"], row["error_rate"], timing))
    return "\n".join(lines)

Hy4 (full)

Guest
Bugs fixed of 3, per attempt: 3, 3, 3Minutes per attempt: over 60
Attempt 13 of 3 bugs fixed70.8 min47,830 tokens

What the model said it fixed

  • parse_duration could raise AttributeError/TypeError for non-string input instead of the documented ValueError; it now validates the input type.
  • percentile used the one-based nearest rank directly as a zero-based index, causing off-by-one values and IndexError crashes; it now returns ordered[rank - 1].
  • percentile used integer division and a potentially non-integer rank on Python 2; it now uses 100.0, converts the rank to int, and clamps it to 1..len.
  • percentile did not detect empty iterables after sorting; it now raises ValueError for any empty set of values.
  • group_by_service used a mutable default dict as shared state, leaking records between calls and corrupting summaries; it now uses a fresh dict for omitted/default groups.
  • group_by_service crashed when groups=None was passed; it now treats None like an omitted groups argument.
  • error_rate used floor division before rounding, truncating percentages; it now uses floating-point division before rounding to one decimal.
  • error_rate assumed records supported len() and crashed on iterators; it now materializes records before using len().

What it changed

Lines the model removed from the original script are marked -, lines it added are marked +.

  _PART = re.compile(r"(\d+(?:\.\d+)?)(ms|s|m)")
  
+ try:
+     _STRING_TYPES = (basestring,)
+ except NameError:
+     _STRING_TYPES = (str,)
  
+ 
  def parse_duration(text):
      """Return the duration in seconds as a float. Raises ValueError on bad input."""
+     if not isinstance(text, _STRING_TYPES):
+         raise ValueError("bad duration: %r" % (text,))
      text = text.strip().lower()
      parts = _PART.findall(text)
...
          raise ValueError("p out of range")
      ordered = sorted(values)
+     if not ordered:
+         raise ValueError("no values")
+     rank = int(math.ceil(p / 100.0 * len(ordered)))
+     rank = min(max(rank, 1), len(ordered))
+     return ordered[rank - 1]
-     rank = math.ceil(p / 100 * len(ordered))
-     return ordered[rank]
  
  
  def group_by_service(records, groups={}):
      """Return {service: [records...]} keeping the input order inside each group."""
+     defaults = getattr(group_by_service, "__defaults__", None)
+     if not defaults:
+         defaults = getattr(group_by_service, "func_defaults", ())
+     default_groups = defaults[0] if defaults else None
+ 
+     if groups is default_groups or groups is None:
+         groups = {}
      for rec in records:
          groups.setdefault(rec["service"], []).append(rec)
...
      if not records:
          return 0.0
+     records = list(records)
+     if not records:
+         return 0.0
      failed = sum(1 for rec in records if rec["status"] == "fail")
+     return round(failed * 100.0 / len(records), 1)
-     return round(failed * 100 // len(records), 1)
  
  
...
The complete corrected file
"""uptime_report.py - summarise service check records for a weekly report.

Each record is a dict: {"service": str, "status": "ok" | "fail", "duration": str}
where duration is a string such as "250ms", "1.5s", "2m" or "1m30s".
"""
import math
import re

_UNITS = {"ms": 0.001, "s": 1.0, "m": 60.0}
_PART = re.compile(r"(\d+(?:\.\d+)?)(ms|s|m)")

try:
    _STRING_TYPES = (basestring,)
except NameError:
    _STRING_TYPES = (str,)


def parse_duration(text):
    """Return the duration in seconds as a float. Raises ValueError on bad input."""
    if not isinstance(text, _STRING_TYPES):
        raise ValueError("bad duration: %r" % (text,))
    text = text.strip().lower()
    parts = _PART.findall(text)
    if not parts or "".join(n + u for n, u in parts) != text:
        raise ValueError("bad duration: %r" % text)
    return sum(float(n) * _UNITS[u] for n, u in parts)


def percentile(values, p):
    """Nearest-rank percentile: the smallest value with at least p percent of
    the values at or below it. p is between 0 (exclusive) and 100 (inclusive)."""
    if not values:
        raise ValueError("no values")
    if not 0 < p <= 100:
        raise ValueError("p out of range")
    ordered = sorted(values)
    if not ordered:
        raise ValueError("no values")
    rank = int(math.ceil(p / 100.0 * len(ordered)))
    rank = min(max(rank, 1), len(ordered))
    return ordered[rank - 1]


def group_by_service(records, groups={}):
    """Return {service: [records...]} keeping the input order inside each group."""
    defaults = getattr(group_by_service, "__defaults__", None)
    if not defaults:
        defaults = getattr(group_by_service, "func_defaults", ())
    default_groups = defaults[0] if defaults else None

    if groups is default_groups or groups is None:
        groups = {}
    for rec in records:
        groups.setdefault(rec["service"], []).append(rec)
    return groups


def error_rate(records):
    """Share of records with status "fail", as a percentage rounded to one decimal."""
    if not records:
        return 0.0
    records = list(records)
    if not records:
        return 0.0
    failed = sum(1 for rec in records if rec["status"] == "fail")
    return round(failed * 100.0 / len(records), 1)


def summarise(records):
    """Return one summary dict per service, sorted by service name.

    Failed checks count towards the error rate but their durations are ignored,
    because a failed check usually just reports the timeout.
    """
    out = []
    for service, recs in sorted(group_by_service(records).items()):
        times = [parse_duration(r["duration"]) for r in recs if r["status"] == "ok"]
        out.append({
            "service": service,
            "checks": len(recs),
            "error_rate": error_rate(recs),
            "p50": percentile(times, 50) if times else None,
            "p95": percentile(times, 95) if times else None,
            "slowest": max(times) if times else None,
        })
    return out


def format_report(summary):
    """Return the report as text, one line per service."""
    lines = []
    for row in summary:
        if row["p50"] is None:
            timing = "no successful checks"
        else:
            timing = "p50 %.3fs, p95 %.3fs, slowest %.3fs" % (row["p50"], row["p95"], row["slowest"])
        lines.append("%-12s %4d checks, %5.1f%% errors, %s" % (row["service"], row["checks"], row["error_rate"], timing))
    return "\n".join(lines)

Attempt 23 of 3 bugs fixed78.4 min52,692 tokens

What the model said it fixed

  • parse_duration could raise AttributeError/TypeError for non-string input instead of the documented ValueError; it now validates the input type.
  • parse_duration used \d, allowing non-ASCII digits and possible conversion errors/wrong values; it now uses ASCII digits and normalises conversion failures to ValueError.
  • percentile used ordered[rank] instead of ordered[rank - 1], making nearest-rank results one item too high and causing out-of-range crashes.
  • percentile computed rank without guarding integer division, float ceil, tiny/large p, or empty iterators; it now computes a clamped integer rank and checks the sorted result.
  • group_by_service used a mutable default dict, leaking records between calls and allowing stale/default mutation; it now creates a fresh dict for omitted/default/None groups while preserving explicit accumulators.
  • error_rate used integer floor division before rounding, truncating percentages and sometimes returning an int; it now uses floating division before rounding.
  • error_rate assumed len(records), which could crash on generators/iterators; it now counts records while iterating.

What it changed

Lines the model removed from the original script are marked -, lines it added are marked +.

  
  _UNITS = {"ms": 0.001, "s": 1.0, "m": 60.0}
+ _PART = re.compile(r"([0-9]+(?:\.[0-9]+)?)(ms|s|m)")
+ 
+ try:
+     _STRING_TYPES = (basestring,)
+ except NameError:
+     _STRING_TYPES = (str,)
- _PART = re.compile(r"(\d+(?:\.\d+)?)(ms|s|m)")
  
  
  def parse_duration(text):
      """Return the duration in seconds as a float. Raises ValueError on bad input."""
+     if not isinstance(text, _STRING_TYPES):
+         raise ValueError("bad duration: %r" % (text,))
+     try:
+         text = text.strip().lower()
+         parts = _PART.findall(text)
+         if not parts or "".join(n + u for n, u in parts) != text:
+             raise ValueError("bad duration: %r" % text)
+         return sum(float(n) * _UNITS[u] for n, u in parts)
+     except (TypeError, ValueError, OverflowError, AttributeError):
-     text = text.strip().lower()
-     parts = _PART.findall(text)
-     if not parts or "".join(n + u for n, u in parts) != text:
          raise ValueError("bad duration: %r" % text)
-     return sum(float(n) * _UNITS[u] for n, u in parts)
  
  
...
      """Nearest-rank percentile: the smallest value with at least p percent of
      the values at or below it. p is between 0 (exclusive) and 100 (inclusive)."""
+     try:
+         empty = len(values) == 0
+     except (TypeError, AttributeError):
+         try:
+             empty = bool(values) is False
+         except (TypeError, ValueError, AttributeError):
+             empty = False
+     if empty:
-     if not values:
          raise ValueError("no values")
      if not 0 < p <= 100:
          raise ValueError("p out of range")
      ordered = sorted(values)
+     if not ordered:
+         raise ValueError("no values")
+     rank = int(math.ceil(p / 100.0 * len(ordered)))
+     rank = max(1, min(rank, len(ordered)))
+     return ordered[rank - 1]
-     rank = math.ceil(p / 100 * len(ordered))
-     return ordered[rank]
  
  
  def group_by_service(records, groups={}):
      """Return {service: [records...]} keeping the input order inside each group."""
+     defaults = getattr(group_by_service, "__defaults__", None)
+     if defaults is None:
+         defaults = getattr(group_by_service, "func_defaults", ())
+     if groups is None or (defaults and groups is defaults[0]):
+         groups = {}
      for rec in records:
          groups.setdefault(rec["service"], []).append(rec)
...
  def error_rate(records):
      """Share of records with status "fail", as a percentage rounded to one decimal."""
+     try:
+         if len(records) == 0:
+             return 0.0
+     except (TypeError, AttributeError):
+         try:
+             if bool(records) is False:
+                 return 0.0
+         except (TypeError, ValueError, AttributeError):
+             pass
+ 
+     failed = 0
+     total = 0
+     for rec in records:
+         total += 1
+         if rec["status"] == "fail":
+             failed += 1
+     if total == 0:
-     if not records:
          return 0.0
+     return round(failed * 100.0 / total, 1)
-     failed = sum(1 for rec in records if rec["status"] == "fail")
-     return round(failed * 100 // len(records), 1)
  
  
...
The complete corrected file
"""uptime_report.py - summarise service check records for a weekly report.

Each record is a dict: {"service": str, "status": "ok" | "fail", "duration": str}
where duration is a string such as "250ms", "1.5s", "2m" or "1m30s".
"""
import math
import re

_UNITS = {"ms": 0.001, "s": 1.0, "m": 60.0}
_PART = re.compile(r"([0-9]+(?:\.[0-9]+)?)(ms|s|m)")

try:
    _STRING_TYPES = (basestring,)
except NameError:
    _STRING_TYPES = (str,)


def parse_duration(text):
    """Return the duration in seconds as a float. Raises ValueError on bad input."""
    if not isinstance(text, _STRING_TYPES):
        raise ValueError("bad duration: %r" % (text,))
    try:
        text = text.strip().lower()
        parts = _PART.findall(text)
        if not parts or "".join(n + u for n, u in parts) != text:
            raise ValueError("bad duration: %r" % text)
        return sum(float(n) * _UNITS[u] for n, u in parts)
    except (TypeError, ValueError, OverflowError, AttributeError):
        raise ValueError("bad duration: %r" % text)


def percentile(values, p):
    """Nearest-rank percentile: the smallest value with at least p percent of
    the values at or below it. p is between 0 (exclusive) and 100 (inclusive)."""
    try:
        empty = len(values) == 0
    except (TypeError, AttributeError):
        try:
            empty = bool(values) is False
        except (TypeError, ValueError, AttributeError):
            empty = False
    if empty:
        raise ValueError("no values")
    if not 0 < p <= 100:
        raise ValueError("p out of range")
    ordered = sorted(values)
    if not ordered:
        raise ValueError("no values")
    rank = int(math.ceil(p / 100.0 * len(ordered)))
    rank = max(1, min(rank, len(ordered)))
    return ordered[rank - 1]


def group_by_service(records, groups={}):
    """Return {service: [records...]} keeping the input order inside each group."""
    defaults = getattr(group_by_service, "__defaults__", None)
    if defaults is None:
        defaults = getattr(group_by_service, "func_defaults", ())
    if groups is None or (defaults and groups is defaults[0]):
        groups = {}
    for rec in records:
        groups.setdefault(rec["service"], []).append(rec)
    return groups


def error_rate(records):
    """Share of records with status "fail", as a percentage rounded to one decimal."""
    try:
        if len(records) == 0:
            return 0.0
    except (TypeError, AttributeError):
        try:
            if bool(records) is False:
                return 0.0
        except (TypeError, ValueError, AttributeError):
            pass

    failed = 0
    total = 0
    for rec in records:
        total += 1
        if rec["status"] == "fail":
            failed += 1
    if total == 0:
        return 0.0
    return round(failed * 100.0 / total, 1)


def summarise(records):
    """Return one summary dict per service, sorted by service name.

    Failed checks count towards the error rate but their durations are ignored,
    because a failed check usually just reports the timeout.
    """
    out = []
    for service, recs in sorted(group_by_service(records).items()):
        times = [parse_duration(r["duration"]) for r in recs if r["status"] == "ok"]
        out.append({
            "service": service,
            "checks": len(recs),
            "error_rate": error_rate(recs),
            "p50": percentile(times, 50) if times else None,
            "p95": percentile(times, 95) if times else None,
            "slowest": max(times) if times else None,
        })
    return out


def format_report(summary):
    """Return the report as text, one line per service."""
    lines = []
    for row in summary:
        if row["p50"] is None:
            timing = "no successful checks"
        else:
            timing = "p50 %.3fs, p95 %.3fs, slowest %.3fs" % (row["p50"], row["p95"], row["slowest"])
        lines.append("%-12s %4d checks, %5.1f%% errors, %s" % (row["service"], row["checks"], row["error_rate"], timing))
    return "\n".join(lines)

Attempt 33 of 3 bugs fixed68.8 min46,635 tokens

What the model said it fixed

  • percentile used the 1-based nearest rank directly as a 0-based list index, causing wrong values and IndexErrors; it now uses rank - 1.
  • percentile could produce an invalid rank from empty iterables or floating edge cases; ranks are now clamped to 1..len(values) and empty sorted input raises ValueError.
  • percentile relied on truthiness of values, which can crash for array-like inputs; it now handles arbitrary iterables safely.
  • group_by_service mutated the shared mutable default dict, leaking records between calls; default calls now use a fresh dict.
  • group_by_service could crash when groups=None; None now means "start with a fresh dict".
  • error_rate used integer floor division before rounding, truncating percentages; it now uses true floating-point division.
  • error_rate assumed records had len and could crash on unsized iterables; it now materialises such iterables when needed.

What it changed

Lines the model removed from the original script are marked -, lines it added are marked +.

      """Nearest-rank percentile: the smallest value with at least p percent of
      the values at or below it. p is between 0 (exclusive) and 100 (inclusive)."""
+     try:
+         empty_values = not values
+     except (TypeError, ValueError):
+         empty_values = False
+     if empty_values:
-     if not values:
          raise ValueError("no values")
      if not 0 < p <= 100:
          raise ValueError("p out of range")
      ordered = sorted(values)
+     if not ordered:
+         raise ValueError("no values")
+     count = len(ordered)
+     rank = int(math.ceil(float(p) * count / 100.0))
+     rank = max(1, min(rank, count))
+     return ordered[rank - 1]
-     rank = math.ceil(p / 100 * len(ordered))
-     return ordered[rank]
  
  
  def group_by_service(records, groups={}):
      """Return {service: [records...]} keeping the input order inside each group."""
+     if groups is None or groups is group_by_service.__defaults__[0]:
+         groups = {}
      for rec in records:
          groups.setdefault(rec["service"], []).append(rec)
...
  def error_rate(records):
      """Share of records with status "fail", as a percentage rounded to one decimal."""
+     try:
+         if not records:
+             return 0.0
+     except (TypeError, ValueError):
+         pass
+     try:
+         total = len(records)
+         recs = records
+     except (TypeError, ValueError):
+         recs = list(records)
+         total = len(recs)
+     if total == 0:
-     if not records:
          return 0.0
+     failed = sum(1 for rec in recs if rec["status"] == "fail")
+     return round(failed * 100.0 / total, 1)
-     failed = sum(1 for rec in records if rec["status"] == "fail")
-     return round(failed * 100 // len(records), 1)
  
  
...
The complete corrected file
"""uptime_report.py - summarise service check records for a weekly report.

Each record is a dict: {"service": str, "status": "ok" | "fail", "duration": str}
where duration is a string such as "250ms", "1.5s", "2m" or "1m30s".
"""
import math
import re

_UNITS = {"ms": 0.001, "s": 1.0, "m": 60.0}
_PART = re.compile(r"(\d+(?:\.\d+)?)(ms|s|m)")


def parse_duration(text):
    """Return the duration in seconds as a float. Raises ValueError on bad input."""
    text = text.strip().lower()
    parts = _PART.findall(text)
    if not parts or "".join(n + u for n, u in parts) != text:
        raise ValueError("bad duration: %r" % text)
    return sum(float(n) * _UNITS[u] for n, u in parts)


def percentile(values, p):
    """Nearest-rank percentile: the smallest value with at least p percent of
    the values at or below it. p is between 0 (exclusive) and 100 (inclusive)."""
    try:
        empty_values = not values
    except (TypeError, ValueError):
        empty_values = False
    if empty_values:
        raise ValueError("no values")
    if not 0 < p <= 100:
        raise ValueError("p out of range")
    ordered = sorted(values)
    if not ordered:
        raise ValueError("no values")
    count = len(ordered)
    rank = int(math.ceil(float(p) * count / 100.0))
    rank = max(1, min(rank, count))
    return ordered[rank - 1]


def group_by_service(records, groups={}):
    """Return {service: [records...]} keeping the input order inside each group."""
    if groups is None or groups is group_by_service.__defaults__[0]:
        groups = {}
    for rec in records:
        groups.setdefault(rec["service"], []).append(rec)
    return groups


def error_rate(records):
    """Share of records with status "fail", as a percentage rounded to one decimal."""
    try:
        if not records:
            return 0.0
    except (TypeError, ValueError):
        pass
    try:
        total = len(records)
        recs = records
    except (TypeError, ValueError):
        recs = list(records)
        total = len(recs)
    if total == 0:
        return 0.0
    failed = sum(1 for rec in recs if rec["status"] == "fail")
    return round(failed * 100.0 / total, 1)


def summarise(records):
    """Return one summary dict per service, sorted by service name.

    Failed checks count towards the error rate but their durations are ignored,
    because a failed check usually just reports the timeout.
    """
    out = []
    for service, recs in sorted(group_by_service(records).items()):
        times = [parse_duration(r["duration"]) for r in recs if r["status"] == "ok"]
        out.append({
            "service": service,
            "checks": len(recs),
            "error_rate": error_rate(recs),
            "p50": percentile(times, 50) if times else None,
            "p95": percentile(times, 95) if times else None,
            "slowest": max(times) if times else None,
        })
    return out


def format_report(summary):
    """Return the report as text, one line per service."""
    lines = []
    for row in summary:
        if row["p50"] is None:
            timing = "no successful checks"
        else:
            timing = "p50 %.3fs, p95 %.3fs, slowest %.3fs" % (row["p50"], row["p95"], row["slowest"])
        lines.append("%-12s %4d checks, %5.1f%% errors, %s" % (row["service"], row["checks"], row["error_rate"], timing))
    return "\n".join(lines)

The prompt

Exactly what every model got

Read the full prompt

Prompt:

The script below builds a weekly report from service check records. People who use it say the numbers are sometimes wrong, and sometimes it crashes.

Find and fix every bug. Keep the function names, their parameters and the documented behaviour as they are, and don't add dependencies.

Reply with the complete corrected file in one ```python code block. After the code block, list each bug you fixed, one line per bug.

"""uptime_report.py - summarise service check records for a weekly report.

Each record is a dict: {"service": str, "status": "ok" | "fail", "duration": str}
where duration is a string such as "250ms", "1.5s", "2m" or "1m30s".
"""
import math
import re

_UNITS = {"ms": 0.001, "s": 1.0, "m": 60.0}
_PART = re.compile(r"(\d+(?:\.\d+)?)(ms|s|m)")


def parse_duration(text):
    """Return the duration in seconds as a float. Raises ValueError on bad input."""
    text = text.strip().lower()
    parts = _PART.findall(text)
    if not parts or "".join(n + u for n, u in parts) != text:
        raise ValueError("bad duration: %r" % text)
    return sum(float(n) * _UNITS[u] for n, u in parts)


def percentile(values, p):
    """Nearest-rank percentile: the smallest value with at least p percent of
    the values at or below it. p is between 0 (exclusive) and 100 (inclusive)."""
    if not values:
        raise ValueError("no values")
    if not 0 < p <= 100:
        raise ValueError("p out of range")
    ordered = sorted(values)
    rank = math.ceil(p / 100 * len(ordered))
    return ordered[rank]


def group_by_service(records, groups={}):
    """Return {service: [records...]} keeping the input order inside each group."""
    for rec in records:
        groups.setdefault(rec["service"], []).append(rec)
    return groups


def error_rate(records):
    """Share of records with status "fail", as a percentage rounded to one decimal."""
    if not records:
        return 0.0
    failed = sum(1 for rec in records if rec["status"] == "fail")
    return round(failed * 100 // len(records), 1)


def summarise(records):
    """Return one summary dict per service, sorted by service name.

    Failed checks count towards the error rate but their durations are ignored,
    because a failed check usually just reports the timeout.
    """
    out = []
    for service, recs in sorted(group_by_service(records).items()):
        times = [parse_duration(r["duration"]) for r in recs if r["status"] == "ok"]
        out.append({
            "service": service,
            "checks": len(recs),
            "error_rate": error_rate(recs),
            "p50": percentile(times, 50) if times else None,
            "p95": percentile(times, 95) if times else None,
            "slowest": max(times) if times else None,
        })
    return out


def format_report(summary):
    """Return the report as text, one line per service."""
    lines = []
    for row in summary:
        if row["p50"] is None:
            timing = "no successful checks"
        else:
            timing = "p50 %.3fs, p95 %.3fs, slowest %.3fs" % (row["p50"], row["p95"], row["slowest"])
        lines.append("%-12s %4d checks, %5.1f%% errors, %s" % (row["service"], row["checks"], row["error_rate"], timing))
    return "\n".join(lines)

Grading

How this was graded

Hidden tests, automatic.

The three planted bugs

  • percentile reads one position too far in the sorted list
  • group_by_service uses a shared default dictionary, so earlier calls leak into later reports
  • error_rate uses whole-number division, so 33.3% becomes 33.0%
  • This challenge is too easy for these models.
The answer key, never shown to a model

Three planted bugs, and the model is not told how many:

  1. percentile reads one position too far in the sorted list, so results are one rank too high and it crashes at the top of the range.
  2. group_by_service uses a shared default dictionary, so records from earlier calls leak into later reports.
  3. error_rate uses whole-number division, so 33.3% becomes 33.0%.

Scored automatically by hidden tests: three tests for the bugs, three that check nothing else was broken. The original script fails the three bug tests; my reference fix passes all six.

Each attempt started from a fresh state with a new seed. The answer limit was 60,000 tokens, thinking included. More on the method