Cache API Calls Politely: a Reusable Fetch Helper (Python)

The fastest way to get your data access shut off is to hammer a free API in a loop while you debug. The fix is simple and you should build it before anything else: a fetch helper that caches every response to disk and rate-limits itself. Write once, reuse in every project. Full code: scripts/polite-api-caching-python.py (and the site's own scripts/_fetch.py).

The whole helper

It's about 15 lines of standard library. Hash the URL to a filename; if a fresh cache file exists, return it; otherwise wait politely, fetch, save, return:

import os, json, time, hashlib, urllib.request
CACHE = "cache"; os.makedirs(CACHE, exist_ok=True)
_last = [0.0]; MIN_INTERVAL = 1.0

def cached_get(url, ttl_days=7):
    path = os.path.join(CACHE, hashlib.sha1(url.encode()).hexdigest() + ".json")
    if os.path.exists(path) and (time.time() - os.path.getmtime(path))/86400 <= ttl_days:
        return json.load(open(path, encoding="utf-8")), "HIT"
    wait = MIN_INTERVAL - (time.time() - _last[0])     # rate-limit live calls
    if wait > 0: time.sleep(wait)
    _last[0] = time.time()
    req = urllib.request.Request(url, headers={"User-Agent": "MyProject/1.0 (you@example.com)"})
    data = json.loads(urllib.request.urlopen(req, timeout=20).read())
    json.dump(data, open(path, "w", encoding="utf-8"))
    return data, "MISS"

See it work

Call the same URL twice. The first is a network round-trip; the second is a disk read:

call 1: MISS  50 games  (262 ms)
call 2: HIT   50 games  (12 ms)

That's a 20x speedup and, more importantly, zero extra load on the API. During development you'll re-run a script dozens of times; with caching, only the first run touches the network. Re-run it again and both calls are HITs.

Why each piece matters

• Caching makes iteration fast and keeps you off rate limits. A hashed URL is a safe, unique filename for any request.
• A TTL (ttl_days) means cached data refreshes eventually — set it short for live scores, long for finished seasons.
• Rate-limiting (the MIN_INTERVAL sleep) spaces out live calls so you're a good guest, even inside a tight loop.
• A descriptive User-Agent with contact info is basic etiquette; some services block blank or default agents outright.

Going further

• Handle errors gracefully — wrap the request in try/except and return None on failure so one bad URL doesn't crash a batch (that's what the site's _fetch.py does).
• Respect robots and terms. Caching is about being polite to APIs you're allowed to use; it is not a tool for evading rate limits, paywalls, or anti-bot measures. Don't.
• Add jitter to the interval if you're running many scripts in parallel.

This is the least glamorous tutorial on the site and maybe the most important. Build the helper first; everything else gets faster and friendlier.