Cache API Calls Politely: a Reusable Fetch Helper (Python)

The fastest way to get your data access shut off is to hammer a free API in a loop while you debug. The fix is simple and you should build it before anything else: a fetch helper that caches every response to disk and rate-limits itself. Write once, reuse in every project. The full script is reproduced below (it uses the shared helper named in the sources below).

The whole helper

It's about 15 lines of standard library. Hash the URL to a filename; if a fresh cache file exists, return it; otherwise wait politely, fetch, save, return:

import os, json, time, hashlib, urllib.request
CACHE = "cache"; os.makedirs(CACHE, exist_ok=True)
_last = [0.0]; MIN_INTERVAL = 1.0

def cached_get(url, ttl_days=7):
    path = os.path.join(CACHE, hashlib.sha1(url.encode()).hexdigest() + ".json")
    if os.path.exists(path) and (time.time() - os.path.getmtime(path))/86400 <= ttl_days:
        return json.load(open(path, encoding="utf-8")), "HIT"
    wait = MIN_INTERVAL - (time.time() - _last[0])     # rate-limit live calls
    if wait > 0: time.sleep(wait)
    _last[0] = time.time()
    req = urllib.request.Request(url, headers={"User-Agent": "MyProject/1.0 (you@example.com)"})
    data = json.loads(urllib.request.urlopen(req, timeout=20).read())
    json.dump(data, open(path, "w", encoding="utf-8"))
    return data, "MISS"

See it work

Call the same URL twice. The first is a network round-trip; the second is a disk read:

call 1: MISS  50 games  (262 ms)
call 2: HIT   50 games  (12 ms)

That's a 20x speedup and, more importantly, zero extra load on the API. During development you'll re-run a script dozens of times; with caching, only the first run touches the network. Re-run it again and both calls are HITs.

Why each piece matters

• Caching makes iteration fast and keeps you off rate limits. A hashed URL is a safe, unique filename for any request.
• A TTL (ttl_days) means cached data refreshes eventually — set it short for live scores, long for finished seasons.
• Rate-limiting (the MIN_INTERVAL sleep) spaces out live calls so you're a good guest, even inside a tight loop.
• A descriptive User-Agent with contact info is basic etiquette; some services block blank or default agents outright.

Going further

• Handle errors gracefully — wrap the request in try/except and return None on failure so one bad URL doesn't crash a batch (that's what the site's _fetch.py does).
• Respect robots and terms. Caching is about being polite to APIs you're allowed to use; it is not a tool for evading rate limits, paywalls, or anti-bot measures. Don't.
• Add jitter to the interval if you're running many scripts in parallel.

This is the least glamorous tutorial on the site and maybe the most important. Build the helper first; everything else gets faster and friendlier.