Indexing is cheap, fetching is expensive

The old design

Two of our SEC modules, the filing index and the 8-K item parser, used to make one request per tracked symbol. The tracked set is a whitelist: the 241 tickers we had decided to process, then. That costs 241 requests a run, and it can only ever see the symbols it asks about.

The second property was the expensive one. Before the change, the 241 whitelisted tickers were current. The other 8,948 symbols with filing history had a median newest filing of 2026-04-29, about four months stale. The Form 4 parser had produced zero rows for them in 90 days. Nothing was failing. The pipeline was doing what it was asked, for the symbols it was asked about.

Discovery scales with filings, not symbols

The SEC's filing search returns hits for a form type over a date window, 100 to a page, for every filer at once. So we stopped asking per symbol and asked per date window. The cost is one request per 100 filings, regardless of how many symbols those filings map into. We then map each hit's CIK against the full symbols table instead of the whitelist.

Measured on a single day, 2026-09-09: sec_filings went from 46 rows to 323, and sec_8k_items from 19 to 604, a 31x widening. Request cost went down. A daily run is about 15 requests and a monthly run about 200, against 241 before. Broader coverage for fewer requests is the unusual part. It only happens because the two costs were never the same quantity: the old design paid per symbol for something that scales with filings.

Fetching is the opposite

Finding out that a filing exists is a line in a search result. Reading it is one document fetch per filing, and for Form 4 that means an XML parse and a row per transaction. Those modules kept a gate. insider_trades was changed to filter to tracking_enabled explicitly, where before the gate had been implicit in what discovery bothered to find.

The arithmetic from the decision log on 2026-09-16 is why. A widened sec_filings handed the Form 4 parser 1.39M unparsed filings. At 9 requests a second, that is about 43 hours of history. Worse, the parser had a per-run cap of 5,000 filings, newest first. Market-wide intake was estimated at about 270k a year against about 33k for the whitelist, roughly 8x. The cap would have been spent on market-wide filings, and the feed we actually served would have gone from always current to competing for its own queue.

That produced the rule we now apply:

Gate on tracking_enabled when the request count scales with the symbol count. Do not gate discovery, which scales with the number of filings.

The same reasoning had already been applied to cross-issuer feeds in general. For late-filing notices, 88% of filers in a 12-month sample map through the full symbols table and none through the whitelist. Nobody in the S&P 100 files a late-filing notice, which is the point of a distress feed. For Form 144 in June 2026, 3,726 of 3,738 filings (99.7%) map through the full table against 383 (10%) through the whitelist.

The paging ceiling

Cheap discovery has a hard limit. The SEC search refuses to page past about 10,000 hits for a query. Form 4 alone runs about 3,300 a week market-wide, so any window wider than about three weeks overflows it. Truncating silently would look exactly like a quiet period. So the iterator splits a window that holds more hits than it can page and recurses on the halves. The splitting is load-bearing, not defensive. Simplified:

def hits(form, start, end):
    page = search(form, start, end, offset=0)
    if page.total > PAGE_CAP and start < end:
        mid = start + (end - start) // 2
        yield from hits(form, start, mid)
        yield from hits(form, mid + 1 day, end)
        return
    if page.total > PAGE_CAP:
        log.warning("single day over cap; truncating")
    yield from page_through(page, form, start, end)

The cap in code is set below the ceiling, at 9,000, to leave margin. A single day over the cap cannot be split further. It logs a warning and takes what the search will give, rather than dropping the day without saying so.

What the rule cost us: the Form 4 gate

The gate had a cost, and it showed up in a different feed. Form 144 notices cover about 11.4k symbols. With Form 4 parsed for only 534, a notice with no Form 4 behind it could not distinguish never sold from never ingested. So on 2026-09-30 the gate on insider_trades was removed and the queue joins on active symbols alone.

Before doing that we re-measured, from the SEC's quarterly filing index (2025 Q1 to 2026 Q2). Market-wide Form 4 and 4/A intake is about 187k accessions a year, not the 270k estimated on 2026-09-16, which probably counted per-symbol rows. Per day the mean is 756, the median 641, the 99th percentile 2,151 and the peak 3,535 (2025-03-04). At 9.5 requests a second that is about 1.3 minutes on a mean day and about 6 on the peak. The peak fits inside the 12,000 per-run limit.

The per-run cap worry was handled by ordering, not by the gate. The queue has three tiers, newest first within each: anything filed in the last 7 days for any symbol, then the rest of the history for tracked symbols, then the rest for untracked symbols. Tracked-first alone was tried and rejected. Tracked symbols still had an 88.5k-filing backlog from 2006 to 2019, which drains at 12k a day, and it would have held today's untracked Form 4s behind it for about a week.

The historical backlog drains through the daily cron: about 988k untracked and 88k tracked unparsed accessions as of 2026-09-30, about 90 daily runs. Fresh filings were never behind it.

Where it stands

As of 2026-11-01, insider_trades holds 2,282,045 rows across 5,741 symbols. The tracked whitelist is 534 symbols out of 11,622 active ones. Both are point-in-time. The row count grows with the backlog drain.

# Form 4 transactions for any covered issuer, not just the whitelist
curl -H "Authorization: Bearer $HEDGEFRIEND_KEY" \
  "https://api.hedgefriend.dev/v1/sec/insider-trades/TICKER?from=2026-09-01"

Caveats

• The 46 to 323 and 19 to 604 figures are one day, 2026-09-09, not an average. They show the size of the effect, not a steady-state ratio.
• The 270k a year Form 4 estimate was wrong. The 187k figure replaced it. The 43-hour figure is for the 1.39M-filing backlog on 2026-09-16, not for a year of intake.
• Request rates (9 and 9.5 a second) are what we have run against, not a published limit. Backlog and row counts are point-in-time and move daily.
• Search hits carry a file description only for some forms. 8-K, Form 4 and 10-K have one; 13F-HR and 1-A do not. Some descriptive fields are slightly thinner than the per-company route gave us.
• Coverage is still uneven across feeds. Feeds that fetch one large document per filing, such as parsed 10-K risk factors, remain gated to tracked symbols.

What to take from it

The question to ask of any pipeline is what its request count scales with. If it scales with filings, widen it, because the symbol list is the wrong thing to gate. If it scales with symbols or with document size, gate it, and when the gate starts hiding data you need, replace it with ordering before you remove it.