How a job board stays fresh without guessing

Freshness on a job board has two halves, and almost everyone talks about one of them. New postings arriving is the half that is easy to measure and easy to advertise. Closed postings leaving is the half that decides whether your application gets a reply.

A dead posting costs you an application, a wait and a silence. This is how the second half works here, in enough detail that you could argue with it.

Reading the source, not a copy of it

Every employer on this board is read through their own applicant tracking system — Greenhouse, Lever, Workday, Ashby, SmartRecruiters and more than twenty others. One request returns that employer’s whole current listing, which is what makes the rest of this affordable: the question “what is on this board right now” costs one call, not one per posting.

Those requests are conditional. We send back the validator the board gave us last time, and a board with nothing new answers “not modified” without sending the listing again. That is not only cheaper — it is a positive statement that the membership has not changed, which is a different and more useful thing than a listing that happens to look the same.

A schedule that adapts to each employer

There is no nightly sweep. A scheduler wakes every five minutes, takes the boards that are due, and reads them — so a busy employer and a dormant one are not waiting on the same clock.

Each board carries its own interval, between 15 minutes and a day, and it moves on what that board actually does:

Two quiet passes rather than one, because a six-hourly employer looks dormant five times out of six and relaxing on the first miss would slow every board on the site. The jitter applied to each interval only ever shortens it, so a 15-minute board stays a 15-minute board rather than drifting to 18.

Being a polite guest

None of these boards asked to be read. The crawl holds a token bucket per host — about two requests a second with a small burst — and that budget is shared between plain API calls and the occasional rendered page. The host cannot tell those two apart, so neither do we. Budgeting only the API path would leave a board politely rate-limited on one channel while a browser hammered the same machine on another.

When a board asks us to wait, we wait exactly as long as it asked. And when our own limiter is the thing that refused a request, that board goes back in the queue a few minutes later with nothing recorded against it — our congestion is not the employer’s fault, and filing it as a failure is how one busy host came to look like the worst-behaved board on the site.

Absence is not evidence

Here is the rule that matters most, and the one that looks wrong until you have been bitten by it. A posting missing from a listing proves nothing.

A posting can vanish from one reading because the employer reskinned their board, because a URL convention drifted, because a firewall answered with an empty list, or because the page timed out halfway. All four look identical to a role that was filled yesterday. Every mass-expiry incident in this kind of system comes from treating one absence as proof.

So a posting is retired only when both of these are true:

• It has been missing from two consecutive complete listings, not one.
• A direct check of the posting itself says it is gone — the page answers not-found, or the employer’s own page says the role has been filled or closed.

Three further refusals sit on top of that:

• An empty listing cancels the whole pass. If a board returns nothing at all, we do not conclude that the employer closed every role overnight. One bad night must not retire a live board.
• A truncated listing may add postings but never retire them. Boards that page newest-first put their oldest roles past the cap, so treating a capped listing as complete would wipe the older half of every large employer.
• Anything ambiguous leaves the posting exactly as it was. Some systems answer a removed posting with a search page, a login wall or a cheerful success code, which is indistinguishable from a live one.

That asymmetry is deliberate, and it comes from which mistake you can see. Wrongly retiring a live posting costs somebody a real opportunity and is invisible — nobody reports a job they never saw. Leaving a dead one up costs one wasted check. So every uncertain answer resolves toward leaving it alone, and only positive evidence of removal takes a posting down.

Nothing is ever deleted

A retired posting does not disappear from the database; it changes status. Expired, not found, or filtered — each meaning something different about how it ended. The page stops being served and answers not-found so search engines drop it, and the checker can still tell you, truthfully, that the posting you were looking at is closed.

Keeping the row is also what makes the repost count possible.

The same role, posted again

Two postings are treated as the same posting when the employer, the normalised title, the first location and the publication date all match. That last field is the interesting one: it means a role advertised again next quarter is a new opening rather than a duplicate of the old one.

That is the right answer for the board — a filled role genuinely reopened is a real vacancy — and it is exactly what makes the repost signal countable. The same employer advertising the same role in the same city six times in a year produces six rows, and six rows is the evidence line you see on the posting.

Deterministic on the way in

No language model touches the ingest path. Locations, salaries, employment types and the duplicate key are parsed by rules, which is what lets the same posting produce the same row twice and what makes an extraction bug findable rather than mysterious.

A model is used in two narrow places, both off the hot path: summarising a description when something actually asks for it, and as the last resort when profiling a company, where its guess is checked against the company’s own site before anything is stored.

What this buys you

Postings appear sooner, because there is no syndication step to wait for. Closed roles leave faster, because leaving is something the system actively checks rather than assumes. And the same history that makes both of those true is what the hiring signal is computed from — how we score every posting explains what we do with it, and what a ghost job is explains why it matters.