Here's some extra implementation stuff I didn't include above. Most of my dev friends seem the most interested in this part.
The whole thing essentially runs on 1 VM with Playwright, Postgres and a handful of systemd timers. It's several nightly batch jobs (hourly to disperse a growing user-base across many hours, hopefully ;)).
The part that took forever to get right was the workflow for determining if a job was actually live on the company's ATS system. There is SO much garbage on the main job boards. The naive version checks the HTTP status, but a dead Workday or aggregator listing happily returns 200. So I'd get confident LLMs excited to link me to a "this position has expired" page... passing garbage is obviously never ok for this to work. That's why I had to go "real browser". It plays nice in the marketing but that gating is a necessity that the whole thing is kind of built around. It waits for the page to actually render, uses LLM intelligence to read the whole body and fails if it can't confirm the job is real. It's kind of awesome.
Here's some extra implementation stuff I didn't include above. Most of my dev friends seem the most interested in this part.
The whole thing essentially runs on 1 VM with Playwright, Postgres and a handful of systemd timers. It's several nightly batch jobs (hourly to disperse a growing user-base across many hours, hopefully ;)).
The part that took forever to get right was the workflow for determining if a job was actually live on the company's ATS system. There is SO much garbage on the main job boards. The naive version checks the HTTP status, but a dead Workday or aggregator listing happily returns 200. So I'd get confident LLMs excited to link me to a "this position has expired" page... passing garbage is obviously never ok for this to work. That's why I had to go "real browser". It plays nice in the marketing but that gating is a necessity that the whole thing is kind of built around. It waits for the page to actually render, uses LLM intelligence to read the whole body and fails if it can't confirm the job is real. It's kind of awesome.
Happy to go deeper.