Web Scraping Specialist
Keep public social data flowing when platforms change the rules overnight — and when they change them again on Friday.
- Location
- Remote — worldwide
- Employment
- Full-time
- Salary
- £55,000 – £85,000 / year
- Hours
- Full-time · flexible async hours
- Visa sponsorship
- Case-by-case — happy to discuss during hiring.
Competitive for role and experience. Final offer depends on location and background.
About the role
Social Fetch normalises public social data into a single REST API. Our customers integrate once and read `followerCount` whether the underlying platform called it something else. When Instagram changes a field name or TikTok rotates an anti-bot challenge, they shouldn't find out from a broken cron job.
You'll own the fetch layer — researching how platforms serve data, implementing and maintaining scrapers, and hardening them when things break in production. That covers profiles, posts, search, comments, transcripts, and ad-library lookups across TikTok, Instagram, YouTube, Reddit, LinkedIn, Facebook, and more. The stack is TypeScript throughout: monorepo, Effect for the fetch layer's control flow, Playwright where we need a browser, plain HTTP where we don't.
Team & how we work
You'll work with the engineers who built the API and support who see failures first. When a platform drifts, support often spots the pattern before monitoring — Slack threads with requestIds, sample URLs, and "this worked yesterday" context.
Fully remote, flexible hours. Incidents don't respect timezones, so sometimes an urgent fix lands outside your usual window — day-to-day work is async. Write up what you found, open a PR, leave notes. No dedicated SRE team; reliability is everyone's job, and yours most of all.
What you'll do
- Build and maintain scrapers and integrations for major social platforms — profiles, posts, search, media, comments, engagement metrics, and platform-specific endpoints.
- Diagnose fetch failures: HTTP errors, schema drift, rate limits, bot detection, geo blocks, CAPTCHAs, and responses that look fine but parse to garbage.
- Design fetch strategies that survive bad days — retries with backoff, fallbacks between mobile and web APIs, session handling, proxy rotation when it's worth the cost.
- Keep normalized API field mappings stable when platforms rename or nest fields differently.
- Write internal notes on platform quirks (auth flows, pagination traps, fields that lie) so the next person doesn't start from zero.
- Watch production quality signals — error rates, empty fields, latency spikes — and fix regressions before customers open tickets.
- Review support escalations that smell like platform drift, not user error.
What we're looking for
- Several years of hands-on web scraping experience. Tutorial-level Playwright scripts aren't enough — we need someone who's been paged when a site changed its HTML.
- Real familiarity with anti-bot systems, browser automation (Playwright/Puppeteer), HTTP clients, and parsing messy HTML and JSON (including data hidden in script tags).
- Comfort with production debugging: logs, reproducing a failing URL, shipping a fix, watching the error rate drop.
- Strong TypeScript or JavaScript — our codebase is TypeScript and you'll be in it daily.
- Pragmatic instincts about public data, ToS grey areas, and when to stop digging.
- Ability to work independently in a remote, async team — you unblock yourself, document what you learned, and ask for help when you're stuck.
Nice to have
- Experience maintaining scrapers across many platforms, not just one site you know inside out.
- Familiarity with proxy providers, CAPTCHA services, and reverse-engineering mobile or web API calls.
- Prior work at a data or API product company where uptime mattered to paying customers.
- Exposure to observability tooling — structured logs, request IDs, error tracking.
- Experience with Effect (Effect-TS) or similar typed-effect/functional TypeScript libraries.
Benefits
Full-time team members receive:
- Flexible hours — Async-first. Ship your work; we don't track screen time or mandate overlap windows unless a role specifically needs it.
- Unlimited PTO — Take the time you need. Give the team a heads-up for longer breaks so support and on-call handoffs stay covered.
- Equipment stipend — Home office setup — desk, monitor, chair, whatever actually makes you productive.
- Learning budget — Courses, books, and conferences tied to your role. Scraping conferences, writing workshops, support tooling — all fair game.
- Health benefits — Health coverage for full-time team members. Details and eligibility depend on location; we walk through it during hiring.
How to apply
- Put the role title in the email subject line.
- Include a short intro and relevant experience.
- Link to GitHub or examples of scrapers you've built or maintained (don't share anything proprietary).
- Optional: describe one platform change that broke your scraper and how you fixed it.
GitHub repos, blog posts, or a brief write-up on the hardest scraping problem you've solved — the weirder the platform behaviour, the better.