> **For coding agents and LLMs:** This is one page from the Social Fetch docs (markdown export). The sections below mirror the orientation block in [`/llms.txt`](https://www.socialfetch.dev/llms.txt); use [`/llms.json`](https://www.socialfetch.dev/llms.json) when you need a structured operation inventory. The catalog covers documented operations with on-site reference pages.

## This page

- **On-site (HTML):** [https://www.socialfetch.dev/docs/api/v1/reddit/search/get](https://www.socialfetch.dev/docs/api/v1/reddit/search/get)
- **Markdown (.mdx) URL:** [https://www.socialfetch.dev/docs/api/v1/reddit/search/get.mdx](https://www.socialfetch.dev/docs/api/v1/reddit/search/get.mdx)

## API base URL and authentication

- **API origin (from OpenAPI `servers`):** `https://api.socialfetch.dev`
- **Authentication:** send `x-api-key: sfk_...` on `/v1/**` routes unless the operation is explicitly anonymous (check OpenAPI `security`, the [API reference hub](https://www.socialfetch.dev/docs/api.mdx), [`/llms.txt`](https://www.socialfetch.dev/llms.txt), or [`/llms.json`](https://www.socialfetch.dev/llms.json) for each route).
- **OpenAPI JSON:** [https://www.socialfetch.dev/openapi.json](https://www.socialfetch.dev/openapi.json)

## Recommended docs entrypoints (this site)

- [Documentation overview](https://www.socialfetch.dev/docs.mdx) — top-level orientation (markdown).
- [Quickstart](https://www.socialfetch.dev/docs/quickstart.mdx) — authenticate with `x-api-key`, validate auth with `whoami`, and understand the JSON envelope.
- [SDK](https://www.socialfetch.dev/docs/sdk.mdx) — official TypeScript SDK guide, including `SocialFetchClient`, `Result`, and `unwrap()`.
- [SDK reference](https://www.socialfetch.dev/docs/sdk-reference.mdx) — exhaustive SDK method inventory and route mapping for agents, tooling, and power users.
- [Choose the right endpoint](https://www.socialfetch.dev/docs/choose-endpoint.mdx) — task-oriented route selection for smoke tests, profiles, list endpoints, and single-item lookups.
- [Capability matrix](https://www.socialfetch.dev/docs/capability-matrix.mdx) — fast comparison of identifiers, pagination, outcomes, media download, and SDK coverage.
- [`/llms.json`](https://www.socialfetch.dev/llms.json) — structured machine-readable operation inventory with parameter names, pagination, outcomes, credits, and SDK mapping.
- [API reference hub](https://www.socialfetch.dev/docs/api.mdx) — human-friendly index of operations with links into generated pages.
- [Errors](https://www.socialfetch.dev/docs/errors.mdx) — shared error envelope and HTTP status guidance.
- [Credits](https://www.socialfetch.dev/docs/credits.mdx) — metering, `402`, and planning batch jobs.
- Outcome semantics such as `found`, `not_found`, and `private` are documented in [Errors](https://www.socialfetch.dev/docs/errors.mdx) and on operation pages when present in the OpenAPI contract.

## Markdown docs convention

- Every docs page has a markdown twin: append **`.mdx`** to the docs pathname (for example `/docs/quickstart` → `/docs/quickstart.mdx`).
- Agents that send `Accept: text/markdown` on `/docs/**` HTML URLs may receive markdown directly (same URL, `Vary: Accept`).

---
# Search Reddit posts (https://www.socialfetch.dev/docs/api/v1/reddit/search/get)

## Summary

Search for posts on Reddit.

**Tags:** `Reddit`

## HTTP

- **Method:** GET
- **Path:** `/v1/reddit/search`
- **Base URL:** `https://api.socialfetch.dev`

## Capability summary

- **SDK mapping:** `client.reddit.search({ query, sortBy?, timeframe?, cursor?, trim? })`
- **Pagination:** cursor via `cursor`, next cursor: `data.page.nextCursor`, has more: `data.page.hasMore`
- **Ordering / list behavior:** Supported sort options: relevance, new, top.
- **Trimmed response mode:** supported via `trim=true`

## Credits

- **Base:** 1 credit per successful lookup.
- **Maximum on success (200):** 1 credit.
- **Normalization failure (502):** 1 credit charged.
- **Authoritative field:** `meta.creditsCharged`.

## Authentication

- **`x-api-key`**: API key (`sfk_...`)

## Parameters

### `query` (query)

- **Required:** yes
- **Constraints:** type `string`; minLength: 1; maxLength: 512
- **Description:** Search query text for public Reddit posts.

### `sortBy` (query)

- **Required:** no
- **Constraints:** type `string`; enum: relevance, new, top
- **Description:** Optional sort order for search results.

### `timeframe` (query)

- **Required:** no
- **Constraints:** type `string`; enum: all, day, week, month, year
- **Description:** Optional time range filter for search results.

### `cursor` (query)

- **Required:** no
- **Constraints:** type `string`; minLength: 1
- **Description:** Opaque pagination cursor returned by a previous response.

### `trim` (query)

- **Required:** no
- **Constraints:** type `boolean`
- **Description:** Whether to request a smaller response shape when available.

## Behavior notes

- **`sortBy`**: Optional sort order for search results.
- **`trim`**: Whether to request a smaller response shape when available.

## Pagination

This endpoint uses **cursor-based pagination** via the `cursor` query parameter.

- Read **hasMore** from `data.page.hasMore`.
- When that value is `true`, read **nextCursor** from `data.page.nextCursor` and pass it as the `cursor` query parameter on the **next** request (URL-encode when building a query string).
- Omit `cursor` on the **first** request.
- Stop when **hasMore** is `false` or **nextCursor** is null (end of list).

## Responses (status codes)

- **200**: Search results for the requested query.
- **400**: Invalid query parameters
- **401**: Missing or invalid API key
- **402**: Insufficient credits
- **500**: Unexpected or billing error
- **502**: Lookup could not be completed from the response (unexpected or invalid data).
- **503**: Service temporarily unavailable; safe to retry with backoff.

## Response body (200)

Search results for the requested query.

### Field outline

- **data** (required) — type `object`. Endpoint-specific response payload.
  - **query** (required) — type `string`. Search query string evaluated for this response.
  - **posts** (required) — type `array`. Posts returned for this page.
    - _items:_
      - **id** (required) — type `string`; minLength: 1. Stable post identifier.
      - **name** (required) — type `string`; nullable. Full post name when available.
      - **title** (required) — type `string`; nullable. Post title when available.
      - **author** (required) — type `string`; nullable. Author username when available.
      - **subreddit** (required) — type `string`; nullable. Subreddit name without a leading r/ when available.
      - **subredditPrefixedName** (required) — type `string`; nullable. Prefixed subreddit label such as r/example when available.
      - **url** (required) — type `string`; nullable. Canonical public post URL when available.
      - **permalink** (required) — type `string`; nullable. Public permalink URL when available.
      - **bodyText** (required) — type `string`; nullable. Post body text when available.
      - **score** (required) — type `integer`; minimum: 0; nullable. Post score when available.
      - **upvotes** (required) — type `integer`; minimum: 0; nullable. Upvote count when available.
      - **upvoteRatio** (required) — type `number`; minimum: 0; maximum: 1; nullable. Upvote ratio between 0 and 1 when available.
      - **commentCount** (required) — type `integer`; minimum: 0; nullable. Comment count when available.
      - **createdAt** (required) — type `string`; nullable. ISO-8601 creation timestamp when available.
      - **isVideo** (required) — type `boolean`. Whether the post is a video post.
      - **subredditSubscriberCount** (optional) — type `integer`; minimum: 0; nullable. Subreddit subscriber count when available.
  - **totalResults** (required) — type `integer`; minimum: 0. Number of posts returned in this response.
  - **page** (required) — type `object`. Pagination information for the current response.
    - **nextCursor** (required) — type `string`; nullable. Cursor to pass in the next request when another page exists.
    - **hasMore** (required) — type `boolean`. Whether another page of results is available.
- **meta** (required) — type `object`. Metadata describing the request and billing outcome.
  - **requestId** (required) — type `string`; minLength: 1. Unique request identifier for tracing this API call.
  - **creditsCharged** (required) — type `integer`; minimum: 0. Credits charged for this request.
  - **version** (required) — type `string`; enum: v1. Public API version that served the response.

### Example JSON (results)

```json
{
  "data": {
    "query": "webscraping",
    "posts": [
      {
        "id": "1sa1sgp",
        "name": "t3_1sa1sgp",
        "title": "A cry to the Webscraping Gods, need web scrapping pro to save us",
        "author": "smokedX",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1sa1sgp/a_cry_to_the_webscraping_gods_need_web_scrapping/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1sa1sgp/a_cry_to_the_webscraping_gods_need_web_scrapping/",
        "bodyText": "we’ve hit a wall, the site we've been scraping for a year has updated their security and we can't figure out a bypass..\n\nour system keeps being labeled as automated/system-generated traffic by the vendor, and the usual fixes we tried did not resolve it permanently. This is now a minor bottleneck for us.\n\nwe’re not looking for short-term tricks. we need to find someone senior who can look at the stack end to end and tell us whether the problem is browser workflow replication, session/cookie/csrf/state handling, infrastructure design\n\nfor anyone who has hired outside help for problems like this, where did you find the right person? what kind of specialist should we be searching for?\n\ntrying to find someone who can come in, diagnose the root cause, and help us build something durable and scalable, rather than wasting time on trial and error.",
        "score": 43,
        "upvotes": 43,
        "upvoteRatio": 0.97,
        "commentCount": 40,
        "createdAt": "2026-04-02T00:02:33.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1t3eal2",
        "name": "t3_1t3eal2",
        "title": "Please help 🥺🙏 | Web Scraping task",
        "author": "FusionMaster-04",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1t3eal2/please_help_web_scraping_task/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1t3eal2/please_help_web_scraping_task/",
        "bodyText": "I’m working on a web scraping task where I need to collect structured data like company name, category, turnover, and basic details from EPC-related listings.\n\nI’m facing a few technical challenges and would appreciate guidance:\n\n1. The website is React-based, so content loads dynamically. What is the best approach to reliably extract such data (Selenium, Playwright, or something else)?\n2. Some elements (like lists) have inconsistent HTML structure (e.g., `<ul>` tags sometimes with classes, sometimes without, sometimes multiple on the same page). How do you design a robust parser for this?\n3. There are “Load more” or dynamically loaded sections. What is the recommended way to handle these in automation scripts?\n4. How do you structure scraping workflows to minimize failures due to layout changes?\n\nI am looking for a **code-based, free solution** (preferably Python).\n\nAny guidance, best practices, or learning resources would help.",
        "score": 5,
        "upvotes": 5,
        "upvoteRatio": 0.86,
        "commentCount": 31,
        "createdAt": "2026-05-04T10:57:54.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1q6pxwn",
        "name": "t3_1q6pxwn",
        "title": "Just Started Web Scraping — Is This a Good Start?",
        "author": "franik33",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1q6pxwn/just_started_web_scraping_is_this_a_good_start/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1q6pxwn/just_started_web_scraping_is_this_a_good_start/",
        "bodyText": "Hi everyone,\n\nI started getting into web scraping about 3–4 days ago. I already have some solid experience with Python, and my first scraping project was a public website. I managed to collect around 7,000 records and everything worked as expected.\n\nI’m curious whether this is considered a decent start for someone new to scraping, or if it’s fairly basic stuff.  \nAlso, I’d like to hear honest opinions: is web scraping still worth investing time in today (for projects, automation, or monetization), or is it becoming a waste of time due to market saturation and restrictions?\n\nAny real-world experiences or insights would be appreciated.\n\nThanks in advance.",
        "score": 34,
        "upvotes": 34,
        "upvoteRatio": 0.9,
        "commentCount": 49,
        "createdAt": "2026-01-07T20:01:30.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1pt0gyd",
        "name": "t3_1pt0gyd",
        "title": "\"We backed up Spotify (~300TB)\"",
        "author": "t-2yrs",
        "subreddit": "Piracy",
        "subredditPrefixedName": "r/Piracy",
        "url": "https://www.reddit.com/r/Piracy/comments/1pt0gyd/we_backed_up_spotify_300tb/",
        "permalink": "https://www.reddit.com/r/Piracy/comments/1pt0gyd/we_backed_up_spotify_300tb/",
        "bodyText": "Context: [https://annas-archive.org/blog/backing-up-spotify.html](https://annas-archive.org/blog/backing-up-spotify.html)",
        "score": 21221,
        "upvotes": 21221,
        "upvoteRatio": 0.98,
        "commentCount": 495,
        "createdAt": "2025-12-22T14:09:17.000Z",
        "isVideo": true,
        "subredditSubscriberCount": 2820831
      },
      {
        "id": "bm31i0",
        "name": "t3_bm31i0",
        "title": "An analysis on the question whether Riot buffs champions which get a new skin (with quite some graphs, data available from webscraping patch notes and leagueofgraphs if you want to do your own analysis)",
        "author": "giantZorg",
        "subreddit": "leagueoflegends",
        "subredditPrefixedName": "r/leagueoflegends",
        "url": "https://www.reddit.com/r/leagueoflegends/comments/bm31i0/an_analysis_on_the_question_whether_riot_buffs/",
        "permalink": "https://www.reddit.com/r/leagueoflegends/comments/bm31i0/an_analysis_on_the_question_whether_riot_buffs/",
        "bodyText": "# Tl,dr: No if you look at winrate, but it's way more likely for them to show up in patch notes.\n\n# Introduction:\n\nHello all. This post is inspired by [u/PriagDE](https://www.reddit.com/user/PriagDE)'s post found [here](https://www.reddit.com/r/leagueoflegends/comments/bkwdqy/history_of_buffs_and_nerfs_and_their_relation_to/) which tried to answer whether Riot buffs champions which will get a new skin. It is a separate post as it has a completely different method and covers a wider array of questions.\n\nTo me as a person, I'm a data scientist working in the retail business with a low number of projects at the time so I used this opportunity to train a little my webscraping skills. I post the results here as I find them interesting and hope you do too.\n\nYou can find all the data, scripts and figures [here](https://github.com/giantZorg/LoL_Analyse_Winrate_Skins) if you want to use the data for your own analysis or scrape something else from the patch notes.\n\n# Table of contents:\n\n* Webscraping of data from leagueofgraphs and the official patch notes\n* Overview on the number of champion changes per patch and skins released\n* Relation between the number of champion changes and skins released\n* Relationship between released skins and winrate changes\n* Final words\n\n# Webscraping of data from leagueofgraphs and the official patch notes\n\nThe first part I needed to do was gather data. To decide whether a champion got buffed when he gets a new skin, I will take the difference between the winrate 20 days before and 20 days after the patch got released. The 20 days are chosen in order to let the winrate settle a bit after patch changes.\n\nThis means I need to gather the patch notes history as well as the winrate of all champions. As mentioned by [u/PriagDE](https://www.reddit.com/user/PriagDE), upcoming skins were only included from patch 4.8 on, so I gathered the patch notes starting from 4.8. This matches nicely the available history on the winrate on leagueofgraphs which starts at the moment approximately at the beginning of season 4. The script which I made to do so is this: [lol\\_skins\\_winrate.py](https://github.com/giantZorg/LoL_Analyse_Winrate_Skins/blob/master/lol_skins_winrate.py). Feel free to use it if you want to do something else with it. It is by no means optimized, but works also behind a proxy. You will need to get a copy of geckodriver to use Firefox as is done in the script.\n\nI also collected the data on popularity and banrate if someone is interested. The saved data files used in the second step are these: [lol\\_champions\\_bannrate.csv](https://github.com/giantZorg/LoL_Analyse_Winrate_Skins/blob/master/lol_champions_bannrate.csv), [lol\\_champions\\_beliebtheit.csv](https://github.com/giantZorg/LoL_Analyse_Winrate_Skins/blob/master/lol_champions_beliebtheit.csv), [lol\\_champions\\_winrate.csv](https://github.com/giantZorg/LoL_Analyse_Winrate_Skins/blob/master/lol_champions_winrate.csv) and [lol\\_patch\\_daten.csv](https://github.com/giantZorg/LoL_Analyse_Winrate_Skins/blob/master/lol_patch_daten.csv) containing the ban rate, popularity, win rate and patch data (which patch was released when with which champion getting changes and which new skins).\n\nTake note that I didn't include the champion changes in 8.23(?) where they changed the runes system and changed base values for all the champions. In addition, I make no difference between skins and chromas released as Riot has an incentive to sell both to you. The skin list should be complete with the exception of some Urgot chromas (I think).\n\n# Overview on the number of champion changes per patch and skins released\n\nThe complete analysis script can be found here: [lol\\_skins\\_nach\\_patches.R](https://github.com/giantZorg/LoL_Analyse_Winrate_Skins/blob/master/lol_skins_nach_patches.R). Again, feel free to use to change it to answer your own questions if you want. It is again not optimized for speed, but for quick development (to all R users, I know there are too many for loops, but they do the job and I wanted to get it done quickly).\n\nSo the first question I wanted to get an answer is how match each champion shows up in the patch notes. This is not completely fair of course as newer released champs have less chances to appear in them, so keep this in mind. The table looks like this:\n\n[Number of champion changes in the patch notes from 4.8 to 9.9. The total number of considered patches is 121.](https://preview.redd.it/uaoadd5m7yw21.png?width=545&format=png&auto=webp&s=292cba413df4b5116a6e10a099df91552b9e815e)\n\nWe can see our favorite problem childs Azir and Ryze showing up in the patch notes a lot. Given that the total number of considered patches is 121, Azir shows up in more than a quarter of them. On the other end of the scale, with have champions like Diana, Blitzcrank or Nami which remain rather stable over the whole time.\n\nThe following two graphs show the number of champion changes over the patches as well as the number of skins/chromas released over the patches. The red line is calculated using a cubic spline smoother with the degrees of freedom determined by cross validation:\n\n[Number of champion changes per patch.](https://preview.redd.it/jkyfwjea8yw21.png?width=1000&format=png&auto=webp&s=7ed31428178a8019e52fddda30479782f415daa3)\n\n[Number of skins\\/chromas released per patch.](https://preview.redd.it/7p953kdc8yw21.png?width=1000&format=png&auto=webp&s=01eaa010ede6157fbd02eefd3f669b30cc473750)\n\nIn the upper plot, we can see that Riot actually slowed down a bit starting in season 7 and does less changes per patch. In the lower plot, we see that the number of released skins/chromas remained fairly constant for quite some time. One might argue that the second half of season 8 and season 9 so far is higher, but the evidence for this is weak.\n\nOne thing not considered here is the amount of work that was necessary when champion reworks were made. The data consideres this as one skin due to the way the data is scraped, but the effort which was necessary by the skin team could have been higher, resulting in less skins published as a result. But in reworks, the also don't need to find a theme for the skins, so they can also be faster than with other skins. Just wanted to mention this as it's a possible source of error.\n\nIn addition, I had a look at the cross correlation between the number of champion changes and the skins released. This comes about that I thought I could see some anticorrelation in the two lines in the graphs above. I will explain below what that exactly means. The cross correlation graph looks like this:\n\n[Cross correlation between the number of champion changes and the number of released skins.](https://preview.redd.it/8fs0zunx9yw21.png?width=1000&format=png&auto=webp&s=3e5ac470e7f1eede55db37f0cec4da929e06ad43)\n\nThe lag is the difference in number of patches considered for the correlation. E.g. the positive value at lag 7 means that if a high number of champion changes is present this patch, the is a tendence for a high number of released skins/chromas 7 patches later. There is also considerable anticorrelation for a lag -5 and -6 which means that if 6 patches ago there were a lot of skins released, this patch has a tendence to have little champion changes.\n\nI have little explanation of these values, only maybe that there a less skins on season start/mid-season/end-of-season as manpower is needed to get mid season right or worlds ready. But it's also only a tendence, the correlations are significant but not too big.\n\n# Relation between the number of champion changes and skins released\n\nHere I had a look whether champions who get a skin released also got champion changes within the last 1 to 3 patches. I remember some comment in Meddlers quick gameplay thoughts where some Rioter wrote that working on a skin puts attention on said champion, making it more likely to get some work (even if only quality of life changes) on them done. Here are the results:\n\n[Number of skins considered in yellow, number of skins with champion getting changes in the last few patches in blue. Considered are the last 1 to 3 patches as well as the patch a skin\\/chroma got released.](https://preview.redd.it/h6vlbyykbyw21.png?width=600&format=png&auto=webp&s=641d7544206041f4a7b7bdb77a89bcbce7fdc450)\n\nWe can see that considering the patch a skin/chroma gets released as well as the last 3 patches, there is a 42% chance of a champion getting changes. This is way higher that the average which is somewhere around 10% if we would assume that the champions selected for changes would happen randomly.\n\nSo this is confirmation that getting a skin/chroma comes with a strong connection to being changed in the patch notes. Remember here that correlation does not imply causation, so we cannot say whether skins/chromas have a causal link to being changed in the patch notes. But there is a strong suggestion that it might be the case.\n\n# Relationship between released skins and winrate changes\n\nAnd finally, we will have a look whether a champion receiving a skin/chroma gets buffed. I will define getting buffed not by appearance in the patch notes, but by comparing the winrate of the corresponding champion 20 days before and 20 days after the patch release for any given skin/chroma. We can debate whether 20 days is a good time period, and maybe I should also consider a longer window before, but I think we can get some good results with 20 days. I excluded release skins for this analysis.\n\nI define getting buffed this way as it also consideres everything that happens in a patch which also includes item changes or systemic jungle changes, where it can be that a champion gets a (compensation) buff on paper, but actually drops in winrate due to core items being changed for example.\n\nI made one boxplot combining all the data available and one separating by season:\n\n[Boxplot with the change in winrate when a champion gets a skin\\/chroma. We see that there is almost no median change in winrate for these champions.](https://preview.redd.it/c10wzto1eyw21.png?width=400&format=png&auto=webp&s=d9b18a716aca58b4fe7f9ca817585dd0141eb658)\n\n&#x200B;\n\n[Boxplots with the change in winrate when a champion gets a skin\\/chroma, separated by season.](https://preview.redd.it/suh6xxsbeyw21.png?width=800&format=png&auto=webp&s=53d319a30e63e1c28b3021b1676236b9e2e2fc6e)\n\nWe can clearly see that there is **no** significant change in winrate when a champion gets a skin/chroma. I also put the data in the following table:\n\n|Minimum|25% Quantile|Median|75% Quantile|Maximum|Mean|\n|:-|:-|:-|:-|:-|:-|\n|\\-8.11|\\-0.83|0.02|0.76|10.3|\\-0.02|\n\nWe also see in the second graph that this remains fairly constant over the seasons, with season 7 actually being that champions which got new skins decrease in winrate.\n\n# Final words\n\nThank you for reading until here. I hope this has been an interesting read as it was interesting for me to do the analysis. If you have other interesting questions that you think could be answered with this data, write it in the comments and I try to answer them with the data. I don't have too much spare time at the moment anymore, so my answer might be a bit delayed, but I will try to get to them.\n\nHave a good day :)",
        "score": 2285,
        "upvotes": 2285,
        "upvoteRatio": 0.96,
        "commentCount": 199,
        "createdAt": "2019-05-08T09:16:26.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 8378481
      },
      {
        "id": "16r3otg",
        "name": "t3_16r3otg",
        "title": "itsAllWebscrapping",
        "author": "JustASandwhich",
        "subreddit": "ProgrammerHumor",
        "subredditPrefixedName": "r/ProgrammerHumor",
        "url": "https://www.reddit.com/r/ProgrammerHumor/comments/16r3otg/itsallwebscrapping/",
        "permalink": "https://www.reddit.com/r/ProgrammerHumor/comments/16r3otg/itsallwebscrapping/",
        "bodyText": null,
        "score": 2875,
        "upvotes": 2875,
        "upvoteRatio": 0.95,
        "commentCount": 64,
        "createdAt": "2023-09-24T17:39:31.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 4714040
      },
      {
        "id": "1ocq87g",
        "name": "t3_1ocq87g",
        "title": "I built something that webscrapes 99% of the internet",
        "author": "rexis_nobilis_",
        "subreddit": "aiagents",
        "subredditPrefixedName": "r/aiagents",
        "url": "https://www.reddit.com/r/aiagents/comments/1ocq87g/i_built_something_that_webscrapes_99_of_the/",
        "permalink": "https://www.reddit.com/r/aiagents/comments/1ocq87g/i_built_something_that_webscrapes_99_of_the/",
        "bodyText": "so this is part of a YouTube video I just released (trying to make the style of the videos fun and entertaining) about a general AI agent I’m building, has a pretty unique infrastructure that lets her do some crazy stuff!\n\neither way, I decided to make a video on how you can use it to web scrape almost any website and even compound tasks on top of it  all without touching a line of code.\n\nFYI: web scraping is just one use-case, it can also do things like:\n* create, read, update, delete files in her operating system\n* browse the web in real-time\n* connect to apps, databases (even personal ones) and IoTs\n* schedule recurring tasks just with prompts…and so much more.\n\nhere are a few of the prompts I show in the video if you want to try them out:\n\n> Go to the Browserbase pricing page. Gather all the pricing tier information, including the plan name, monthly and yearly cost, features included in each plan, and any usage limits. Convert this data into a clean JSON format where each plan is an object with its corresponding details. Then save the JSON file into agentic storage under the name browserbase_pricing.json.\n\n> Search Amazon for the top running backpack listings. For each listing, extract the title, product link, price, and description. Organize all this information into a well-formatted Excel file, with each column labeled clearly (Title, Link, Price, Description). Save the file in agentic storage.\n\n> Search LinkedIn for posts about AI in Healthcare. Summarize each post, collect the author’s full name, a quick description about them, and the post link in a CSV file. Save everything into a folder called \"Linkedin healthcare leads\".\n\nI’m also beta testing a new feature that will let you run thousands of tasks at scale. For example, you could just write:\n\n> “Fetch me 2,000 manufacturing companies in Europe and the U.S. that have 10–200 employees, founded after 2010. Include the company name, website, HQ location, description, and score from 1–10 on how well it matches what we’re currently selling in an excel file (based on company_products.txt in the storage).”\n\n…and it will handle it, all with just a prompt!\nif you want to test it out, just lmk, I’d love to get your feedback :)",
        "score": 191,
        "upvotes": 191,
        "upvoteRatio": 0.89,
        "commentCount": 46,
        "createdAt": "2025-10-21T21:51:42.000Z",
        "isVideo": true,
        "subredditSubscriberCount": 108663
      },
      {
        "id": "1rzudf1",
        "name": "t3_1rzudf1",
        "title": "(OC) Taler Socialdemokraterne med to tunger omkring boligskat? Jeg har lavet et webscrape, der tyder på det",
        "author": "Purple-Highway-4048",
        "subreddit": "Denmark",
        "subredditPrefixedName": "r/Denmark",
        "url": "https://www.reddit.com/r/Denmark/comments/1rzudf1/oc_taler_socialdemokraterne_med_to_tunger_omkring/",
        "permalink": "https://www.reddit.com/r/Denmark/comments/1rzudf1/oc_taler_socialdemokraterne_med_to_tunger_omkring/",
        "bodyText": "Hej Hestenettet.\n\nJeg har siden nytår været ret interesseret i, hvad Socialdemokratiet vil gøre i forhold til boligskat, hvis/når de får magten igen efter valget. I nytårstalen insinuerede Mette Frederiksen nemlig ret kraftigt, at en skat på boliggevinster kunne være på vej, da hun sagde, at \"Der er noget galt, når nogen er blevet rige alene ved at bo det rigtig sted.\"\n\nSiden har hun dog trukket i land igen og lovet, at der ikke ville komme en skat på boliggevinster.\n\nMen som vi ved, så hænder det jo, at Socialdemokratiet finder på ting efter et valg, som de ikke var helt klare omkring før et valg (looking at you, Store Bededag).\n\nOg da jeg så flere S-kandidater i København campaigne ret hårdt på beskatning af boliggevinster, satte jeg mig for at lave et webscrape for faktisk at se, hvor partiets kandidater står i DR's kandidattest.\n\nOg ja, I kan selv se resultatet i billedet. 47 ud af 79 kandidater var enten \"lidt enig\" eller \"enig\" i DR's udsagn, der lyder \"De boligejere, der tjener mest på prisstigninger skal betale mere i skat\". Altså 60 pct. af Socialdemokratiets samlede antal kandidater. Kigger man kun på kandidaterne i København, er tallet 7 ud af 10.\n\nJeg er selvfølgelig biased, fordi jeg har købt en lejlighed i Valby, som jeg håber at kunne få noget friværdi ud af, så jeg kan opgradere på et tidspunkt, men mest af alt ærgrer jeg mig over, at der skal være sådan en tvivl omkring kursen. \n\nFair nok, at man er for en skat på boliggevinster (det tror jeg såmænd også mange her i tråden vil være, og fred være med det), men jeg synes da i det mindste, at man som statsministerparti skal være soleklar omkring det inden et valg, så borgerne kan planlægge derefter.\n\nHvad tænker I? Tror I, at der kommer en boligskat, hvis rød blok kommer til magten?",
        "score": 0,
        "upvotes": 0,
        "upvoteRatio": 0.45,
        "commentCount": 40,
        "createdAt": "2026-03-21T15:30:58.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 855373
      },
      {
        "id": "1nw8ejy",
        "name": "t3_1nw8ejy",
        "title": "Why haven't LLMs solved webscraping?",
        "author": "Live_Baker_6532",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1nw8ejy/why_havent_llms_solved_webscraping/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1nw8ejy/why_havent_llms_solved_webscraping/",
        "bodyText": "Why is it that LLMs have not revolutionized webscraping where we can simply make a request or a call and have an LLM scrape our desired site?",
        "score": 36,
        "upvotes": 36,
        "upvoteRatio": 0.82,
        "commentCount": 54,
        "createdAt": "2025-10-02T16:23:23.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1i6tfec",
        "name": "t3_1i6tfec",
        "title": "Why does webscraping cause this facial expression?",
        "author": "DM_Me_Summits_In_UAE",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1i6tfec/why_does_webscraping_cause_this_facial_expression/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1i6tfec/why_does_webscraping_cause_this_facial_expression/",
        "bodyText": null,
        "score": 378,
        "upvotes": 378,
        "upvoteRatio": 0.99,
        "commentCount": 36,
        "createdAt": "2025-01-21T21:10:48.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1siuxmq",
        "name": "t3_1siuxmq",
        "title": "webscraping company careers pages",
        "author": "Pale-Row-4817",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1siuxmq/webscraping_company_careers_pages/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1siuxmq/webscraping_company_careers_pages/",
        "bodyText": "I work in sales prospecting and need to analyze job openings across multiple company career pages to identify hiring patterns. I need to scrape company job boards on on ATS's like Greenhouse, Workday, and icims to extract data like total job count, job category breakdown, frontline versus management roles, and posting frequency for each role type. Manually collecting this from each company's careers page is incredibly time-consuming. I'm looking for web scraping or automation solutions, ideally an agent-based approach, that can pull this data directly from company career pages rather than aggregated job boards like Indeed or ZipRecruiter, since the company data is more accurate and complete. Any suggestions on tools or approaches?",
        "score": 9,
        "upvotes": 9,
        "upvoteRatio": 0.91,
        "commentCount": 22,
        "createdAt": "2026-04-11T21:05:17.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "14agito",
        "name": "t3_14agito",
        "title": "Meta will make their next LLM free for commercial use, putting immense pressure on OpenAI and Google",
        "author": "ShotgunProxy",
        "subreddit": "ChatGPT",
        "subredditPrefixedName": "r/ChatGPT",
        "url": "https://www.reddit.com/r/ChatGPT/comments/14agito/meta_will_make_their_next_llm_free_for_commercial/",
        "permalink": "https://www.reddit.com/r/ChatGPT/comments/14agito/meta_will_make_their_next_llm_free_for_commercial/",
        "bodyText": "IMO, this is a major development in the open-source AI world as Meta's foundational LLaMA LLM is already one of the most popular base models for researchers to use. \n\n[My full deepdive is here](https://www.artisana.ai/articles/metas-plan-to-offer-free-commercial-ai-models-puts-pressure-on-google-and), but I've summarized all the key points on why this is important below for Reddit community discussion.\n\n**Why does this matter?**\n\n* **Meta plans on offering a commercial license for their next open-source LLM,** which means companies can freely adopt and profit off their AI model for the first time.\n* **Meta's current LLaMA LLM is already the most popular open-source LLM foundational model in use**. Many of the new open-source LLMs you're seeing released use LLaMA as the foundation. \n* **But LLaMA is only for research use; opening this up for commercial use would truly really drive adoption.** And this in turn places massive pressure on Google + OpenAI.\n* **There's likely massive demand for this already:** I speak with ML engineers in my day job and many are tinkering with LLaMA on the side. But they can't productionize these models into their commercial software, so the commercial license from Meta would be the big unlock for rapid adoption.\n\n**How are OpenAI and Google responding?**\n\n* **Google seems pretty intent on the closed-source route.** Even though an internal memo from an AI engineer called them out for having \"no moat\" with their closed-source strategy, executive leadership isn't budging.\n* **OpenAI is feeling the heat and plans on releasing their own open-source model.** Rumors have it this won't be anywhere near GPT-4's power, but it clearly shows they're worried and don't want to lose market share. Meanwhile, Altman is pitching global regulation of AI models as his big policy goal.\n* **Even the US government seems worried about open source;** last week a bipartisan Senate group sent a letter to Meta asking them to explain why they irresponsibly released a powerful open-source model into the wild\n\n**Meta, in the meantime, is really enjoying their limelight from the contrarian approach.** \n\n* In an interview this week, Meta's Chief AI scientist Yan LeCun dismissed any worries about AI posing dangers to humanity as \"preposterously ridiculous.\"\n\n**P.S. If you like this kind of analysis,** I write [a free newsletter](https://artisana.beehiiv.com/subscribe?utm_source=reddit&utm_campaign=chatgpt230615) that tracks the biggest issues and implications of generative AI tech. It's sent once a week and helps you stay up-to-date in the time it takes to have your Sunday morning coffee.",
        "score": 5439,
        "upvotes": 5439,
        "upvoteRatio": 0.95,
        "commentCount": 634,
        "createdAt": "2023-06-15T23:10:57.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 11492146
      },
      {
        "id": "wg5s68",
        "name": "t3_wg5s68",
        "title": "Valkia found while webscraping the contents.totalwar.com website",
        "author": "mrhugo13",
        "subreddit": "totalwar",
        "subredditPrefixedName": "r/totalwar",
        "url": "https://www.reddit.com/r/totalwar/comments/wg5s68/valkia_found_while_webscraping_the/",
        "permalink": "https://www.reddit.com/r/totalwar/comments/wg5s68/valkia_found_while_webscraping_the/",
        "bodyText": null,
        "score": 577,
        "upvotes": 577,
        "upvoteRatio": 0.97,
        "commentCount": 67,
        "createdAt": "2022-08-04T16:36:06.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 483502
      },
      {
        "id": "145f1r8",
        "name": "t3_145f1r8",
        "title": "People forget why they make their API free.",
        "author": "propjX",
        "subreddit": "ProgrammerHumor",
        "subredditPrefixedName": "r/ProgrammerHumor",
        "url": "https://www.reddit.com/r/ProgrammerHumor/comments/145f1r8/people_forget_why_they_make_their_api_free/",
        "permalink": "https://www.reddit.com/r/ProgrammerHumor/comments/145f1r8/people_forget_why_they_make_their_api_free/",
        "bodyText": null,
        "score": 9971,
        "upvotes": 9971,
        "upvoteRatio": 0.98,
        "commentCount": 371,
        "createdAt": "2023-06-09T19:55:50.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 4714040
      },
      {
        "id": "1s3mzzb",
        "name": "t3_1s3mzzb",
        "title": "Webscraping, crear un Bot",
        "author": "JesusMendoza2145",
        "subreddit": "ColombiaDevs",
        "subredditPrefixedName": "r/ColombiaDevs",
        "url": "https://www.reddit.com/r/ColombiaDevs/comments/1s3mzzb/webscraping_crear_un_bot/",
        "permalink": "https://www.reddit.com/r/ColombiaDevs/comments/1s3mzzb/webscraping_crear_un_bot/",
        "bodyText": "Saludos colegas, actualmente estoy trabajando en un proyecto personal, estoy creando un bot para hacer scraping a unas paginas. pero presento dificultad para hacerle bypass a cloudflare alguien me puede ayudar?",
        "score": 5,
        "upvotes": 5,
        "upvoteRatio": 0.86,
        "commentCount": 19,
        "createdAt": "2026-03-25T20:53:42.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 12359
      },
      {
        "id": "1k8ml01",
        "name": "t3_1k8ml01",
        "title": "How important is webscraping as a skill for Data Engineers?",
        "author": "godz_ares",
        "subreddit": "dataengineering",
        "subredditPrefixedName": "r/dataengineering",
        "url": "https://www.reddit.com/r/dataengineering/comments/1k8ml01/how_important_is_webscraping_as_a_skill_for_data/",
        "permalink": "https://www.reddit.com/r/dataengineering/comments/1k8ml01/how_important_is_webscraping_as_a_skill_for_data/",
        "bodyText": "Hi all,\n\nI am teaching myself Data Engineering. I am working on a project that incorporates everything I know so far and this includes getting data via Web scraping. \n\nI think I underestimated how hard it would be. I've taken a course on webscraping but I underestimated the depth that exists, the tools available as well as the fact that the site itself can be an antagonist and try to stop you from scraping. \n\nThis is not to mention that you need a good understanding of HTML and website; which for me, as a person who only knows coding through the eyes of databases and pandas was quite a shock. \n\nAnyways, I just wanted to know how relevant webscraping is in the toolbox of a data engineers. \n\nThanks",
        "score": 53,
        "upvotes": 53,
        "upvoteRatio": 0.85,
        "commentCount": 61,
        "createdAt": "2025-04-26T20:37:14.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 454134
      },
      {
        "id": "1s2d4hp",
        "name": "t3_1s2d4hp",
        "title": "Weekly Webscrapers - Hiring, FAQs, etc",
        "author": "AutoModerator",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1s2d4hp/weekly_webscrapers_hiring_faqs_etc/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1s2d4hp/weekly_webscrapers_hiring_faqs_etc/",
        "bodyText": "**Welcome to the weekly discussion thread!**\n\nThis is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:\n\n* Hiring and job opportunities\n* Industry news, trends, and insights\n* Frequently asked questions, like \"How do I scrape LinkedIn?\"\n* Marketing and monetization tips\n\nIf you're new to web scraping, make sure to check out the [Beginners Guide](https://webscraping.fyi) 🌱\n\nCommercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the [monthly thread](https://reddit.com/r/webscraping/about/sticky?num=1)",
        "score": 3,
        "upvotes": 3,
        "upvoteRatio": 1,
        "commentCount": 16,
        "createdAt": "2026-03-24T13:01:49.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1rovtae",
        "name": "t3_1rovtae",
        "title": "Beginner need help trying to build a webscraper",
        "author": "Prestigious-Cup-4722",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1rovtae/beginner_need_help_trying_to_build_a_webscraper/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1rovtae/beginner_need_help_trying_to_build_a_webscraper/",
        "bodyText": "Hello, i've build a scraper that should collect data from idealo. For now, theres only one product from which im trying to get all the offers with ranking, company, prices, shipping info and reviews...\n\nAside from that, i want to get the data sorted by two categories: product price and total price with a screenshot of both so that i can check the data. \n\nI'm using python and playwright, data should be collected in one csv file. \n\nNow I'm facing a few problems:\n\n1. Idealo changes their website so that my scraper cant differentiate between different prices (promotions like \"shipping free from X€\" become total costs...) and companys are suddenly \"unknown\"\n\n2. screenshots are not taken, i only got the screenshot with the category 'product', so i cant check the total price data\n\n3. the last time i started the scraper, a new csv file was opened altough the csv file i had should be carried on (worked for 1-2 weeks) \n\ni'm building this scraper for my professor but i don't have any knowledge about programming, also he needs the data for about a month so i thought about doing it manually since this wont be the last product i need to scrape & i don't know much about the maintenance and the limitations - been doing it with the free versions of chatgpt & claude because there is no budget",
        "score": 1,
        "upvotes": 1,
        "upvoteRatio": 0.6,
        "commentCount": 18,
        "createdAt": "2026-03-09T09:44:10.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1ro6w0j",
        "name": "t3_1ro6w0j",
        "title": "Trawl: Self healing AI webscraper written in go",
        "author": "KingRonra",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1ro6w0j/trawl_self_healing_ai_webscraper_written_in_go/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1ro6w0j/trawl_self_healing_ai_webscraper_written_in_go/",
        "bodyText": "I've been lurking here for a while and the #1 recurring pain point is obvious: selectors break. Site redesigns, A/B tests, minor template changes — and your scraper is silently returning garbage.\n\nSo I built trawl. You tell it what fields you want in plain English:\n\ntrawl \"[https://books.toscrape.com](https://books.toscrape.com)\" --fields \"title, price, rating, in\\_stock\"\n\nIt fetches a sample page, sends simplified HTML to an LLM (Claude), and gets back a full extraction strategy — CSS selectors, fallbacks, type mappings, pagination rules. Then it caches that strategy and applies it to every page using Go + goquery. No LLM calls after the first one.\n\nSite changes? The structural fingerprint won't match the cache, so it re-derives automatically.\n\nWhere it gets really useful is pages with multiple data sections. Say you hit a company page that has a leadership team table, a financials summary, and a product grid all on one page. Instead of writing selectors that target the right section, you just tell it what you're after:\n\ntrawl \"[https://example.com/about](https://example.com/about)\" \\\\\n\n\\--query \"executive leadership team\" \\\\\n\n\\--fields \"name, title, bio\" \\\\\n\n\\--format json\n\nThe LLM understands you want the leadership section, not the financials table, and scopes the extraction to the right container. No manual DOM inspection needed.\n\nThe --plan flag lets you see exactly what it came up with before extracting anything, so you're not trusting a black box:\n\n$ trawl \"[https://example.com/about](https://example.com/about)\" \\\\\n\n\\--query \"executive leadership team\" \\\\\n\n\\--fields \"name, title, bio\" --plan\n\nStrategy for [https://example.com/about](https://example.com/about)\n\nContainer: section#leadership\n\nItem selector: div.team-member\n\nFields:\n\nname:  h3.member-name -> text (string)\n\ntitle: span.role -> text (string)\n\nbio:   [p.bio](http://p.bio) \\-> text (string)\n\nConfidence: 0.93\n\nSome other things it handles that I'm especially happy with:\n\n\\- JS-rendered SPAs: headless browser with DOM stability detection, waits for element count to stabilize, scrolls for lazy loading, clicks through \"Show more\" buttons\n\n\\- Self-healing: tracks extraction success rate per batch, re-derives if it drops below 70%\n\n\\- Iframes: auto-detects when iframe content has richer data than the outer page\n\nOutputs JSON, JSONL, CSV, or Parquet. Pipes to jq, csvkit, etc.:\n\ntrawl \"[https://example.com/products](https://example.com/products)\" --fields \"name, price\" --format jsonl | jq 'select(.price > 50)'\n\nGo binary, so no Python env to manage. MIT licensed.\n\nGitHub: [https://github.com/akdavidsson/trawl](https://github.com/akdavidsson/trawl)\n\nWould love feedback from this community, you all know the edge cases better than anyone.",
        "score": 57,
        "upvotes": 57,
        "upvoteRatio": 0.93,
        "commentCount": 11,
        "createdAt": "2026-03-08T14:56:31.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1rw5v6k",
        "name": "t3_1rw5v6k",
        "title": "Weekly Webscrapers - Hiring, FAQs, etc",
        "author": "AutoModerator",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1rw5v6k/weekly_webscrapers_hiring_faqs_etc/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1rw5v6k/weekly_webscrapers_hiring_faqs_etc/",
        "bodyText": "**Welcome to the weekly discussion thread!**\n\nThis is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:\n\n* Hiring and job opportunities\n* Industry news, trends, and insights\n* Frequently asked questions, like \"How do I scrape LinkedIn?\"\n* Marketing and monetization tips\n\nIf you're new to web scraping, make sure to check out the [Beginners Guide](https://webscraping.fyi) 🌱\n\nCommercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the [monthly thread](https://reddit.com/r/webscraping/about/sticky?num=1)",
        "score": 6,
        "upvotes": 6,
        "upvoteRatio": 1,
        "commentCount": 15,
        "createdAt": "2026-03-17T13:00:56.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1le7web",
        "name": "t3_1le7web",
        "title": "NSEIndia  webscraping  has become impossible (Akamai)",
        "author": "gripped909",
        "subreddit": "IndianStockMarket",
        "subredditPrefixedName": "r/IndianStockMarket",
        "url": "https://www.reddit.com/r/IndianStockMarket/comments/1le7web/nseindia_webscraping_has_become_impossible_akamai/",
        "permalink": "https://www.reddit.com/r/IndianStockMarket/comments/1le7web/nseindia_webscraping_has_become_impossible_akamai/",
        "bodyText": "Earlier it was easily possible to scrape data of NSEIndia(Optionschain - [link](https://www.nseindia.com/option-chain)) for personal use using a simple Python-requests call .\n\nHowever recently looks like NSEIndia has switched to Akamai [API Protector](https://www.akamai.com/products/app-and-api-protector) , making the task almost impossible .\n\nIndividual requests with ample delay of 5 seconds between them are also getting flagged as bots .\n\nTried using Selenium and Playwright to simulate a real user's browser clicks , but with even a single request Akamai detects the bot and gives a 403 ERROR.\n\nHas anyone tried any other successful solution?",
        "score": 160,
        "upvotes": 160,
        "upvoteRatio": 0.95,
        "commentCount": 29,
        "createdAt": "2025-06-18T04:15:25.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 1320632
      },
      {
        "id": "1tbfx25",
        "name": "t3_1tbfx25",
        "title": "Webscraping and screenshotting?",
        "author": "jeyunh",
        "subreddit": "Playwright",
        "subredditPrefixedName": "r/Playwright",
        "url": "https://www.reddit.com/r/Playwright/comments/1tbfx25/webscraping_and_screenshotting/",
        "permalink": "https://www.reddit.com/r/Playwright/comments/1tbfx25/webscraping_and_screenshotting/",
        "bodyText": "Has anyone successfully used Playwright for long-running high-volume screenshot rendering in production? I keep running into memory issues.\n\nContext:  \n\\-Java 17 + Playwright  \n\\-Using Chromium headless browser  \n\\-Purpose is ONLY chart/page rendering + screenshots.",
        "score": 0,
        "upvotes": 0,
        "upvoteRatio": 0.5,
        "commentCount": 5,
        "createdAt": "2026-05-12T21:49:32.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 8386
      },
      {
        "id": "1q47k94",
        "name": "t3_1q47k94",
        "title": "How much does webscraping cost?",
        "author": "TangerineBetter855",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1q47k94/how_much_does_webscraping_cost/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1q47k94/how_much_does_webscraping_cost/",
        "bodyText": "is it possible to scrape large sites like youtube or tinder and is scraping apps possible or is it only sites?",
        "score": 13,
        "upvotes": 13,
        "upvoteRatio": 0.68,
        "commentCount": 23,
        "createdAt": "2026-01-05T01:08:09.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1sx61e7",
        "name": "t3_1sx61e7",
        "title": "Letting your AI get used in a war legit but asking so to find you a cheap proxy for webscraping is a reason for ban nice 🤠",
        "author": "CarGold87",
        "subreddit": "codex",
        "subredditPrefixedName": "r/codex",
        "url": "https://www.reddit.com/r/codex/comments/1sx61e7/letting_your_ai_get_used_in_a_war_legit_but/",
        "permalink": "https://www.reddit.com/r/codex/comments/1sx61e7/letting_your_ai_get_used_in_a_war_legit_but/",
        "bodyText": "Hi gang is there any one got banned like me!",
        "score": 0,
        "upvotes": 0,
        "upvoteRatio": 0.5,
        "commentCount": 8,
        "createdAt": "2026-04-27T14:53:10.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 97697
      },
      {
        "id": "1scld0y",
        "name": "t3_1scld0y",
        "title": "Problemas legales con webscrapping",
        "author": "moon-cat99",
        "subreddit": "DerechoGenial",
        "subredditPrefixedName": "r/DerechoGenial",
        "url": "https://www.reddit.com/r/DerechoGenial/comments/1scld0y/problemas_legales_con_webscrapping/",
        "permalink": "https://www.reddit.com/r/DerechoGenial/comments/1scld0y/problemas_legales_con_webscrapping/",
        "bodyText": "Buenas! estoy pensando un proyecto personal que involucra hacer webscrapping de paginas de inmobiliarias/clasificados y quería saber que tantas posibilidades tengo de terminar vetado de todas las inmobiliarias de la ciudad y tener que irme a vivir abajo de un puente. Considerando que no tienen por lo general TyC, deberia evitar datos de contacto/nombres, imagino, asumiendo que se consideran datos personales? La dirección del inmueble entra como dato personal?",
        "score": 0,
        "upvotes": 0,
        "upvoteRatio": 0.36,
        "commentCount": 11,
        "createdAt": "2026-04-04T21:31:35.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 96579
      }
    ],
    "totalResults": 25,
    "page": {
      "nextCursor": "t3_1scld0y",
      "hasMore": true
    }
  },
  "meta": {
    "requestId": "req_01example",
    "creditsCharged": 1,
    "version": "v1"
  }
}
```

### Example JSON (empty)

```json
{
  "data": {
    "query": "socialfetch_fixture_no_results_000000",
    "posts": [],
    "totalResults": 0,
    "page": {
      "nextCursor": null,
      "hasMore": false
    }
  },
  "meta": {
    "requestId": "req_01example_empty",
    "creditsCharged": 1,
    "version": "v1"
  }
}
```

### Example JSON (paginated)

```json
{
  "data": {
    "query": "webscraping",
    "posts": [
      {
        "id": "1ksizv5",
        "name": "t3_1ksizv5",
        "title": "Fiz webscrapping em todos os anuncios do DF imóveis e obtive os seguintes resultados dos imóveis de brasilia. Antes de tudo, não sou engenheiro de dados, apenas um curioso que queria ver como Ruby sairia nessa atividade.",
        "author": "CabecaRaspada",
        "subreddit": "brasilia",
        "subredditPrefixedName": "r/brasilia",
        "url": "https://www.reddit.com/r/brasilia/comments/1ksizv5/fiz_webscrapping_em_todos_os_anuncios_do_df/",
        "permalink": "https://www.reddit.com/r/brasilia/comments/1ksizv5/fiz_webscrapping_em_todos_os_anuncios_do_df/",
        "bodyText": "https://preview.redd.it/ll7mbt3uq92f1.png?width=1054&format=png&auto=webp&s=facd8d9de45bd402d5c7fe641683bb1253e0ca81\n\nhttps://preview.redd.it/pgayo02uq92f1.png?width=1075&format=png&auto=webp&s=aedc0e3dc4c306c44859bf9fe0e1f369713ea286\n\no projeto ta em fase beta, ainda não hospedei ele (e não sei se vou hospedar sendo sincero, até porque o scrapping demorou HORAS para ser finalizado, e eu não conseguiria arcar com os custos de um servidor).\n\nmas ta ai algum dos resultados que obtive",
        "score": 114,
        "upvotes": 114,
        "upvoteRatio": 0.99,
        "commentCount": 37,
        "createdAt": "2025-05-22T05:25:55.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 108160
      },
      {
        "id": "1jxeouq",
        "name": "t3_1jxeouq",
        "title": "Purpose of webscraping?",
        "author": "Mizzen_Twixietrap",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1jxeouq/purpose_of_webscraping/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1jxeouq/purpose_of_webscraping/",
        "bodyText": "What's the purpose of it? \n\nI get that you get a lot of information, but this information can be outdated by a mile. And what are you to use of this information anyway?\n\nYes you can get Emails, which you then can sell to other who'll make cold calls, but the rest I find hard to see any purpose with?\n\nSorry if this is a stupid question.\n\nEdit - Thanks for all the replies. It has shown me that scraping is used for a lot of things mostly AI. (Trading bots, ChatGPT etc.) Thank you for taking your time to tell me ☺️",
        "score": 6,
        "upvotes": 6,
        "upvoteRatio": 0.58,
        "commentCount": 63,
        "createdAt": "2025-04-12T10:57:58.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1qj654h",
        "name": "t3_1qj654h",
        "title": "Webscraper gratis. Que opinan?",
        "author": "AverellPSG",
        "subreddit": "CharruaDevs",
        "subredditPrefixedName": "r/CharruaDevs",
        "url": "https://www.reddit.com/r/CharruaDevs/comments/1qj654h/webscraper_gratis_que_opinan/",
        "permalink": "https://www.reddit.com/r/CharruaDevs/comments/1qj654h/webscraper_gratis_que_opinan/",
        "bodyText": "Soy vibe coder en serie y hice otra poronga\n\n[https://scrapeurls.xyz](https://scrapeurls.xyz) \n\n  \nQue les parece? \n\nFront: typescript\n\nBack: n8n con custom api de crawl4ai",
        "score": 3,
        "upvotes": 3,
        "upvoteRatio": 0.62,
        "commentCount": 20,
        "createdAt": "2026-01-21T18:36:33.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 32871
      },
      {
        "id": "1rbmnrb",
        "name": "t3_1rbmnrb",
        "title": "Hiring - WebScraping + iMessage to CrM",
        "author": "rationalbou896",
        "subreddit": "WebDeveloperJobs",
        "subredditPrefixedName": "r/WebDeveloperJobs",
        "url": "https://www.reddit.com/r/WebDeveloperJobs/comments/1rbmnrb/hiring_webscraping_imessage_to_crm/",
        "permalink": "https://www.reddit.com/r/WebDeveloperJobs/comments/1rbmnrb/hiring_webscraping_imessage_to_crm/",
        "bodyText": "Hi guys,\n\nLooking to get two things done.\n\nOne. Looking to scrape Google Maps and then push the information to my CRM.\n\n2. Looking forward iMessage using Open bubbles of blue bubbles to my CRM need to wear integration for this.\n\nIf you have any experience in doing this please message me. I’m not sure if the budget or the course because I’m not sure if this can be done if you can let’s discuss thanks",
        "score": 5,
        "upvotes": 5,
        "upvoteRatio": 0.86,
        "commentCount": 15,
        "createdAt": "2026-02-22T14:04:19.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 81964
      },
      {
        "id": "1rqsvgp",
        "name": "t3_1rqsvgp",
        "title": "Python + Selenium at scale (50 nodes, 3.9M records)",
        "author": "SuccessfulFact5324",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1rqsvgp/python_selenium_at_scale_50_nodes_39m_records/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1rqsvgp/python_selenium_at_scale_50_nodes_39m_records/",
        "bodyText": "I've been running this scrapper for 2+ years across 50  nodes, 3.9M+ records collected from a very popular job site. Here are the few scraping challenges — would love feedback from people who've solved these better.\n\n\\## Full browser over browserless\n\nThe target site fingerprints navigator.webdriver, so I override it via JS and disable automation flags in Chrome. Headless mode got detected faster than a visible browser, so I run full Chrome on each node with random user-agent rotation. Each node also runs through a VPN before the script starts.\n\n*## Avoiding brittle class selectors*\n\nThe site redesigns frequently. I target elements by tag name or text content via XPATH wherever possible instead of class names. For pagination I match button text rather than the button's class. For job links I target the a tag directly — stable across every redesign so far.\n\n*## 429 handling*\n\nAt \\~50 nodes running in parallel, rate limiting is constant. The site doesn’t return a proper HTTP error and instead renders a “Reload” button in the page source, so I detect it via `page_source`, locate the button with XPath using the inner text, and retry up to 5 times. After each reload I also check for auth-wall redirects since the site sometimes sends you to login instead. I run traffic through regular VPN endpoints to reduce rate limits, but those occasionally get flagged or banned by the target site too.\n\n*## Sign-in modal interception*\n\nLogin Modals block content on almost every page load. I use a 3-fallback dismissal strategy: X button → Escape key via ActionChains → JavaScript CSS force-hide. The JS fallback handles cases where the modal intercepts all click events and neither of the first two approaches work.\n\n**## Stacks used**\n\nScraping: Python, Selenium, BeautifulSoup, spaCy\n\nInfrastructure:  50 nodes, NAS, a VPN per node, WiFi smart power strip for auto power-cycling failed nodes\n\nMonitoring:  Custom dashboard showing real-time node status\n\n*## Questions:*\n\n\\- How do you handle sites that A/B test their UI constantly — multiple selector fallbacks or parse raw HTML offline?\n\n\\- VPN at this scale vs residential proxies — worth the switch?\n\n\\- Any better approach to modal dismissal than layered fallbacks?",
        "score": 882,
        "upvotes": 882,
        "upvoteRatio": 0.99,
        "commentCount": 181,
        "createdAt": "2026-03-11T12:50:40.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1sevbhd",
        "name": "t3_1sevbhd",
        "title": "Weekly Webscrapers - Hiring, FAQs, etc",
        "author": "AutoModerator",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1sevbhd/weekly_webscrapers_hiring_faqs_etc/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1sevbhd/weekly_webscrapers_hiring_faqs_etc/",
        "bodyText": "**Welcome to the weekly discussion thread!**\n\nThis is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:\n\n* Hiring and job opportunities\n* Industry news, trends, and insights\n* Frequently asked questions, like \"How do I scrape LinkedIn?\"\n* Marketing and monetization tips\n\nIf you're new to web scraping, make sure to check out the [Beginners Guide](https://webscraping.fyi) 🌱\n\nCommercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the [monthly thread](https://reddit.com/r/webscraping/about/sticky?num=1)",
        "score": 6,
        "upvotes": 6,
        "upvoteRatio": 0.88,
        "commentCount": 7,
        "createdAt": "2026-04-07T13:01:02.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1p8dg89",
        "name": "t3_1p8dg89",
        "title": "WebScraping in golang",
        "author": "North_Fall_8333",
        "subreddit": "golang",
        "subredditPrefixedName": "r/golang",
        "url": "https://www.reddit.com/r/golang/comments/1p8dg89/webscraping_in_golang/",
        "permalink": "https://www.reddit.com/r/golang/comments/1p8dg89/webscraping_in_golang/",
        "bodyText": "Is webscraping in go a good idea? I'm used to using playwright and selenium for webscraping in java/kotlin but i've been focusing on learning golang recently is this a good idea and if yes than what should I use for it?",
        "score": 16,
        "upvotes": 16,
        "upvoteRatio": 0.73,
        "commentCount": 24,
        "createdAt": "2025-11-27T21:24:09.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 359685
      },
      {
        "id": "1rsohsy",
        "name": "t3_1rsohsy",
        "title": "Need advice on webscraping business registry",
        "author": "XSymbiose",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1rsohsy/need_advice_on_webscraping_business_registry/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1rsohsy/need_advice_on_webscraping_business_registry/",
        "bodyText": "Hi all,\n\nI built a company data enrichment scraper in Python and I think I may have designed the network side badly. \n\nIt mainly uses standard **HTTP requests** plus **Playwright** for some website fetches, and I also added **multithreading**, **rotating proxies**, and random **user agents** to make the scraping more resilient, but I’m now wondering if that was a mistake for this type of workflow.\n\nThe goal is simple:\n\n* start from company website URLs\n* fetch the homepage and try to find the legal/company info page\n* extract the company's business registry number from the site\n* query a public company data source\n* save the enriched result locally\n\nThe issue is that my proxy provider (100 proxy servers) flagged my account after noticing a large volume of requests going through their network to public company data sources, institutional websites, and a business registry. \n\nThe original idea was to keep proxies limited to regular website fetches and avoid using them for public endpoints, but the proxy provider blocked the account before I could properly separate that traffic. So before making major changes, I’d like to do a proper check on the overall setup.\n\nAnother thing I probably got wrong is that, once proxies were added, I didn’t pay attention to the API’s own rate-limit signals. I wasn’t really using the timing/cooldown information returned when request volume got too high, which was probably the wrong approach.\n\nI’d really appreciate feedback on how people usually handle this kind of scraping project. It’s my first time building something with this many requests, so I’m mostly trying to understand whether the overall setup makes sense and whether the scraping / network logic is coherent.\n\nWould really appreciate advice, thanks!\n\nPS: There is no monthly or weekly thread, that's why the repost.",
        "score": 2,
        "upvotes": 2,
        "upvoteRatio": 0.67,
        "commentCount": 11,
        "createdAt": "2026-03-13T14:15:19.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1sr22xx",
        "name": "t3_1sr22xx",
        "title": "POV: Webscraping for training data",
        "author": "Ready_Row3788",
        "subreddit": "WeLoveYouCaine",
        "subredditPrefixedName": "r/WeLoveYouCaine",
        "url": "https://www.reddit.com/r/WeLoveYouCaine/comments/1sr22xx/pov_webscraping_for_training_data/",
        "permalink": "https://www.reddit.com/r/WeLoveYouCaine/comments/1sr22xx/pov_webscraping_for_training_data/",
        "bodyText": null,
        "score": 36,
        "upvotes": 36,
        "upvoteRatio": 0.98,
        "commentCount": 2,
        "createdAt": "2026-04-20T20:33:08.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 11580
      },
      {
        "id": "1tb0xrq",
        "name": "t3_1tb0xrq",
        "title": "Weekly Webscrapers - Hiring, FAQs, etc",
        "author": "AutoModerator",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1tb0xrq/weekly_webscrapers_hiring_faqs_etc/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1tb0xrq/weekly_webscrapers_hiring_faqs_etc/",
        "bodyText": "**Welcome to the weekly discussion thread!**\n\nThis is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:\n\n* Hiring and job opportunities\n* Industry news, trends, and insights\n* Frequently asked questions, like \"How do I scrape LinkedIn?\"\n* Marketing and monetization tips\n\nIf you're new to web scraping, make sure to check out the [Beginners Guide](https://webscraping.fyi) 🌱\n\nCommercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the [monthly thread](https://reddit.com/r/webscraping/about/sticky?num=1)",
        "score": 4,
        "upvotes": 4,
        "upvoteRatio": 0.84,
        "commentCount": 2,
        "createdAt": "2026-05-12T13:00:35.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1t900dz",
        "name": "t3_1t900dz",
        "title": "Guide me how shall I learn webscraping",
        "author": "Positive-Union-3868",
        "subreddit": "thewebscrapingclub",
        "subredditPrefixedName": "r/thewebscrapingclub",
        "url": "https://www.reddit.com/r/thewebscrapingclub/comments/1t900dz/guide_me_how_shall_i_learn_webscraping/",
        "permalink": "https://www.reddit.com/r/thewebscrapingclub/comments/1t900dz/guide_me_how_shall_i_learn_webscraping/",
        "bodyText": null,
        "score": 5,
        "upvotes": 5,
        "upvoteRatio": 1,
        "commentCount": 2,
        "createdAt": "2026-05-10T07:19:34.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 1065
      },
      {
        "id": "1srmz1n",
        "name": "t3_1srmz1n",
        "title": "Weekly Webscrapers - Hiring, FAQs, etc",
        "author": "AutoModerator",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1srmz1n/weekly_webscrapers_hiring_faqs_etc/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1srmz1n/weekly_webscrapers_hiring_faqs_etc/",
        "bodyText": "**Welcome to the weekly discussion thread!**\n\nThis is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:\n\n* Hiring and job opportunities\n* Industry news, trends, and insights\n* Frequently asked questions, like \"How do I scrape LinkedIn?\"\n* Marketing and monetization tips\n\nIf you're new to web scraping, make sure to check out the [Beginners Guide](https://webscraping.fyi) 🌱\n\nCommercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the [monthly thread](https://reddit.com/r/webscraping/about/sticky?num=1)",
        "score": 11,
        "upvotes": 11,
        "upvoteRatio": 1,
        "commentCount": 4,
        "createdAt": "2026-04-21T13:00:36.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1pc2ow0",
        "name": "t3_1pc2ow0",
        "title": "Built fast webscraper",
        "author": "New_Needleworker7830",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1pc2ow0/built_fast_webscraper/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1pc2ow0/built_fast_webscraper/",
        "bodyText": "It’s not about anti-bot techniques .. it’s about raw speed.  \nThe system is designed for large scale crawling, thousands of websites at once.  \nIt uses multiprocessing and multithreading, wth optimized internal queues to avoid bottlenecks.  \nI reached **32,000 pages per minute** on a 32-CPU machine (Scrapy: **7,000**).\n\nIt supports robots.txt, sitemaps, and standard spider techniques.  \nAll network parameters are stored in JSON.  \nRetry mechanism that switches between **httpx** and **curl**.\n\nI’m also integrating SeleniumBase, but multiprocessing is still giving me issues with that.\n\nGiven a python domain list doms = \\[\"a.com\", \"b.com\"...\\]  \nyou can begin scraping just like\n\n`from ispider_core import ISpider`  \n`with ISpider(domains=doms) as spider:`  \n`spider.run()`\n\nI'm maintaining it on pypi too:  \n`pip install ispider`\n\nGithub opensource: [https://github.com/danruggi/ispider](https://github.com/danruggi/ispider)",
        "score": 25,
        "upvotes": 25,
        "upvoteRatio": 0.88,
        "commentCount": 20,
        "createdAt": "2025-12-02T07:51:57.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1t4es34",
        "name": "t3_1t4es34",
        "title": "Weekly Webscrapers - Hiring, FAQs, etc",
        "author": "AutoModerator",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1t4es34/weekly_webscrapers_hiring_faqs_etc/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1t4es34/weekly_webscrapers_hiring_faqs_etc/",
        "bodyText": "**Welcome to the weekly discussion thread!**\n\nThis is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:\n\n* Hiring and job opportunities\n* Industry news, trends, and insights\n* Frequently asked questions, like \"How do I scrape LinkedIn?\"\n* Marketing and monetization tips\n\nIf you're new to web scraping, make sure to check out the [Beginners Guide](https://webscraping.fyi) 🌱\n\nCommercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the [monthly thread](https://reddit.com/r/webscraping/about/sticky?num=1)",
        "score": 7,
        "upvotes": 7,
        "upvoteRatio": 0.9,
        "commentCount": 2,
        "createdAt": "2026-05-05T13:00:38.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1qod78a",
        "name": "t3_1qod78a",
        "title": "Weekly Webscrapers - Hiring, FAQs, etc",
        "author": "AutoModerator",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1qod78a/weekly_webscrapers_hiring_faqs_etc/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1qod78a/weekly_webscrapers_hiring_faqs_etc/",
        "bodyText": "**Welcome to the weekly discussion thread!**\n\nThis is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:\n\n* Hiring and job opportunities\n* Industry news, trends, and insights\n* Frequently asked questions, like \"How do I scrape LinkedIn?\"\n* Marketing and monetization tips\n\nIf you're new to web scraping, make sure to check out the [Beginners Guide](https://webscraping.fyi) 🌱\n\nCommercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the [monthly thread](https://reddit.com/r/webscraping/about/sticky?num=1)",
        "score": 9,
        "upvotes": 9,
        "upvoteRatio": 1,
        "commentCount": 15,
        "createdAt": "2026-01-27T13:01:04.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1m30ydj",
        "name": "t3_1m30ydj",
        "title": "Restart your webscraping journey, what would you do differently?",
        "author": "Acceptable-Fox590",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1m30ydj/restart_your_webscraping_journey_what_would_you/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1m30ydj/restart_your_webscraping_journey_what_would_you/",
        "bodyText": "I am quite new in the game, but have seen the insane potential that webscraping offers. If you had to restart from the beginning, what do you wish you knew then that you know now? What tools would you use? What strategies? I am a professor, and I am trying to learn this to educate students on how to utilize this both for their business and studies. \n\n  \nAll the best, Adam",
        "score": 25,
        "upvotes": 25,
        "upvoteRatio": 0.92,
        "commentCount": 36,
        "createdAt": "2025-07-18T12:25:46.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1sy0r80",
        "name": "t3_1sy0r80",
        "title": "Weekly Webscrapers - Hiring, FAQs, etc",
        "author": "AutoModerator",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1sy0r80/weekly_webscrapers_hiring_faqs_etc/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1sy0r80/weekly_webscrapers_hiring_faqs_etc/",
        "bodyText": "**Welcome to the weekly discussion thread!**\n\nThis is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:\n\n* Hiring and job opportunities\n* Industry news, trends, and insights\n* Frequently asked questions, like \"How do I scrape LinkedIn?\"\n* Marketing and monetization tips\n\nIf you're new to web scraping, make sure to check out the [Beginners Guide](https://webscraping.fyi) 🌱\n\nCommercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the [monthly thread](https://reddit.com/r/webscraping/about/sticky?num=1)",
        "score": 4,
        "upvotes": 4,
        "upvoteRatio": 0.83,
        "commentCount": 2,
        "createdAt": "2026-04-28T13:00:34.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1rhvsr0",
        "name": "t3_1rhvsr0",
        "title": "webscraping websites for arbitrage",
        "author": "misterno123",
        "subreddit": "FulfillmentByAmazon",
        "subredditPrefixedName": "r/FulfillmentByAmazon",
        "url": "https://www.reddit.com/r/FulfillmentByAmazon/comments/1rhvsr0/webscraping_websites_for_arbitrage/",
        "permalink": "https://www.reddit.com/r/FulfillmentByAmazon/comments/1rhvsr0/webscraping_websites_for_arbitrage/",
        "bodyText": "Currently I am running a webscraper from home using data center proxies. I scrape only the ASINs in websites where same item has low rank on amazon. It is scraping sites with items for sale in bulk and I buy them on the cheap and sell them on amazon as new. This is just 1 item so to expand , I tried this with electronics and auto parts but most sites asking for physical location to buy in bulk\n\nIt does not have to be on amazon I can sell on ebay also, but I am looking for websites to buy in bulk. Any ideas? or is there a better subreddit to ask this question?",
        "score": 0,
        "upvotes": 0,
        "upvoteRatio": 0.5,
        "commentCount": 10,
        "createdAt": "2026-03-01T12:12:35.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 138042
      },
      {
        "id": "1rpj8t0",
        "name": "t3_1rpj8t0",
        "title": "Webscraping",
        "author": "iceonme1912",
        "subreddit": "DadosBrasil",
        "subredditPrefixedName": "r/DadosBrasil",
        "url": "https://www.reddit.com/r/DadosBrasil/comments/1rpj8t0/webscraping/",
        "permalink": "https://www.reddit.com/r/DadosBrasil/comments/1rpj8t0/webscraping/",
        "bodyText": "Fala galera, boa noite, to querendo aprender web scraping com Python, quais bibliotecas eu uso e por onde eu estudo esse assunto ? Tem alguma recomendação? Abraços !",
        "score": 5,
        "upvotes": 5,
        "upvoteRatio": 1,
        "commentCount": 8,
        "createdAt": "2026-03-10T01:22:27.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 8311
      },
      {
        "id": "1r75el6",
        "name": "t3_1r75el6",
        "title": "Weekly Webscrapers - Hiring, FAQs, etc",
        "author": "AutoModerator",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1r75el6/weekly_webscrapers_hiring_faqs_etc/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1r75el6/weekly_webscrapers_hiring_faqs_etc/",
        "bodyText": "**Welcome to the weekly discussion thread!**\n\nThis is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:\n\n* Hiring and job opportunities\n* Industry news, trends, and insights\n* Frequently asked questions, like \"How do I scrape LinkedIn?\"\n* Marketing and monetization tips\n\nIf you're new to web scraping, make sure to check out the [Beginners Guide](https://webscraping.fyi) 🌱\n\nCommercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the [monthly thread](https://reddit.com/r/webscraping/about/sticky?num=1)",
        "score": 9,
        "upvotes": 9,
        "upvoteRatio": 0.86,
        "commentCount": 10,
        "createdAt": "2026-02-17T13:02:04.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1ohh5s3",
        "name": "t3_1ohh5s3",
        "title": "Caut oameni pasionati de webscrapping",
        "author": "Early-Employ5597",
        "subreddit": "programare",
        "subredditPrefixedName": "r/programare",
        "url": "https://www.reddit.com/r/programare/comments/1ohh5s3/caut_oameni_pasionati_de_webscrapping/",
        "permalink": "https://www.reddit.com/r/programare/comments/1ohh5s3/caut_oameni_pasionati_de_webscrapping/",
        "bodyText": "Dupa cum spune si titlul caut oameni descurcareti cu interese comune pentru un grup de tg/discord privat in care sa discutam, ne ajutam reciproc etc, poate sa punem in practica anumite proiecte. Caut oameni ce au propriile site-uri/apps cu traffic. Eu am vreo 80 de site uri din diferite domenii facute de a lungul anilor. De la porn la screener pentru stock uri la site uri de statistici luate de pe site urile govs.\n\n  \nle: lucrez singur de ani de zile si sunt satul sa n am cu cine sa schmb o vorba sau o idee. de a lungul timpului am cunoscut doar 2 persoane pe nisa asta.",
        "score": 22,
        "upvotes": 22,
        "upvoteRatio": 0.74,
        "commentCount": 23,
        "createdAt": "2025-10-27T15:16:29.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 111622
      },
      {
        "id": "1rdfm7a",
        "name": "t3_1rdfm7a",
        "title": "Weekly Webscrapers - Hiring, FAQs, etc",
        "author": "AutoModerator",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1rdfm7a/weekly_webscrapers_hiring_faqs_etc/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1rdfm7a/weekly_webscrapers_hiring_faqs_etc/",
        "bodyText": "**Welcome to the weekly discussion thread!**\n\nThis is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:\n\n* Hiring and job opportunities\n* Industry news, trends, and insights\n* Frequently asked questions, like \"How do I scrape LinkedIn?\"\n* Marketing and monetization tips\n\nIf you're new to web scraping, make sure to check out the [Beginners Guide](https://webscraping.fyi) 🌱\n\nCommercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the [monthly thread](https://reddit.com/r/webscraping/about/sticky?num=1)",
        "score": 5,
        "upvotes": 5,
        "upvoteRatio": 0.86,
        "commentCount": 9,
        "createdAt": "2026-02-24T13:01:04.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1dxo99o",
        "name": "t3_1dxo99o",
        "title": "Self-hosted Webscraper",
        "author": "bluesanoo",
        "subreddit": "selfhosted",
        "subredditPrefixedName": "r/selfhosted",
        "url": "https://www.reddit.com/r/selfhosted/comments/1dxo99o/selfhosted_webscraper/",
        "permalink": "https://www.reddit.com/r/selfhosted/comments/1dxo99o/selfhosted_webscraper/",
        "bodyText": "I have created a self-hosted webscraper, \"Scraperr\". This is the first one I have seen on here and its pretty simple, but I could add more features to it in the future.   \n[https://github.com/jaypyles/Scraperr](https://github.com/jaypyles/Scraperr)\n\nCurrently you can:   \n- Scrape sites using xpath elements  \n- Download and view results of scrape jobs  \n- Rerun scrape jobs\n\nFeel free to leave suggestions",
        "score": 122,
        "upvotes": 122,
        "upvoteRatio": 0.96,
        "commentCount": 55,
        "createdAt": "2024-07-07T19:11:25.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 766144
      },
      {
        "id": "1swxrpl",
        "name": "t3_1swxrpl",
        "title": "[HIRING] Part-time webscraper, remote, 20h/week",
        "author": "betherosports",
        "subreddit": "webscraping",
        "subredditPrefixedName": "r/webscraping",
        "url": "https://www.reddit.com/r/webscraping/comments/1swxrpl/hiring_parttime_webscraper_remote_20hweek/",
        "permalink": "https://www.reddit.com/r/webscraping/comments/1swxrpl/hiring_parttime_webscraper_remote_20hweek/",
        "bodyText": "Hey everyone,\n\nWe're [Bet Hero](https://betherosports.com/), a sports betting analytics company that scrapes sportsbooks in real time to flag mispriced odds before the books correct them.\n\n# The work\n\nMost of the job is keeping scrapers alive. Books change their website, update endpoints, geo-block proxies, etc. You diagnose, fix, move on. Plus adding new books (\\~400 references to copy from) and fighting anti-bot: Cloudflare, Akamai, Datadome, TLS fingerprinting, plus books that rolled their own.\n\n# What we want\n\nA couple of years scraping things that don't want to be scraped. Solid Python. You've fought antibots and won. **No headless browsers.** If Playwright is your default tool, this isn't the role.\n\nNice-to-haves: Rust, Go, K8s, Ray and sports betting knowledge.\n\n# Interested?\n\nShoot us a DM :)",
        "score": 11,
        "upvotes": 11,
        "upvoteRatio": 0.68,
        "commentCount": 0,
        "createdAt": "2026-04-27T08:36:00.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 94589
      },
      {
        "id": "1agv0xn",
        "name": "t3_1agv0xn",
        "title": "Is Webscraping a work of a data scientist?",
        "author": "medyosuper",
        "subreddit": "datascience",
        "subredditPrefixedName": "r/datascience",
        "url": "https://www.reddit.com/r/datascience/comments/1agv0xn/is_webscraping_a_work_of_a_data_scientist/",
        "permalink": "https://www.reddit.com/r/datascience/comments/1agv0xn/is_webscraping_a_work_of_a_data_scientist/",
        "bodyText": "I used to believe that the work of a data scientist is from getting the data to creating models. But if solely web scraping, is that a job for data scientist or are there any discipline to better do that job?",
        "score": 80,
        "upvotes": 80,
        "upvoteRatio": 0.88,
        "commentCount": 78,
        "createdAt": "2024-02-02T04:19:29.000Z",
        "isVideo": false,
        "subredditSubscriberCount": 2744518
      }
    ],
    "totalResults": 25,
    "page": {
      "nextCursor": "t3_1agv0xn",
      "hasMore": true
    }
  },
  "meta": {
    "requestId": "req_01example_page2",
    "creditsCharged": 1,
    "version": "v1"
  }
}
```

### Machine-readable error codes

When an error JSON body is returned, it may include one of these `error.code` values (derived from the OpenAPI schemas for this operation; additional codes may exist at runtime):

- `bad_request`

## Error handling & retries

Interpret HTTP status codes using the descriptions below. Do not assume a JSON body unless the OpenAPI schema defines one for that status.

- **400**: Invalid query parameters **Retry:** Fix the request; retrying the same invalid payload will not help.
- **401**: Missing or invalid API key **Retry:** Fix the API key first; retrying without changes will not help.
- **402**: Insufficient credits **Retry:** Do not retry without resolving billing/credits (retrying the same request will not help).
- **500**: Unexpected or billing error
- **502**: Lookup could not be completed from the response (unexpected or invalid data). **Retry:** May be transient; a few retries with backoff are reasonable.
- **503**: Service temporarily unavailable; safe to retry with backoff. **Retry:** Usually safe to retry with exponential backoff and jitter.

### Suggested client defaults

- Send the API key using the `x-api-key` header on every request.
- On `503` (and sometimes `502`), retry with backoff; cap retries and surface a clear error to the user.
- On `402`, surface an actionable billing message rather than blind retries.

## Examples

### TypeScript SDK

```typescript
import { SocialFetchClient } from "@socialfetch/sdk";

const client = new SocialFetchClient({
  apiKey: process.env.SOCIALFETCH_API_KEY!,
});

const result = await client.reddit.search({
  query: "web scraping",
});

if (!result.ok) {
  console.error(result.error);
} else {
  console.log(result.value.data);
}
```

### Node.js

```javascript
const response = await fetch(
  "https://api.socialfetch.dev/v1/reddit/search?query=web scraping",
  {
    headers: {
      "x-api-key": "YOUR_API_KEY",
    }
  }
);

const data = await response.json();
console.log(data);
```

### cURL

```bash
curl "https://api.socialfetch.dev/v1/reddit/search?query=web scraping" \
  -H "x-api-key: YOUR_API_KEY"
```

### Python

```python
import requests

response = requests.get(
    "https://api.socialfetch.dev/v1/reddit/search?query=web scraping",
    headers={"x-api-key": "YOUR_API_KEY"},
)
data = response.json()
print(data)
```

### Example: optional query parameters

First request illustrating common optional query flags (adjust values to your integration):

```javascript
const url = "https://api.socialfetch.dev/v1/reddit/search?sortBy=relevance";
const response = await fetch(url, {
  headers: { "x-api-key": "YOUR_API_KEY" },
});
const data = await response.json();
```

### Example: next page (pagination)

After a successful response, if pagination is not finished, request the next page using `cursor` (URL-encode when composing the query string):

```javascript
const previous = await response.json();
const nextCursor = previous?.data?.page?.nextCursor;
const hasMore = previous?.data?.page?.hasMore;
if (!hasMore || nextCursor == null) {
  // no more pages
} else {
  const nextUrl = new URL("https://api.socialfetch.dev/v1/reddit/search");
  nextUrl.searchParams.set("cursor", nextCursor);
  // optionally preserve sort: nextUrl.searchParams.set("sortBy", "latest");
  const nextResponse = await fetch(nextUrl.toString(), {
    headers: { "x-api-key": "YOUR_API_KEY" },
  });
  const nextData = await nextResponse.json();
}
```