Skip to content

docs(academy): add tutorial for scraping Shopify stores via JSON endpoints#2736

Draft
DaveHanns wants to merge 1 commit into
masterfrom
docs-shopify-json-api-tutorial
Draft

docs(academy): add tutorial for scraping Shopify stores via JSON endpoints#2736
DaveHanns wants to merge 1 commit into
masterfrom
docs-shopify-json-api-tutorial

Conversation

@DaveHanns

Copy link
Copy Markdown
Contributor

Summary

Adds a new Apify Academy tutorial that documents the JSON endpoints Shopify storefronts expose, and shows how to scrape them from a Crawlee Actor:

  • /collections.json — enumerates public collections
  • /collections/{handle}/products.json?limit=250&page=N — paginates a collection's products with variants, prices, images
  • /products.json?limit=250&page=N — store-wide equivalent

The tutorial leads with the JSON pattern, shows a paginating loop that pushes to a Crawlee Dataset, contrasts with HTML scraping, and lists caveats (disabled endpoints, ~10k page cap, rate limits, published-only products).

Motivation

Almost every Shopify store exposes an unauthenticated JSON API that returns exactly the data most scrapers actually want — variants, prices, availability, SKUs, image URLs, tags — without HTML parsing, without a headless browser, and with deterministic pagination. It is dramatically cheaper and more reliable than scraping the rendered storefront.

Today the Academy has no tutorial that points at this pattern. Developers writing a Shopify scraper tend to start with a headless browser (apify/actor-node-puppeteer-chrome), fight with theme-dependent selectors and infinite-scroll widgets, and only rediscover /collections.json after burning time on the HTML path. This tutorial makes the JSON path the default recommendation.

Evidence that the endpoints work

Public Shopify storefronts return JSON at these paths without authentication. Example:

$ curl -sSf 'https://www.allbirds.com/collections.json' | jq '.collections[0] | {id, handle, title, products_count}'
{
  "id": 268053250,
  "handle": "mens",
  "title": "Men's Shoes",
  "products_count": 46
}

$ curl -sSf 'https://www.allbirds.com/collections/mens/products.json?limit=250&page=1' | jq '.products[0] | {id, handle, title, variants: (.variants | length)}'
{
  "id": 1234567890,
  "handle": "mens-wool-runner",
  "title": "Men's Wool Runners",
  "variants": 18
}

Contrast with parsing the same data out of the storefront HTML: hundreds of PDP requests, JS-heavy pages, theme-specific selectors, and pagination that varies per storefront theme.

What this PR changes

  • New file sources/academy/tutorials/node_js/scraping_shopify_stores.mdsidebar_position: 15.3 (fills a gap in the existing node_js tutorial ordering), slug /node-js/scraping-shopify-stores.

No existing files are modified.

Test plan

  • npm run start renders the new tutorial without warnings.
  • Sidebar shows the new entry in the Academy → Tutorials → Node.js section at position 15.3.
  • Code blocks (js, console) render with the correct syntax highlighting.
  • Internal links (/platform/proxy) resolve.

Follow-ups (out of scope for this PR)

Suggest also adding a one-line hint in the Apify agent skill index pointing at this tutorial (separate repo, separate PR).


Surfaced during an evaluation of Apify surfaces for agent-driven Actor development.

…oints

Most Shopify storefronts expose their catalog as unauthenticated JSON at
/collections.json and /collections/{handle}/products.json?limit=250&page=N.
These endpoints return structured product data with variants, prices, and
images, and paginate deterministically — dramatically cheaper and more
reliable than HTML-scraping a JS-heavy storefront.

There's currently no Apify Academy tutorial pointing developers at this
pattern, so people writing a Shopify scraper tend to start with a headless
browser and rediscover the JSON endpoints only after burning time on the
HTML path. This tutorial documents the two endpoints, shows a paginating
loop that pushes to a Crawlee Dataset, contrasts with HTML scraping, and
notes the caveats (disabled endpoints, ~10k page cap, rate limits).

Surfaced during an evaluation of Apify surfaces for agent-driven Actor development.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@apify-service-account

apify-service-account commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

✅ Preview for this PR (commit 3c778b3) is ready at https://pr-2736.preview.docs.apify.com (see action run).

@DaveHanns DaveHanns marked this pull request as draft July 5, 2026 11:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants