docs(academy): add tutorial for scraping Shopify stores via JSON endpoints#2736
Draft
DaveHanns wants to merge 1 commit into
Draft
docs(academy): add tutorial for scraping Shopify stores via JSON endpoints#2736DaveHanns wants to merge 1 commit into
DaveHanns wants to merge 1 commit into
Conversation
…oints
Most Shopify storefronts expose their catalog as unauthenticated JSON at
/collections.json and /collections/{handle}/products.json?limit=250&page=N.
These endpoints return structured product data with variants, prices, and
images, and paginate deterministically — dramatically cheaper and more
reliable than HTML-scraping a JS-heavy storefront.
There's currently no Apify Academy tutorial pointing developers at this
pattern, so people writing a Shopify scraper tend to start with a headless
browser and rediscover the JSON endpoints only after burning time on the
HTML path. This tutorial documents the two endpoints, shows a paginating
loop that pushes to a Crawlee Dataset, contrasts with HTML scraping, and
notes the caveats (disabled endpoints, ~10k page cap, rate limits).
Surfaced during an evaluation of Apify surfaces for agent-driven Actor development.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
|
✅ Preview for this PR (commit |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new Apify Academy tutorial that documents the JSON endpoints Shopify storefronts expose, and shows how to scrape them from a Crawlee Actor:
/collections.json— enumerates public collections/collections/{handle}/products.json?limit=250&page=N— paginates a collection's products with variants, prices, images/products.json?limit=250&page=N— store-wide equivalentThe tutorial leads with the JSON pattern, shows a paginating loop that pushes to a Crawlee
Dataset, contrasts with HTML scraping, and lists caveats (disabled endpoints, ~10k page cap, rate limits, published-only products).Motivation
Almost every Shopify store exposes an unauthenticated JSON API that returns exactly the data most scrapers actually want — variants, prices, availability, SKUs, image URLs, tags — without HTML parsing, without a headless browser, and with deterministic pagination. It is dramatically cheaper and more reliable than scraping the rendered storefront.
Today the Academy has no tutorial that points at this pattern. Developers writing a Shopify scraper tend to start with a headless browser (
apify/actor-node-puppeteer-chrome), fight with theme-dependent selectors and infinite-scroll widgets, and only rediscover/collections.jsonafter burning time on the HTML path. This tutorial makes the JSON path the default recommendation.Evidence that the endpoints work
Public Shopify storefronts return JSON at these paths without authentication. Example:
Contrast with parsing the same data out of the storefront HTML: hundreds of PDP requests, JS-heavy pages, theme-specific selectors, and pagination that varies per storefront theme.
What this PR changes
sources/academy/tutorials/node_js/scraping_shopify_stores.md—sidebar_position: 15.3(fills a gap in the existingnode_jstutorial ordering), slug/node-js/scraping-shopify-stores.No existing files are modified.
Test plan
npm run startrenders the new tutorial without warnings.js,console) render with the correct syntax highlighting./platform/proxy) resolve.Follow-ups (out of scope for this PR)
Suggest also adding a one-line hint in the Apify agent skill index pointing at this tutorial (separate repo, separate PR).
Surfaced during an evaluation of Apify surfaces for agent-driven Actor development.