API Documentation · Magpie Scraping

🔐 Authentication

All endpoints require a Bearer token in the Authorization header:

Authorization: your-api-token

Contact us for trial access or manage your token in Magpie Data.

⏳ Approval Flow

Jobs for Variant Sold, Merchant Items and Search Items endpoints go through manual approval:

You submit a job → Status: AWAITING_APPROVAL
Our team reviews and approves (within 24 hours)
Job starts processing → Status: PROCESSING
Results ready → Status: COMPLETED or PARTIAL_COMPLETE

Note: Shopee Product Page, Tokopedia PDP Bulk, Blibli PDP Bulk, TikTok PDP Bulk, Amazon PDP Bulk, and Amazon Search Bulk do NOT require approval - tasks are processed immediately.

1. Variant Sold

Batch scraping for Shopee product variant and sales data. Up to 50,000 items per request.

Supported regions: ID · SG · TH · PH · MY · VN · TW · BR

🔑 Endpoint

POST /v1/shopee/variant_sold_v1/submit
GET  /v1/shopee/variant_sold_v1/retrieve/{job_id}
GET  /v1/shopee/variant_sold_v1/download/{job_id}

📥 Request Body

Parameter	Type	Required	Description
`country`	string	Yes	Country code: `id`, `sg`, `th`, `ph`, `my`, `vn`, `tw`, `br`
`items`	array	Yes	Array of objects with `item_id` and `shop_id` (max 50,000)

💻 Code Examples

# Submit Job
curl -X POST "https://api.magpieiq.com/scraping/v1/shopee/variant_sold_v1/submit" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "country": "id",
    "items": [
      {"item_id": 13490493252, "shop_id": 651735813},
      {"item_id": 12345678901, "shop_id": 987654321}
    ]
  }'

# Poll Status
curl "https://api.magpieiq.com/scraping/v1/shopee/variant_sold_v1/retrieve/{job_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

# Download Results
curl "https://api.magpieiq.com/scraping/v1/shopee/variant_sold_v1/download/{job_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

View Python Example

import requests
import time

API_HOST = "https://api.magpieiq.com/scraping"
headers = {
    "Authorization": "Bearer YOUR_API_TOKEN",
    "Content-Type": "application/json"
}

# 1. Submit Job
payload = {
    "country": "id",
    "items": [
        {"item_id": 13490493252, "shop_id": 651735813},
        {"item_id": 12345678901, "shop_id": 987654321}
    ]
}
response = requests.post(f"{API_HOST}/v1/shopee/variant_sold_v1/submit", json=payload, headers=headers)
job_id = response.json()["job_id"]
print(f"Job submitted: {job_id}")

# 2. Poll until complete
while True:
    status_resp = requests.get(f"{API_HOST}/v1/shopee/variant_sold_v1/retrieve/{job_id}", headers=headers)
    status = status_resp.json()["status"]
    print(f"Status: {status}")
    if status in ["COMPLETED", "PARTIAL_COMPLETE", "FAILED"]:
        break
    time.sleep(30)

# 3. Download results
download_resp = requests.get(f"{API_HOST}/v1/shopee/variant_sold_v1/download/{job_id}", headers=headers)
print(download_resp.json())

Behavior

Cost: billed per item — see the rate card
Approval: Required (24h timeout)
Batching: Items are chunked into batches of 50 for processing
Results: Merged into a single JSON file per job
Collection time: collected_at is the source collection time as an integer Unix timestamp in UTC seconds.

Result Example

Variant Sold JSON Click to expand

{
  "collected_at": 1758006720,
  "cbOption": 0,
  "cmtCount": 56,
  "condition": 1,
  "createTime": 1651126772,
  "currency": "IDR",
  "description": "Philips Sonicare Brush Head Opt Wht 4x Wht HX6064/67\n\nGigi lebih putih hingga 100% hanya dalam satu minggu*. \nKepala sikat W2 Optimal White sangat cocok untuk mereka yang ingin melakukan lebih dari sekadar pembersihan mendalam untuk menghilangkan noda di permukaan untuk senyum putih dan bercahaya. Kepala sikat ini juga bagus untuk menjaga kecerahan di antara perawatan pemutihan profesional. Bulu sikat berkualitas tinggi yang padat menghilangkan plak hingga 7x lebih banyak daripada sikat gigi manual.\n\nKepala sikat menjadi kurang efektif setelah 3 bulan penggunaan, tetapi dengan BrushSync™ Anda akan diingatkan sebelum hal ini terjadi. Sikat gigi pintar Anda akan melacak seberapa sering dan seberapa keras Anda menyikat, dan akan memberi tahu Anda kapan saatnya untuk mengganti.\n\nKepala sikat W2 Optimal White Anda sangat cocok dengan gagang sikat gigi Philips Sonicare, kecuali Philips One dan Essence. Cukup klik on dan off untuk penggantian dan pembersihan yang mudah.\n\nKeunggulan:\n1. Gigi lebih putih dalam 7 hari\n2. Penyandingan mode BrushSync™  \n3. Pemasangan mudah",
  "flag": 917504,
  "globalBrandId": 1146897,
  "images": [
    "id-11134207-7ra0s-mcvqfkodg4yg05",
    "id-11134207-7ra0u-mcvqfkodhjiw79",
    "id-11134207-7ra0h-mcvqfkodiy3cac",
    "id-11134207-7ra0i-mcvqfkodkcns17",
    "id-11134207-7ra0h-mcvqfkodlr88d6",
    "id-11134207-7ra0m-mcvqfkodn5so0d",
    "id-11134207-7ra0m-mcvqfkodokd43e"
  ],
  "itemCardV2": {
    "displayPrice": {
      "currency": "IDR",
      "discount": 0,
      "discountPrice": "44862400000",
      "discountText": {
        "buyerTranslatedText": {
          "source": "",
          "translatedLanguage": "",
          "translatedText": ""
        },
        "text": "-29%"
      },
      "displayFinalPriceIcon": true,
      "priceMask": false
    }
  },
  "itemId": "13490493252",
  "likedCount": 47,
  "localBrand": "",
  "localCatIds": null,
  …

Preview only — first 40 lines. Download the full sample JSON (4 KB)

2. Shopee Product Page

Extended product details from Shopee product detail page. Single task per request. No Approval Required

Supported regions: ID · SG · TH · PH · MY · VN · TW · BR

🔑 Endpoint

POST /v1/shopee/get_pc/submit
GET  /v1/shopee/get_pc/retrieve/{task_id}

📥 Request Body

Parameter	Type	Required	Description
`region`	string	Yes	Region code: `id`, `sg`, `th`, `ph`, `my`, `vn`, `tw`, `br`
`item_id`	string	Yes	Shopee item ID
`shop_id`	string	Yes	Shopee shop ID

💻 Code Examples

# Submit Task
curl -X POST "https://api.magpieiq.com/scraping/v1/shopee/get_pc/submit" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"region": "br", "shop_id": "34907167", "item_id": "40869177342"}'

# Retrieve Result (poll until status is "completed")
curl "https://api.magpieiq.com/scraping/v1/shopee/get_pc/retrieve/{task_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

View Python Example

import requests
import time

API_HOST = "https://api.magpieiq.com/scraping"
headers = {"Authorization": "Bearer YOUR_API_TOKEN", "Content-Type": "application/json"}

# Submit
resp = requests.post(f"{API_HOST}/v1/shopee/get_pc/submit",
    json={"region": "br", "shop_id": "34907167", "item_id": "40869177342"}, headers=headers)
task_id = resp.json()["task_id"]

# Poll until ready
while True:
    result = requests.get(f"{API_HOST}/v1/shopee/get_pc/retrieve/{task_id}", headers=headers).json()
    if result.get("status") == "completed":
        print(result["result"])
        break
    time.sleep(5)

📦 Bulk by CSV (get_pc_bulk)

Need many product pages at once? Upload a CSV of products instead of submitting one at a time. Billed per product id, charged upfront for the whole file — see the rate card. No Approval Required

POST /v1/shopee/get_pc_bulk/submit     (multipart/form-data)
GET  /v1/shopee/get_pc_bulk/tasks
POST /v1/shopee/get_pc_bulk/terminate/{task_id}
GET  /v1/shopee/get_pc_bulk/retrieve/{task_id}

CSV columns (header row required): region, shop_id, item_id. One row per product.

region,shop_id,item_id
id,34907167,40869177342
id,555954448,14016184405
id,308956069,21462729864

# Submit up to 2,000 products (result_format defaults to csv; jsonl is faster for large batches)
curl -X POST "https://api.magpieiq.com/scraping/v1/shopee/get_pc_bulk/submit" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -F "file=@products.csv" \
  -F "rescrape=false" \
  -F "result_format=jsonl"

# Recover all task IDs previously submitted by this user
curl "https://api.magpieiq.com/scraping/v1/shopee/get_pc_bulk/tasks" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

# Poll one task for current progress and its download_url
curl "https://api.magpieiq.com/scraping/v1/shopee/get_pc_bulk/retrieve/{task_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

Cost: the per-product-id rate × number of product rows, checked and deducted at submit — see the rate card
Validation: the file is rejected (and nothing charged) if columns are missing, a row lacks shop_id/item_id, or it isn't UTF-8
rescrape: false (default) may serve recent cached crawls; true forces fresh fetches
Task history: /tasks returns task IDs only; call /retrieve/{task_id} for current status
Results: same product-page JSON as the single endpoint, one result per row
Collection time: collected_at is the source collection time as an integer Unix timestamp in UTC seconds.

⚡ Behavior

Cost: billed per request — see the rate card
Approval: NOT required - immediate processing
Processing: Single task processing
Results: Full product detail page data including attributes, models, description
Collection time: collected_at is the source collection time as an integer Unix timestamp in UTC seconds.

Result Example

Shopee Product Page JSON Click to expand

{
    "collected_at": 1758006720,
    "bff_meta": null,
    "error": null,
    "error_msg": null,
    "data": {
        "item": {
            "item_id": 40869177342,
            "shop_id": 34907167,
            "item_status": "normal",
            "status": 1,
            "item_type": 0,
            "reference_item_id": "",
            "title": "Apple iPhone 15 - Garansi Resmi",
            "image": "id-11134207-82250-mkkn12wketc0d8",
            "label_ids": [
                1400066568,
                47,
                1000255,
                1000544,
                1000167,
                1000559,
                1000560,
                2018619,
                1000584,
                1012729,
                2018618,
                2153644,
                1428713,
                1718087960,
                844931064601283,
                1119699,
                1015914,
                700190087,
                1400285055,
                2213652,
                2008656,
                700765096,
                700005503,
                1049134,
  …

Preview only — first 40 lines. Download the full sample JSON (239 KB)

3. Search / Category Items

Scrape all products from Shopee search or category pages. Up to 50 URLs per request.

Supported regions: ID · SG · TH · PH · MY · VN · TW · BR

🔑 Endpoint

POST /v1/shopee/search-items/submit
GET  /v1/shopee/search-items/retrieve/{job_id}

📥 Request Body

Parameter	Type	Required	Description
`urls`	array	Yes	Array of Shopee search/category URLs (max 50)

🔗 Supported URL Formats

Search: https://shopee.co.id/search?keyword=iphone
Category: https://shopee.co.id/Elektronik-cat.11044070

💻 Code Examples

# Submit Job
curl -X POST "https://api.magpieiq.com/scraping/v1/shopee/search-items/submit" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://shopee.co.id/search?keyword=iphone",
      "https://shopee.co.id/Elektronik-cat.11044070"
    ]
  }'

# Poll Status & collect download URLs
curl "https://api.magpieiq.com/scraping/v1/shopee/search-items/retrieve/{job_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

# Download a specific result (download_url requires the same Bearer token)
curl "https://api.magpieiq.com/scraping/v1/shopee/pgnt/jobs/{job_id}/items/{item_token}/data" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

View Python Example

import requests
import time

API_HOST = "https://api.magpieiq.com/scraping"
headers = {"Authorization": "Bearer YOUR_API_TOKEN", "Content-Type": "application/json"}

# Submit
payload = {"urls": ["https://shopee.co.id/search?keyword=iphone"]}
resp = requests.post(f"{API_HOST}/v1/shopee/search-items/submit", json=payload, headers=headers)
job_id = resp.json()["job_id"]

# Poll until complete
while True:
    result = requests.get(f"{API_HOST}/v1/shopee/search-items/retrieve/{job_id}", headers=headers).json()
    if result["status"] in ["COMPLETED", "PARTIAL_COMPLETE"]:
        for item in result["items"]:
            print(f"URL: {item['url']}, Status: {item['status']}, Download: {item.get('download_url')}")
        break
    time.sleep(30)

⚡ Behavior

Cost: billed per URL — see the rate card
Approval: Required (24h timeout)
Region: Automatically extracted from URL (e.g., shopee.co.id → ID, shopee.com.br → BR)
Results: One JSON file per URL containing all items from the search/category
Collection time: collected_at is the source collection time as an integer Unix timestamp in UTC seconds.
Download: Each download_url points to /v1/shopee/pgnt/jobs/{job_id}/items/{token}/data and requires the same Bearer token.
Data source: Returns items from search_items (all pages)

Result Example

Search Items JSON Click to expand

{
  "collected_at": 1788339600,
  "success": true,
  "url": "https://shopee.co.id/search?keyword=iphone",
  "total_items": 94,
  "items": [
    {
      "itemid": 40011228763,
      "shopid": 24843249,
      "name": "iPhone 15 Pro Max 256GB",
      "price": 800000000,
      "price_min": 800000000,
      "price_max": 800000000,
      "stock": 1,
      "sold": 1000,
      "historical_sold": 1000,
      "liked_count": 221,
      "cmt_count": 624,
      "item_rating": {
        "rating_star": 4.815705128205129,
        "rating_count": [624, 21, 2, 7, 11, 583]
      },
      "shop_location": "KOTA JAKARTA UTARA",
      "shop_name": "tokojualbarangmurahbanget",
      "is_official_shop": false,
      "tier_variations": [
        {
          "name": "Garansi",
          "options": ["1 Bulan", "3 Bulan", "6 Bulan", "Lifetime"]
        }
      ]
    }
  ]
}
  …

Preview only — first 40 lines. Download the full sample JSON (859 B)

4. Merchant Items

Scrape all products from Shopee merchant/shop pages. Up to 50 URLs per request.

Supported regions: ID · SG · TH · PH · MY · VN · TW · BR

🔑 Endpoint

POST /v1/shopee/merchant-items/submit
GET  /v1/shopee/merchant-items/retrieve/{job_id}

📥 Request Body

Parameter	Type	Required	Description
`urls`	array	Yes	Array of Shopee shop/merchant URLs (max 50)

🔗 Supported URL Formats

Shop ID: https://shopee.co.id/shop/24843249

💻 Code Examples

# Submit Job
curl -X POST "https://api.magpieiq.com/scraping/v1/shopee/merchant-items/submit" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://shopee.co.id/shop/24843249",
      "https://shopee.co.id/shop/651735813"
    ]
  }'

# Poll Status & collect download URLs
curl "https://api.magpieiq.com/scraping/v1/shopee/merchant-items/retrieve/{job_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

# Download a specific result
curl "https://api.magpieiq.com/scraping/v1/shopee/pgnt/jobs/{job_id}/items/{item_token}/data" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

View Python Example

import requests
import time

API_HOST = "https://api.magpieiq.com/scraping"
headers = {"Authorization": "Bearer YOUR_API_TOKEN", "Content-Type": "application/json"}

# Submit
payload = {"urls": ["https://shopee.co.id/shop/24843249"]}
resp = requests.post(f"{API_HOST}/v1/shopee/merchant-items/submit", json=payload, headers=headers)
job_id = resp.json()["job_id"]

# Poll & download results
while True:
    result = requests.get(f"{API_HOST}/v1/shopee/merchant-items/retrieve/{job_id}", headers=headers).json()
    if result["status"] in ["completed", "partial_complete"]:
        for item in result["items"]:
            if item["status"] == "SUCCESS" and item["download_url"]:
                data = requests.get(f"{API_HOST}{item['download_url']}", headers=headers).json()
                print(item["url"], data.keys())
        break
    time.sleep(30)

⚡ Behavior

Cost: billed per URL — see the rate card
Approval: Required (24h timeout)
Region: Automatically extracted from URL (e.g., shopee.co.id → ID, shopee.com.br → BR)
Results: One JSON file per shop URL containing all products
Collection time: collected_at is the source collection time as an integer Unix timestamp in UTC seconds.
Download: Use the download_url from retrieve to call /v1/shopee/pgnt/jobs/{job_id}/items/{token}/data with your Bearer token.
Data sources: Returns items from both:
- rcmd_items - Recommended/active items
- search_items?filter_sold_out=1 - Sold out items (if any)

Result Example

Merchant Items JSON Click to expand

{
  "success": true,
  "shops": {
    "https://shopee.co.id/shop/24843249": {
      "page_url": "https://shopee.co.id/shop/24843249",
      "page_type": "shop",
      "page_params": {
        "shop_id": 24843249
      },
      "items": {
        "items": [],
        "total_fetched": 0,
        "total_count": 0,
        "pages_fetched": 0,
        "has_more": true
      },
      "rcmd_items": {
        "items": [
          {
            "itemid": 40011228763,
            "shopid": 24843249,
            "name": "Lifetime N3TFLXX Premium UHD 4K Akun Netflix",
            "label_ids": [
              2018619,
              700025282,
              1718087960,
              1428713,
              1059152,
              1049122,
              822059908662278,
              822120592853526,
              1015914,
              700190087,
              700830032,
              298933384,
              1400285055
            ],
            "image": "id-11134207-7ra0i-mcwe7x8ljik5bb",
            "images": [
              "id-11134207-7ra0i-mcwe7x8ljik5bb"
  …

Preview only — first 40 lines. Download the full sample JSON (111 KB)

5. Tokopedia PDP Bulk Beta

Scrape product details from multiple Tokopedia product pages in one request. No Approval Required

Region: ID (Tokopedia Indonesia)

🔑 Endpoint

POST /v1/tokopedia/pdp/bulk/submit
GET  /v1/tokopedia/pdp/bulk/retrieve/{job_id}

📥 Request Body

Parameter	Type	Required	Description
`items`	array	Yes	Array of objects with `product_key` and `shop_domain`. Max 2,000 items per request.

💻 Code Examples

# Submit Bulk Job (max 2,000 items)
curl -X POST "https://api.magpieiq.com/scraping/v1/tokopedia/pdp/bulk/submit" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {"product_key": "huawei-freebuds-se-4-anc-tws-l-up-to-50db-l-10mm-dynamic-driver-50h-long-battery-life-l-ip54-1732996711831668540", "shop_domain": "huawei"},
      {"product_key": "kasur-pocket-spring-bed-turu-boss-ukuran-160x200-queen-free-bantal-kasur-58791", "shop_domain": "turubed"}
    ]
  }'

# Retrieve Job Status (poll until status is "completed")
curl "https://api.magpieiq.com/scraping/v1/tokopedia/pdp/bulk/retrieve/{job_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

# Download the compiled JSONL result (download_url from retrieve response)
curl -o result.jsonl.gz "https://storage.googleapis.com/magpie_dev/..."

View Python Example

import requests
import time
import gzip
import json

API_HOST = "https://api.magpieiq.com/scraping"
headers = {"Authorization": "Bearer YOUR_API_TOKEN", "Content-Type": "application/json"}

# 1. Submit Bulk Job (max 2,000 items)
payload = {
    "items": [
        {"product_key": "product-slug-1", "shop_domain": "shop-1"},
        {"product_key": "product-slug-2", "shop_domain": "shop-2"},
    ]
}
resp = requests.post(f"{API_HOST}/v1/tokopedia/pdp/bulk/submit", json=payload, headers=headers)
job_id = resp.json()["job_id"]
print(f"Job submitted: {job_id}")

# 2. Poll until complete
while True:
    result = requests.get(f"{API_HOST}/v1/tokopedia/pdp/bulk/retrieve/{job_id}", headers=headers).json()
    print(f"Status: {result['status']} ({result['success']}/{result['total']} done)")
    if result["status"] in ["completed", "partial_complete", "failed"]:
        break
    time.sleep(30)

# 3. Download the compiled JSONL file (gzip-compressed)
if result.get("download_url"):
    resp = requests.get(result["download_url"])
    with open("result.jsonl.gz", "wb") as f:
        f.write(resp.content)
    # Decompress and read
    with gzip.open("result.jsonl.gz", "rt") as f:
        for line in f:
            record = json.loads(line)
            print(record["product_key"], record["shop_domain"], record["response"].keys())
else:
    print(f"No download_url yet: {result.get('message')}")

⚡ Behavior

Cost: billed per item — see the rate card (beta pricing)
Approval: NOT required - immediate processing
Batch limit: Up to 2,000 items per request
Results: All results are compiled into a single gzip-compressed JSONL file. The download_url in the retrieve response points directly to the file (no Bearer token needed — it's a signed GCS URL valid for 7 days).
JSONL format: Each line is a JSON object with shop_domain, product_key, and response fields
Partial results: If some items fail, credits are refunded for failed items only
Caching: Retrieve responses are cached for 120s while processing and 24h after completion

Retrieve Response Example

Tokopedia PDP Bulk Retrieve JSON Click to expand

// While processing
{
  "job_id": "233f82dc-942a-496a-b6ee-dff920a68775",
  "status": "processing",
  "total": 2000,
  "success": 850,
  "failed": 0,
  "pending": 1150,
  "progress": 0.425,
  "download_url": null,
  "message": null
}

// All items scraped, JSONL being compiled
{
  "job_id": "233f82dc-942a-496a-b6ee-dff920a68775",
  "status": "completed",
  "total": 2000,
  "success": 2000,
  "failed": 0,
  "pending": 0,
  "progress": 1.0,
  "download_url": null,
  "message": "All items scraped. Results are being compiled into a JSONL file — please wait."
}

// Completed — JSONL ready
{
  "job_id": "233f82dc-942a-496a-b6ee-dff920a68775",
  "status": "completed",
  "total": 2000,
  "success": 2000,
  "failed": 0,
  "pending": 0,
  "progress": 1.0,
  "download_url": "https://storage.googleapis.com/magpie_dev/tokopedia/bulk_result/{job_id}/result.jsonl.gz?X-Goog-Algorithm=...",
  "message": null
}

Result Data Example

Each line in the JSONL file contains the full product detail data in the response field:

Tokopedia PDP Result JSON Click to expand

[
  {
    "data": {
      "variants": {
        "basicInfo": {
          "id": 1234567890,
          "shopID": 987654,
          "alias": "no-void-minds-formlock-oversized-t-shirt-core",
          "createdAt": "2024-10-21T08:00:00Z",
          "category": {
            "detail": {
              "id": 1009,
              "name": "T-Shirt",
              "breadcrumbURL": "https://www.tokopedia.com/p/fashion-pria/kemeja"
            }
          },
          "stats": {
            "countView": 1500,
            "countReview": 42,
            "countTalk": 5,
            "rating": 4.8
          },
          "txStats": {
            "transactionSuccess": 120,
            "transactionReject": 3,
            "countSold": 120
          },
          "minOrder": 1,
          "maxOrder": 100,
          "weight": 0.2,
          "weightUnit": "kg",
          "url": "https://www.tokopedia.com/novoidminds/no-void-minds-formlock-oversized-t-shirt-core",
          "condition": "NEW",
          "status": "ACTIVE",
          "isLeasing": false,
          "catalogID": "",
          "menu": [
            {"name": "Fashion Pria"},
            {"name": "T-Shirt"}
          ],
  …

Preview only — first 40 lines. Download the full sample JSON (4 KB)

6. Blibli PDP Bulk Beta

Scrape product details from multiple Blibli product pages in one request. No Approval Required

Region: ID (Blibli Indonesia)

🔑 Endpoint

POST /v1/blibli/pdp/bulk/submit
GET  /v1/blibli/pdp/bulk/retrieve/{job_id}

📥 Request Body

Parameter	Type	Required	Description
`items`	array	Yes	Array of objects with `formattedId` and `sku` (optional: `pickupPointCode`, `id`). Max 2,000 items per request.

💻 Code Examples

# Submit Bulk Job (max 2,000 items)
curl -X POST "https://api.magpieiq.com/scraping/v1/blibli/pdp/bulk/submit" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {"formattedId": "ps--16M-70001-00016", "sku": "16M-70001-00016-00001"},
      {"formattedId": "ps--16M-70001-00017", "sku": "16M-70001-00017-00001", "pickupPointCode": "PP-3491245"}
    ]
  }'

# Retrieve Job Status (poll until status is "completed")
curl "https://api.magpieiq.com/scraping/v1/blibli/pdp/bulk/retrieve/{job_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

# Download the compiled JSONL result (download_url from retrieve response)
curl -o result.jsonl.gz "https://storage.googleapis.com/magpie_dev/..."

View Python Example

import requests
import time
import gzip
import json

API_HOST = "https://api.magpieiq.com/scraping"
headers = {"Authorization": "Bearer YOUR_API_TOKEN", "Content-Type": "application/json"}

# 1. Submit Bulk Job (max 2,000 items)
payload = {
    "items": [
        {"formattedId": "ps--16M-70001-00016", "sku": "16M-70001-00016-00001"},
        {"formattedId": "ps--16M-70001-00017", "sku": "16M-70001-00017-00001", "pickupPointCode": "PP-3491245"},
    ]
}
resp = requests.post(f"{API_HOST}/v1/blibli/pdp/bulk/submit", json=payload, headers=headers)
job_id = resp.json()["job_id"]
print(f"Job submitted: {job_id}")

# 2. Poll until complete
while True:
    result = requests.get(f"{API_HOST}/v1/blibli/pdp/bulk/retrieve/{job_id}", headers=headers).json()
    print(f"Status: {result['status']} ({result['success']}/{result['total']} done)")
    if result["status"] in ["completed", "partial_complete", "failed"]:
        break
    time.sleep(30)

# 3. Download the compiled JSONL file (gzip-compressed)
if result.get("download_url"):
    resp = requests.get(result["download_url"])
    with open("result.jsonl.gz", "wb") as f:
        f.write(resp.content)
    # Decompress and read
    with gzip.open("result.jsonl.gz", "rt") as f:
        for line in f:
            record = json.loads(line)
            print(record["item_id"], record["response"].keys())
else:
    print(f"No download_url yet: {result.get('message')}")

⚡ Behavior

Cost: billed per item — see the rate card (beta pricing)
Approval: NOT required - immediate processing
Batch limit: Up to 2,000 items per request
Results: All results are compiled into a single gzip-compressed JSONL file. The download_url in the retrieve response points directly to the file (no Bearer token needed — it's a signed GCS URL valid for 7 days).
JSONL format: Each line is a JSON object with item_id, formattedId, sku, pickupPointCode, and response fields
Partial results: If some items fail, credits are refunded for failed items only
Caching: Retrieve responses are cached for 120s while processing and 24h after completion

Retrieve Response Example

Blibli PDP Bulk Retrieve JSON Click to expand

// While processing
{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "processing",
  "total": 2000,
  "success": 850,
  "failed": 0,
  "pending": 1150,
  "progress": 0.425,
  "download_url": null,
  "message": null
}

// All items scraped, JSONL being compiled
{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "completed",
  "total": 2000,
  "success": 2000,
  "failed": 0,
  "pending": 0,
  "progress": 1.0,
  "download_url": null,
  "message": "All items scraped. Results are being compiled into a JSONL file — please wait."
}

// Completed — JSONL ready
{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "completed",
  "total": 2000,
  "success": 2000,
  "failed": 0,
  "pending": 0,
  "progress": 1.0,
  "download_url": "https://storage.googleapis.com/magpie_dev/blibli/bulk_result/{job_id}/result.jsonl.gz?X-Goog-Algorithm=...",
  "message": null
}

JSONL Format

The download_url points to a gzip-compressed JSONL file. Each line is a JSON object:

Blibli PDP JSONL Line Example Click to expand

{"item_id": "16M-70001-00016-00001", "formattedId": "ps--16M-70001-00016", "sku": "16M-70001-00016-00001", "pickupPointCode": "PP-3491245", "response": { ... full Blibli PDP JSON ... }}

Result Data Example

The response field in each JSONL line contains the full Blibli PDP JSON:

Blibli PDP Result JSON Click to expand

[
  {
    "status": "OK",
    "statusCode": 200,
    "message": null,
    "data": {
      "summary": {
        "id": "16M-70001-00016",
        "name": "Sample Blibli Product",
        "brand": {
          "id": "BR-001",
          "name": "Sample Brand"
        },
        "category": {
          "id": "CAT-001",
          "name": "Electronics"
        },
        "price": {
          "price": 1500000,
          "originalPrice": 2000000,
          "discountPercentage": 25,
          "currency": "IDR"
        },
        "stock": 50,
        "sold": 120,
        "rating": 4.8,
        "totalReview": 42,
        "merchant": {
          "id": "M-001",
          "name": "Sample Official Store",
          "type": "OFFICIAL"
        },
        "url": "https://www.blibli.com/p/sample-product/ps--16M-70001-00016",
        "condition": "NEW",
        "status": "ACTIVE"
      },
      "items": [
        {
          "id": "16M-70001-00016-00001",
          "sku": "16M-70001-00016-00001",
  …

Preview only — first 40 lines. Download the full sample JSON (1 KB)

7. Tokopedia Search Bulk Beta

Scrape product listings from multiple Tokopedia search or category pages in one request. No Approval Required

Region: ID (Tokopedia Indonesia)

🔑 Endpoint

POST /v1/tokopedia/search/bulk/submit
GET  /v1/tokopedia/search/bulk/retrieve/{job_id}

📥 Request Body

Parameter	Type	Required	Description
`items`	array	Yes	Array of search task objects (max 50)
`items[].list_type`	string	No	`search` (default) or `category`
`items[].keyword`	string	Yes	Search keyword or numeric category ID
`items[].category`	string	No	Category slug (required when `list_type` is `category`)
`items[].pmin`	integer	No	Minimum price filter in IDR
`items[].product_limit`	integer	No	Max products to collect (default: 5000)

💻 Code Examples

# Submit Bulk Job
curl -X POST "https://api.magpieiq.com/scraping/v1/tokopedia/search/bulk/submit" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {"list_type": "search", "keyword": "air purifier", "product_limit": 100},
      {"list_type": "category", "keyword": "3935", "category": "elektronik_alat-pendingin-ruangan_air-conditioner"}
    ]
  }'

# Retrieve Job Status (poll until status is "completed")
curl "https://api.magpieiq.com/scraping/v1/tokopedia/search/bulk/retrieve/{job_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

View Python Example

import requests
import time

API_HOST = "https://api.magpieiq.com/scraping"
headers = {"Authorization": "Bearer YOUR_API_TOKEN", "Content-Type": "application/json"}

# 1. Submit Bulk Job
payload = {
    "items": [
        {"list_type": "search", "keyword": "air purifier", "product_limit": 100},
        {"list_type": "search", "keyword": "sepatu nike"},
    ]
}
resp = requests.post(f"{API_HOST}/v1/tokopedia/search/bulk/submit", json=payload, headers=headers)
job_id = resp.json()["job_id"]
print(f"Job submitted: {job_id}")

# 2. Poll until complete
while True:
    result = requests.get(f"{API_HOST}/v1/tokopedia/search/bulk/retrieve/{job_id}", headers=headers).json()
    print(f"Status: {result['status']} ({result['success']}/{result['total']} done)")
    if result["status"] in ["completed", "partial_complete", "failed"]:
        break
    time.sleep(30)

# 3. Download results for each successful item
for item in result["items"]:
    if item["status"] == "SUCCESS" and item.get("download_url"):
        data = requests.get(item["download_url"], headers=headers).json()
        print(f"\n{item['keyword']}: {len(data)} products")

⚡ Behavior

Cost: billed per item — see the rate card (beta pricing)
Approval: NOT required - immediate processing
Batch: Up to 50 search tasks per request
Results: Each item's result is fetched via its download_url (signed GCS URL, requires Bearer token)
Partial results: If some items fail, credits are refunded for failed items only

Retrieve Response Example

Tokopedia Search Bulk Retrieve JSON Click to expand

{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "completed",
  "total": 2,
  "success": 2,
  "failed": 0,
  "pending": 0,
  "items": [
    {
      "task_id": "N2Y1MmUzMzUt...",
      "keyword": "air purifier",
      "list_type": "search",
      "status": "SUCCESS",
      "download_url": "https://storage.googleapis.com/magpie_dev/..."
    },
    {
      "task_id": "YzVkNGRjNTIt...",
      "keyword": "3935",
      "list_type": "category",
      "status": "SUCCESS",
      "download_url": "https://storage.googleapis.com/magpie_dev/..."
    }
  ]
}

Result Data Example

The download_url returns a JSON array of products from the search results:

Tokopedia Search Result JSON Click to expand

[
  {
    "batch_id": "a1b2c3d4e5f6",
    "keyword": "air purifier",
    "list_type": "search",
    "products": [
      {
        "id": 1234567890,
        "name": "Sharp Air Purifier FP-J30EY",
        "price": 2500000,
        "original_price": 3000000,
        "discount_percentage": 17,
        "rating": 4.9,
        "review_count": 1250,
        "sold_count": 5000,
        "shop_id": 987654,
        "shop_name": "Sharp Official Store",
        "shop_domain": "sharp-official",
        "url": "https://www.tokopedia.com/sharp-official/sharp-air-purifier-fp-j30ey",
        "image_url": "https://images.tokopedia.net/img/example.jpg",
        "location": "Jakarta Pusat",
        "condition": "NEW",
        "free_shipping": true
      },
      {
        "id": 1234567891,
        "name": "Philips Air Purifier AC0820",
        "price": 1800000,
        "original_price": 2200000,
        "discount_percentage": 18,
        "rating": 4.8,
        "review_count": 890,
        "sold_count": 3200,
        "shop_id": 987655,
        "shop_name": "Philips Official Store",
        "shop_domain": "philips-official",
        "url": "https://www.tokopedia.com/philips-official/philips-air-purifier-ac0820",
        "image_url": "https://images.tokopedia.net/img/example2.jpg",
        "location": "Jakarta Barat",
        "condition": "NEW",
  …

Preview only — first 40 lines. Download the full sample JSON (1 KB)

8. Tokopedia Merchant Bulk Beta

Scrape all products from multiple Tokopedia merchant/shop pages in one request. No Approval Required

Region: ID (Tokopedia Indonesia)

🔑 Endpoint

POST /v1/tokopedia/merchant/bulk/submit
GET  /v1/tokopedia/merchant/bulk/retrieve/{job_id}

📥 Request Body

Parameter	Type	Required	Description
`items`	array	Yes	Array of merchant task objects (max 50)
`items[].shopid`	string	Yes	Tokopedia shop ID
`items[].etalase_id`	string	No	Etalase ID filter (default: `etalase` = all products)
`items[].keyword`	string	No	Keyword filter within the shop
`items[].max_pages`	integer	No	Max pages to fetch (omit for all pages)

💻 Code Examples

# Submit Bulk Job
curl -X POST "https://api.magpieiq.com/scraping/v1/tokopedia/merchant/bulk/submit" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {"shopid": "12345678", "max_pages": 5},
      {"shopid": "87654321", "keyword": "sepatu"}
    ]
  }'

# Retrieve Job Status (poll until status is "completed")
curl "https://api.magpieiq.com/scraping/v1/tokopedia/merchant/bulk/retrieve/{job_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

View Python Example

import requests
import time

API_HOST = "https://api.magpieiq.com/scraping"
headers = {"Authorization": "Bearer YOUR_API_TOKEN", "Content-Type": "application/json"}

# 1. Submit Bulk Job
payload = {
    "items": [
        {"shopid": "12345678", "max_pages": 5},
        {"shopid": "87654321"},
    ]
}
resp = requests.post(f"{API_HOST}/v1/tokopedia/merchant/bulk/submit", json=payload, headers=headers)
job_id = resp.json()["job_id"]
print(f"Job submitted: {job_id}")

# 2. Poll until complete
while True:
    result = requests.get(f"{API_HOST}/v1/tokopedia/merchant/bulk/retrieve/{job_id}", headers=headers).json()
    print(f"Status: {result['status']} ({result['success']}/{result['total']} done)")
    if result["status"] in ["completed", "partial_complete", "failed"]:
        break
    time.sleep(30)

# 3. Download results for each successful item
for item in result["items"]:
    if item["status"] == "SUCCESS" and item.get("download_url"):
        data = requests.get(item["download_url"], headers=headers).json()
        print(f"\nShop {item['shopid']}: {len(data)} products")

⚡ Behavior

Cost: billed per item — see the rate card (beta pricing)
Approval: NOT required - immediate processing
Batch: Up to 50 merchant tasks per request
Results: Each item's result is fetched via its download_url (signed GCS URL, requires Bearer token)
Partial results: If some items fail, credits are refunded for failed items only

Retrieve Response Example

Tokopedia Merchant Bulk Retrieve JSON Click to expand

{
  "job_id": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
  "status": "completed",
  "total": 2,
  "success": 2,
  "failed": 0,
  "pending": 0,
  "items": [
    {
      "task_id": "N2Y1MmUzMzUt...",
      "shopid": "12345678",
      "status": "SUCCESS",
      "download_url": "https://storage.googleapis.com/magpie_dev/..."
    },
    {
      "task_id": "YzVkNGRjNTIt...",
      "shopid": "87654321",
      "status": "SUCCESS",
      "download_url": "https://storage.googleapis.com/magpie_dev/..."
    }
  ]
}

Result Data Example

The download_url returns a JSON array of products from the merchant shop:

Tokopedia Merchant Result JSON Click to expand

[
  {
    "shopid": "12345678",
    "store_name": "Nike Official Store",
    "batch_id": "a1b2c3d4e5f6",
    "products": [
      {
        "id": 9876543210,
        "name": "Nike Air Force 1 '07 - White",
        "price": 1299000,
        "original_price": 1599000,
        "discount_percentage": 19,
        "rating": 4.9,
        "review_count": 2100,
        "sold_count": 8500,
        "etalase": "Sepatu Pria",
        "url": "https://www.tokopedia.com/nike-official/nike-air-force-1-07-white",
        "image_url": "https://images.tokopedia.net/img/nike-af1.jpg",
        "condition": "NEW",
        "stock": 120
      },
      {
        "id": 9876543211,
        "name": "Nike Dunk Low Retro - Black White",
        "price": 1499000,
        "original_price": 1799000,
        "discount_percentage": 17,
        "rating": 4.8,
        "review_count": 1500,
        "sold_count": 4200,
        "etalase": "Sepatu Pria",
        "url": "https://www.tokopedia.com/nike-official/nike-dunk-low-retro-black-white",
        "image_url": "https://images.tokopedia.net/img/nike-dunk.jpg",
        "condition": "NEW",
        "stock": 85
      }
    ],
    "total_products": 2
  }
]
  …

Preview only — first 40 lines. Download the full sample JSON (1 KB)

9. Blibli Merchant Bulk Beta

Scrape all products from multiple Blibli merchant/shop pages in one request. No Approval Required

Region: ID (Blibli Indonesia)

🔑 Endpoint

POST /v1/blibli/merchant/bulk/submit
GET  /v1/blibli/merchant/bulk/retrieve/{job_id}

📥 Request Body

Parameter	Type	Required	Description
`items`	array	Yes	Array of merchant task objects (max 50)
`items[].shopid`	string	Yes	Blibli merchant/shop ID
`items[].categoryids`	string	No	Comma-separated Blibli category IDs to filter by
`items[].keyword`	string	No	Keyword filter within the shop
`items[].brand`	string	No	Brand filter
`items[].limit`	integer	No	Max products to collect (default: 1500)

💻 Code Examples

# Submit Bulk Job
curl -X POST "https://api.magpieiq.com/scraping/v1/blibli/merchant/bulk/submit" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {"shopid": "SAM-10001", "limit": 1500},
      {"shopid": "SAM-10002", "keyword": "lamp", "brand": "Philips"}
    ]
  }'

# Retrieve Job Status (poll until status is "completed")
curl "https://api.magpieiq.com/scraping/v1/blibli/merchant/bulk/retrieve/{job_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

View Python Example

import requests
import time

API_HOST = "https://api.magpieiq.com/scraping"
headers = {"Authorization": "Bearer YOUR_API_TOKEN", "Content-Type": "application/json"}

# 1. Submit Bulk Job
payload = {
    "items": [
        {"shopid": "SAM-10001", "limit": 1500},
        {"shopid": "SAM-10002"},
    ]
}
resp = requests.post(f"{API_HOST}/v1/blibli/merchant/bulk/submit", json=payload, headers=headers)
job_id = resp.json()["job_id"]
print(f"Job submitted: {job_id}")

# 2. Poll until complete
while True:
    result = requests.get(f"{API_HOST}/v1/blibli/merchant/bulk/retrieve/{job_id}", headers=headers).json()
    print(f"Status: {result['status']} ({result['success']}/{result['total']} done)")
    if result["status"] in ["completed", "partial_complete", "failed"]:
        break
    time.sleep(30)

# 3. Download results for each successful item
for item in result["items"]:
    if item["status"] == "SUCCESS" and item.get("download_url"):
        data = requests.get(item["download_url"], headers=headers).json()
        print(f"\nShop {item['shopid']}: {len(data)} products")

⚡ Behavior

Cost: billed per item — see the rate card (beta pricing)
Approval: NOT required - immediate processing
Batch: Up to 50 merchant tasks per request
Results: Each item's result is fetched via its download_url (signed GCS URL, requires Bearer token)
Partial results: If some items fail, credits are refunded for failed items only

Retrieve Response Example

Blibli Merchant Bulk Retrieve JSON Click to expand

{
  "job_id": "c3d4e5f6-a7b8-9012-cdef-123456789012",
  "status": "completed",
  "total": 2,
  "success": 2,
  "failed": 0,
  "pending": 0,
  "items": [
    {
      "task_id": "N2Y1MmUzMzUt...",
      "shopid": "SAM-10001",
      "status": "SUCCESS",
      "download_url": "https://storage.googleapis.com/magpie_dev/..."
    },
    {
      "task_id": "YzVkNGRjNTIt...",
      "shopid": "SAM-10002",
      "status": "SUCCESS",
      "download_url": "https://storage.googleapis.com/magpie_dev/..."
    }
  ]
}

Result Data Example

The download_url returns a JSON array of products from the merchant shop:

Blibli Merchant Result JSON Click to expand

[
  {
    "shopid": "SAM-10001",
    "store_name": "Samsung Official Store",
    "batch_id": "a1b2c3d4e5f6",
    "products": [
      {
        "id": "PROD-001",
        "name": "Samsung Galaxy S24 Ultra 256GB",
        "price": 18999000,
        "original_price": 20999000,
        "discount_percentage": 10,
        "rating": 4.9,
        "review_count": 3500,
        "sold_count": 12000,
        "brand": "Samsung",
        "category": "Smartphone",
        "url": "https://www.blibli.com/p/samsung-galaxy-s24-ultra/ps--SAM-10001",
        "image_url": "https://www.blibli.com/images/example.jpg",
        "condition": "NEW",
        "stock": 200,
        "merchant": {
          "id": "SAM-10001",
          "name": "Samsung Official Store",
          "type": "OFFICIAL"
        }
      },
      {
        "id": "PROD-002",
        "name": "Samsung Galaxy Watch 6 Classic 47mm",
        "price": 5999000,
        "original_price": 6999000,
        "discount_percentage": 14,
        "rating": 4.8,
        "review_count": 890,
        "sold_count": 2300,
        "brand": "Samsung",
        "category": "Smartwatch",
        "url": "https://www.blibli.com/p/samsung-galaxy-watch-6/ps--SAM-10002",
        "image_url": "https://www.blibli.com/images/example2.jpg",
  …

Preview only — first 40 lines. Download the full sample JSON (1 KB)

10. TikTok PDP Bulk Beta

Scrape product details from multiple TikTok Shop product pages in one request. No Approval Required

Regions: ID, MY, SG, TH, VN, PH (per-item country)

🔑 Endpoint

POST /v1/tiktok/pdp/bulk/submit
GET  /v1/tiktok/pdp/bulk/retrieve/{job_id}

📥 Request Body

Parameter	Type	Required	Description
`items`	array	Yes	Array of TikTok PDP task objects (max 2,000)
`items[].product_id`	string	Yes	TikTok Shop product ID
`items[].country`	string	Yes	Country code: `id`, `my`, `sg`, `th`, `vn`, or `ph`

💻 Code Examples

# Submit Bulk Job (max 2,000 items, each with a country)
curl -X POST "https://api.magpieiq.com/scraping/v1/tiktok/pdp/bulk/submit" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {"product_id": "1234567890", "country": "id"},
      {"product_id": "9876543210", "country": "my"},
      {"product_id": "5555555555", "country": "sg"}
    ]
  }'

# Retrieve Job Status (poll until status is "completed")
curl "https://api.magpieiq.com/scraping/v1/tiktok/pdp/bulk/retrieve/{job_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

# Download the compiled JSONL result (download_url from retrieve response)
curl -L -o result.jsonl.gz ""

Python Example Click to expand

import requests, time

API_HOST = "https://api.magpieiq.com/scraping"
headers = {"Authorization": "Bearer YOUR_API_TOKEN", "Content-Type": "application/json"}

# 1. Submit bulk job
payload = {
    "items": [
        {"product_id": "1234567890", "country": "id"},
        {"product_id": "9876543210", "country": "my"},
    ]
}
resp = requests.post(f"{API_HOST}/v1/tiktok/pdp/bulk/submit", json=payload, headers=headers)
job_id = resp.json()["job_id"]
print(f"Job submitted: {job_id}")

# 2. Poll until complete
while True:
    result = requests.get(f"{API_HOST}/v1/tiktok/pdp/bulk/retrieve/{job_id}", headers=headers).json()
    print(f"Status: {result['status']} ({result['success']}/{result['total']} done)")
    if result["status"] in ["completed", "partial_complete", "failed"]:
        break
    time.sleep(30)

# 3. Download the compiled JSONL result
if result.get("download_url"):
    data = requests.get(result["download_url"]).content
    print(f"Downloaded {len(data)} bytes (gzip JSONL)")

⚡ Behavior

Cost: billed per item — see the rate card (beta pricing)
Approval: NOT required - immediate processing
Batch: Up to 2,000 items per request
Per-item country: Each item must specify a country (id, my, sg, th, vn, ph). Items are grouped by country and processed in country-specific batches.
Results: Single gzip-compressed JSONL file via download_url (signed GCS URL)
Partial results: If some items fail, credits are refunded for failed items only

Submit Response Example

{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "processing",
  "total": 3,
  "message": null
}

Retrieve Response Example

TikTok PDP Bulk Retrieve JSON Click to expand

{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "completed",
  "total": 3,
  "success": 3,
  "failed": 0,
  "pending": 0,
  "progress": 1.0,
  "download_url": "https://storage.googleapis.com/magpie-scraping-inhouse/tiktok/bulk_result/a1b2c3d4-e5f6-7890-abcd-ef1234567890/result.jsonl.gz?X-Goog-Algorithm=...",
  "message": null
}

JSONL Format

The download_url points to a gzip-compressed JSONL file. Each line is a JSON object:

TikTok PDP JSONL Line Example Click to expand

{"item_id": "1234567890", "country": "id", "response": { ... full TikTok PDP JSON ... }}

Result Data Example

The response field in each JSONL line contains the full TikTok PDP JSON:

TikTok PDP Result JSON Click to expand

[
  {
    "_scraped_at": "2026-07-23T12:00:00Z",
    "_product_id": "1234567890",
    "_country": "id",
    "product": {
      "id": "1234567890",
      "title": "Sample TikTok Shop Product",
      "description": "Sample product description from TikTok Shop",
      "category": {
        "id": "CAT-001",
        "name": "Electronics"
      },
      "brand": {
        "id": "BR-001",
        "name": "Sample Brand"
      },
      "price": {
        "price": 1500000,
        "originalPrice": 2000000,
        "discountPercentage": 25,
        "currency": "IDR"
      },
      "stock": 50,
      "sold": 120,
      "rating": 4.8,
      "totalReview": 42,
      "seller": {
        "id": "S-001",
        "name": "Sample Official Store",
        "type": "OFFICIAL"
      },
      "url": "https://www.tiktok.com/@samplestore/product/1234567890",
      "condition": "NEW",
      "status": "ACTIVE"
    },
    "skus": [
      {
        "id": "SKU-001",
        "name": "Default Variant",
  …

Preview only — first 40 lines. Download the full sample JSON (1 KB)

11. Amazon PDP Bulk Beta

Scrape product details from multiple Amazon product pages in one request. No Approval Required

Regions: SG, JP, US, UK, DE, FR, AU, IT, ES, CA (per-item country)

🔑 Endpoint

POST /v1/amazon/pdp/bulk/submit
GET  /v1/amazon/pdp/bulk/retrieve/{job_id}

📥 Request Body

Parameter	Type	Required	Description
`items`	array	Yes	Array of Amazon PDP task objects (max 2,000)
`items[].asin`	string	Yes	Amazon Standard Identification Number (ASIN)
`items[].country`	string	Yes	Country code: `sg`, `jp`, `us`, `uk`, `de`, `fr`, `au`, `it`, `es`, or `ca`

💻 Code Examples

# Submit Bulk Job (max 2,000 items, each with a country)
curl -X POST "https://api.magpieiq.com/scraping/v1/amazon/pdp/bulk/submit" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {"asin": "B018IZAME2", "country": "sg"},
      {"asin": "B01XYZ1234", "country": "us"},
      {"asin": "B08ABC5678", "country": "jp"}
    ]
  }'

# Retrieve Job Status (poll until status is "completed")
curl "https://api.magpieiq.com/scraping/v1/amazon/pdp/bulk/retrieve/{job_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

# Download the compiled JSONL result (download_url from retrieve response)
curl -L -o result.jsonl.gz ""

Python Example Click to expand

import requests, time

API_HOST = "https://api.magpieiq.com/scraping"
headers = {"Authorization": "Bearer YOUR_API_TOKEN", "Content-Type": "application/json"}

# 1. Submit bulk job
payload = {
    "items": [
        {"asin": "B018IZAME2", "country": "sg"},
        {"asin": "B01XYZ1234", "country": "us"},
    ]
}
resp = requests.post(f"{API_HOST}/v1/amazon/pdp/bulk/submit", json=payload, headers=headers)
job_id = resp.json()["job_id"]
print(f"Job submitted: {job_id}")

# 2. Poll until complete
while True:
    result = requests.get(f"{API_HOST}/v1/amazon/pdp/bulk/retrieve/{job_id}", headers=headers).json()
    print(f"Status: {result['status']} ({result['success']}/{result['total']} done)")
    if result["status"] in ["completed", "partial_complete", "failed"]:
        break
    time.sleep(30)

# 3. Download the compiled JSONL result
if result.get("download_url"):
    data = requests.get(result["download_url"]).content
    print(f"Downloaded {len(data)} bytes (gzip JSONL)")

⚡ Behavior

Cost: billed per item — see the rate card (beta pricing)
Approval: NOT required - immediate processing
Batch: Up to 2,000 items per request
Per-item country: Each item must specify a country (sg, jp, us, uk, de, fr, au, it, es, ca).
Results: Single gzip-compressed JSONL file via download_url (signed GCS URL)
Partial results: If some items fail, credits are refunded for failed items only

Retrieve Response Example

Amazon PDP Bulk Retrieve JSON Click to expand

{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "completed",
  "total": 3,
  "success": 3,
  "failed": 0,
  "pending": 0,
  "progress": 1.0,
  "download_url": "https://storage.googleapis.com/magpie-scraping-inhouse/amazon/bulk_result/a1b2c3d4-e5f6-7890-abcd-ef1234567890/result.jsonl.gz?X-Goog-Algorithm=...",
  "message": null
}

JSONL Format

The download_url points to a gzip-compressed JSONL file. Each line is a JSON object:

Amazon PDP JSONL Line Example Click to expand

{"item_id": "B018IZAME2", "country": "sg", "response": { ... full Amazon PDP JSON ... }}

Result Data Example

The response field in each JSONL line contains the full Amazon PDP JSON:

Amazon PDP Result JSON Click to expand

[
  {
    "_scraped_at": "2026-07-23T12:00:00Z",
    "_asin": "B018IZAME2",
    "_country": "sg",
    "title": "Sample Amazon Product Title",
    "brand": "Sample Brand",
    "price": {
      "amount": 29.99,
      "currency": "SGD",
      "original_price": 39.99
    },
    "availability": "In Stock",
    "rating": 4.5,
    "review_count": 1234,
    "url": "https://www.amazon.sg/dp/B018IZAME2",
    "images": [
      "https://m.media-amazon.com/images/I/sample1.jpg",
      "https://m.media-amazon.com/images/I/sample2.jpg"
    ],
    "features": [
      "Feature 1",
      "Feature 2"
    ],
    "description": "Sample product description from Amazon",
    "specifications": [
      {"name": "Weight", "value": "0.5 kg"},
      {"name": "Dimensions", "value": "10 x 20 x 5 cm"}
    ],
    "variants": [
      {
        "asin": "B018IZAME3",
        "title": "Variant 1",
        "price": {"amount": 34.99, "currency": "SGD"}
      }
    ],
    "seller": {
      "name": "Sample Seller",
      "id": "A123456789"
    },
  …

Preview only — first 40 lines. Download the full sample JSON (1 KB)

12. Amazon Search Bulk Beta

Scrape Amazon search results and category listings in bulk. No Approval Required

Regions: SG, JP, US, UK, DE, FR, AU, IT, ES, CA (per-item country)

Submit a Batch

Submit up to 50 Amazon search/category tasks in one request. Billed per item, deducted upfront — see the rate card.

curl -X POST https://api.magpieiq.com/scraping/v1/amazon/search/bulk/submit \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {"list_type": "search", "keyword": "iphone 15", "country": "sg", "pages": 0},
      {"list_type": "search", "keyword": "samsung galaxy", "country": "us", "pages": 5},
      {"list_type": "category", "keyword": "6650160031", "category": "6650160031", "country": "jp", "pages": 0}
    ]
  }'

Request Parameters

Parameter	Type	Required	Description
`items`	array	Yes	Array of Amazon Search task objects (max 50)
`items[].list_type`	string	Yes	`search` or `category`
`items[].keyword`	string	Yes	Search keyword (for search) or category node ID (for category)
`items[].category`	string	No	Category node ID. Required when list_type is `category`
`items[].country`	string	Yes	Country code: `sg`, `jp`, `us`, `uk`, `de`, `fr`, `au`, `it`, `es`, or `ca`
`items[].pages`	integer	No	Number of pages to scrape. 0 = all discovered pages (default: 0)

Submit Response Example

Amazon Search Bulk Submit JSON Click to expand

{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "processing",
  "cost": 3,
  "total": 3,
  "items": [
    {"task_id": "abc123...", "keyword": "iphone 15", "list_type": "search", "status": "pending"},
    {"task_id": "def456...", "keyword": "samsung galaxy", "list_type": "search", "status": "pending"},
    {"task_id": "ghi789...", "keyword": "6650160031", "list_type": "category", "status": "pending"}
  ]
}

Retrieve Batch Status

Poll the retrieve endpoint to check batch progress and get per-item download URLs for successful results.

curl -X GET https://api.magpieiq.com/scraping/v1/amazon/search/bulk/retrieve/{job_id} \
  -H "Authorization: Bearer YOUR_API_KEY"

Retrieve Response Example

Amazon Search Bulk Retrieve JSON Click to expand

{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "completed",
  "total": 3,
  "success": 2,
  "failed": 1,
  "pending": 0,
  "items": [
    {"task_id": "abc123...", "keyword": "iphone_15_0", "list_type": "search", "status": "SUCCESS", "download_url": "https://storage.googleapis.com/..."},
    {"task_id": "def456...", "keyword": "samsung_galaxy_1", "list_type": "search", "status": "SUCCESS", "download_url": "https://storage.googleapis.com/..."},
    {"task_id": "ghi789...", "keyword": "6650160031_2", "list_type": "category", "status": "FAILED", "download_url": null}
  ]
}

Result Data Example

Each download_url points to a JSON file with the search results:

Amazon Search Result JSON Click to expand

{
  "batch_id": "a1b2c3d4e5f67890",
  "list_type": "search",
  "keyword": "iphone 15",
  "country": "sg",
  "total_products": 48,
  "products": [
    {
      "asin": "B0CHX2F5QT",
      "title": "Apple iPhone 15 (128 GB) - Pink",
      "price": "$1,099.00",
      "original_price": "$1,199.00",
      "currency": "SGD",
      "rating": "4.6",
      "review_count": "1,234",
      "thumbnail": "https://m.media-amazon.com/images/I/31example.jpg",
      "url": "/dp/B0CHX2F5QT",
      "is_prime": true,
      "is_sponsored": false
    },
    {
      "asin": "B0CHX2F5RV",
      "title": "Apple iPhone 15 (256 GB) - Blue",
      "price": "$1,249.00",
      "original_price": null,
      "currency": "SGD",
      "rating": "4.5",
      "review_count": "892",
      "thumbnail": "https://m.media-amazon.com/images/I/31example2.jpg",
      "url": "/dp/B0CHX2F5RV",
      "is_prime": true,
      "is_sponsored": false
    }
  ],
  "scraped_pages": 3,
  "submit_date": "2025-01-15",
  "submit_hour": "14"
}
  …

Preview only — first 40 lines. Download the full sample JSON (999 B)

13. Shopify Search Bulk Beta

Scrape Shopify collection and search pages in bulk. No Approval Required

Works on any Shopify store that exposes /products.json or /collections/{handle}/products.json

Submit a Batch

Submit up to 50 Shopify URLs in one request. 1 credit per item, deducted upfront.

curl -X POST https://api.magpieiq.com/scraping/v1/shopify/search/bulk/submit \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {"url": "https://www.allbirds.com/collections/all"},
      {"url": "https://www.kyliecosmetics.com/collections/all"}
    ]
  }'

Request Parameters

Parameter	Type	Required	Description
`items`	array	Yes	Array of Shopify URL objects (max 50)
`items[].url`	string	Yes	Full Shopify URL — a collection page, search page, or products.json URL

Submit Response Example

Shopify Search Bulk Submit JSON Click to expand

{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "processing",
  "cost": 2,
  "total": 2,
  "items": [
    {
      "task_id": "abc123...",
      "url": "https://www.allbirds.com/collections/all",
      "status": "pending"
    },
    {
      "task_id": "def456...",
      "url": "https://www.kyliecosmetics.com/collections/all",
      "status": "pending"
    }
  ]
}

Retrieve Batch Status

curl https://api.magpieiq.com/scraping/v1/shopify/search/bulk/retrieve/{job_id} \
  -H "Authorization: Bearer YOUR_API_KEY"

Retrieve Response Example

Shopify Search Bulk Retrieve JSON Click to expand

{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "completed",
  "total": 2,
  "success": 2,
  "failed": 0,
  "pending": 0,
  "items": [
    {
      "task_id": "abc123...",
      "url": "https://www.allbirds.com/collections/all",
      "status": "SUCCESS",
      "download_url": "https://storage.googleapis.com/magpie-scraping-inhouse/...signed-url..."
    },
    {
      "task_id": "def456...",
      "url": "https://www.kyliecosmetics.com/collections/all",
      "status": "SUCCESS",
      "download_url": "https://storage.googleapis.com/magpie-scraping-inhouse/...signed-url..."
    }
  ]
}

Result Data Example

Each download_url points to a JSON file with the scraped products:

Shopify Search Result JSON Click to expand

{
  "products": [
    {
      "id": 1234567890,
      "title": "Men's Wool Runner",
      "handle": "mens-wool-runner",
      "url": "https://www.allbirds.com/products/mens-wool-runner",
      "price": "110.00",
      "compare_at_price": null,
      "currency": "USD",
      "vendor": "Allbirds",
      "product_type": "Shoes",
      "tags": "mens, shoes, wool",
      "available": true,
      "inventory_quantity": 45,
      "featured_image": "https://cdn.shopify.com/s/files/1/0004/0809/5893/products/example.jpg",
      "images": [
        "https://cdn.shopify.com/s/files/1/0004/0809/5893/products/example.jpg"
      ],
      "options": [
        {"name": "Size", "values": ["8", "9", "10", "11", "12"]},
        {"name": "Color", "values": ["Natural Grey", "Black"]}
      ],
      "variants": [
        {"id": 9876543210, "title": "8 / Natural Grey", "price": "110.00", "available": true},
        {"id": 9876543211, "title": "9 / Natural Grey", "price": "110.00", "available": true}
      ]
    },
    {
      "id": 1234567891,
      "title": "Women's Tree Runner",
      "handle": "womens-tree-runner",
      "url": "https://www.allbirds.com/products/womens-tree-runner",
      "price": "105.00",
      "compare_at_price": "120.00",
      "currency": "USD",
      "vendor": "Allbirds",
      "product_type": "Shoes",
      "tags": "womens, shoes, tree",
      "available": true,
  …

Preview only — first 40 lines. Download the full sample JSON (2 KB)

14. Shopify PDP Bulk

Scrape Shopify product detail pages in bulk. No Approval Required

Works on any Shopify store product page (URL must contain /products/{handle})

Submit a Batch

Submit up to 2,000 Shopify product URLs in one request. 1 credit per item, deducted upfront.

curl -X POST https://api.magpieiq.com/scraping/v1/shopify/pdp/bulk/submit \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {"product_url": "https://www.allbirds.com/products/womens-tree-dasher-relay"},
      {"product_url": "https://www.allbirds.com/products/mens-tree-runners"}
    ]
  }'

Parameter	Type	Required	Description
`items`	array	Yes	Array of product URL objects (max 2,000)
`items[].product_url`	string	Yes	Full Shopify product URL (must contain /products/{handle})

Submit Response Example

Shopify PDP Bulk Submit JSON Click to expand

{
  "job_id": "a1b2c3d4-1234-5678-90ab-cdef12345678",
  "status": "processing",
  "cost": 2,
  "total": 2,
  "items": [
    {
      "task_id": "abc123...",
      "product_url": "https://www.allbirds.com/products/womens-tree-dasher-relay",
      "status": "pending"
    },
    {
      "task_id": "def456...",
      "product_url": "https://www.allbirds.com/products/mens-tree-runners",
      "status": "pending"
    }
  ]
}

Retrieve Batch Status

curl https://api.magpieiq.com/scraping/v1/shopify/pdp/bulk/retrieve/{job_id} \
  -H "Authorization: Bearer YOUR_API_KEY"

Retrieve Response Example

Shopify PDP Bulk Retrieve JSON Click to expand

{
  "job_id": "a1b2c3d4-1234-5678-90ab-cdef12345678",
  "status": "completed",
  "total": 2,
  "success": 2,
  "failed": 0,
  "pending": 0,
  "progress": 1.0,
  "download_url": "https://storage.googleapis.com/magpie-scraping-inhouse/...",
  "message": null
}

Result JSON

The download_url points to a gzip-compressed JSONL file. Each line contains a product result:

Shopify PDP Result JSON Click to expand

{
  "product_url": "https://www.allbirds.com/products/womens-tree-dasher-relay",
  "handle": "womens-tree-dasher-relay",
  "domain": "www.allbirds.com",
  "scraped_at": "2026-07-29T07:00:00+00:00",
  "source": "js",
  "fetch_status": "ok",
  "data": {
    "id": 1234567890,
    "title": "Women's Tree Dasher Relay",
    "handle": "womens-tree-dasher-relay",
    "vendor": "Allbirds",
    "product_type": "Shoes",
    "tags": ["running", "sustainable", "eucalyptus"],
    "variants": [
      {
        "id": 9876543210,
        "title": "8 / Natural White",
        "price": "135.00",
        "sku": "TDR-W-8-NW",
        "available": true
      }
    ],
    "images": [
      {
        "id": 1111111111,
        "src": "https://cdn.shopify.com/s/files/1/0428/1234/products/tdr-nw-front.jpg"
      }
    ],
    "options": [
      {"name": "Size", "values": ["7", "8", "9", "10", "11"]},
      {"name": "Color", "values": ["Natural White", "Black", "Navy"]}
    ]
  }
}
  …

Preview only — first 40 lines. Download the full sample JSON (968 B)

📋 Job Status Reference

Job Statuses:

AWAITING_APPROVAL - Pending team approval
PROCESSING - Approved, scraping in progress
COMPLETED - All items successful
PARTIAL_COMPLETE - Some items failed
TERMINATED - Bulk task stopped; completed results are ready
FAILED - All items failed
REJECTED - Rejected or timed out

HTTP Status Codes:

200 - Success
202 - Job accepted
401 - Invalid/missing token
402 - Insufficient credits
404 - Job not found
422 - Validation error

API Integration Guide

🔐 Authentication

⏳ Approval Flow

1. Variant Sold

🔑 Endpoint

📥 Request Body

💻 Code Examples

Behavior

Result Example

2. Shopee Product Page

🔑 Endpoint

📥 Request Body

💻 Code Examples

📦 Bulk by CSV (get_pc_bulk)

⚡ Behavior

Result Example

3. Search / Category Items

🔑 Endpoint

📥 Request Body

🔗 Supported URL Formats

💻 Code Examples

⚡ Behavior

Result Example

4. Merchant Items

🔑 Endpoint

📥 Request Body

🔗 Supported URL Formats

💻 Code Examples

⚡ Behavior

Result Example

5. Tokopedia PDP Bulk Beta

🔑 Endpoint

📥 Request Body

💻 Code Examples

⚡ Behavior

Retrieve Response Example

Result Data Example

6. Blibli PDP Bulk Beta

🔑 Endpoint

📥 Request Body

💻 Code Examples

⚡ Behavior

Retrieve Response Example

JSONL Format

Result Data Example

7. Tokopedia Search Bulk Beta

🔑 Endpoint

📥 Request Body

💻 Code Examples

⚡ Behavior

Retrieve Response Example

Result Data Example

8. Tokopedia Merchant Bulk Beta

🔑 Endpoint

📥 Request Body

💻 Code Examples

⚡ Behavior

Retrieve Response Example

Result Data Example

9. Blibli Merchant Bulk Beta

🔑 Endpoint

📥 Request Body

💻 Code Examples

⚡ Behavior

Retrieve Response Example

Result Data Example

10. TikTok PDP Bulk Beta

🔑 Endpoint

📥 Request Body

💻 Code Examples

⚡ Behavior

Submit Response Example

Retrieve Response Example

JSONL Format

Result Data Example

11. Amazon PDP Bulk Beta

🔑 Endpoint

📥 Request Body

💻 Code Examples

⚡ Behavior