Best Practices

In order to use the API you need to call an endpoint URL with your private access token. You can generate your call URL on our query builder (you must be logged in to use it).

Data Integrity

Each request may return up to 100 posts matching your query. However, there may be significantly more results matching your filter parameters. To consume all the data, make sure you keep on calling the same URL string value that appears in the "next" key found in the output for each request.

totalResults: 367865,
moreResultsAvailable: 367765,
next: "/reviewFilter?token=XXXX-XXXXXXXXX-XXXXXXXXXX-XXXXXXXXXXXX&format=json&ts=1498215346008&q=*&sort=crawled",
requestsLeft: 999889


Depending on the sort order you want, the "next" URL includes one of a two paging parameters:

  • Sort by crawl date (consume the data as a stream - default):
    The "next" URL contains the "ts" (timestamp) paging parameter which returns posts crawled after this timestamp.

  • Sort by Relevancy or by any other output sort field:
    The "next" URL contains the "from" paging parameter.

Important Note
Sorting by any sort parameter value other than crawl date (default) may result in missing important posts. If data integrity is important, stick with the default recommended sort parameter value of crawl date, and consume the data like a stream.

Updated 2 years ago

Best Practices

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.