Reference
Crawler API
2min
A Crawler is a service that takes a source URL and the number of levels in depth to traverse and a script to extract data from a web page. Then the service builds a list of child links by traversing from the source URL to all the HREFs and retrieves each page and apply the script to the HTML document to generate JSON documents that can be injected into a search engine for full-text search.
- Asynchronous
- Headless browser (parses js enabled sites)
Updated 03 Mar 2023
Did this page help you?