A Crawler is a service that takes a source URL and the number of levels in depth to traverse and a script to extract data from a web page. Then the service builds a list of child links by traversing from the source URL to all the HREFs and retrieves each page and apply the script to the HTML document to generate JSON documents that can be injected into a search engine for full-text search.

Crawler API

Create/Update a Job

hyper

Introduction 

Getting started

Create a Database

Delete a Database

List Databases

Run a Query

Create an Index

List Documents

Retrieve a Document By Id

Create a Document

Update a Document

Delete a Document

Create, Update or Delete multiple documents

Query Selector

Data API 

Create Cache Store

Delete Cache Store

Query Cache Store

Cache Value with Key

Get Cache Value by Key

Update Value by Key

Delete Key/Value Pair

Adapters

Cache API 

Create Search Index

Delete Search Index

Search Index

Add document to search index

Add Documents to Search Index

Get Search Document

Remove Search Document

Search Adapters

Search API 

Create a Bucket

Delete a Bucket

Upload an Object

Get Object

Update Object

Delete Object

Storage API

Create a queue

Post a Job/Task

Get Jobs from queue

Delete a queue

Queue API

Run a Job

Get a Job Document

Delete a Job

List Crawler Jobs

Reference

Authentication 

Configuration

Building your own adapter

Code of Conduct 

hyper RFCs

Style Guide

Bugs

Before Submitting A Bug Report

Building your own plugin

Contributing to hyper

Build your own client 🛠️

hyper connect

NodeJS Client 🚀

Deno Client 🦕

API Clients 

Middleware 

Hooks  

Deploy hyper on Render

Deploy hyper on AWS

Deploy hyper on Digital Ocean

Deploy hyper on Google Cloud

Deploy hyper on Azure

Deploy

Changelog

Parameters

What is REST?

What is JSON?

Terminology

Glossary

Crawler API

Features

API Commands