> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gorbit.cloud/llms.txt
> Use this file to discover all available pages before exploring further.

# Web

> Index public or internal web pages

## How it works

The Web Connector scrapes sites based on a base URL.

* It only indexes files from the same domain and containing the same base path.
* It will index pages reachable via hyperlinks from the base URL.
* The text contents are cleaned up via some heuristics and some metadata such as the page Title
  is extracted.

## Setting up

### Authorization

* As long as the page is reachable, no additional authorization is necessary.

### Indexing

<Steps>
  <Step title="Open Web connector">
    Navigate to the Admin Panel and select the **Web** Connector.
  </Step>

  <Step title="Enter base URL and index">
    Input the base URL to index and click on Index.

    <img className="rounded-image" src="https://mintcdn.com/gorbit/klR7erNMN33AvYm0/assets/admins/connectors/web/WebConnector.png?fit=max&auto=format&n=klR7erNMN33AvYm0&q=85&s=5f07085ecb5dfd7a2434edfb559bcdb5" alt="Gorbit Web connector form to enter base URL for indexing" width="1895" height="1254" data-path="assets/admins/connectors/web/WebConnector.png" />
  </Step>
</Steps>

To see the status of the indexing, visit the Connectors Status page (top left).

<img className="rounded-image" src="https://mintcdn.com/gorbit/klR7erNMN33AvYm0/assets/admins/connectors/web/WebConnectorStatus.png?fit=max&auto=format&n=klR7erNMN33AvYm0&q=85&s=734f5dbf1727b835ecbc0a7b37cef67a" alt="Gorbit Connectors Status page showing web crawl progress" width="2491" height="989" data-path="assets/admins/connectors/web/WebConnectorStatus.png" />