wehub-resource-sync
147 lines
4.6 KiB
Plaintext
147 lines
4.6 KiB
Plaintext
---
|
|
id: experiments-request-locking
|
|
title: Request Locking
|
|
description: Parallelize crawlers with ease using request locking
|
|
---
|
|
|
|
import ApiLink from '@site/src/components/ApiLink';
|
|
|
|
:::tip Release announcement
|
|
|
|
As of **May 2024** (`crawlee` version `3.10.0`), this experiment is now enabled by default! With that said, if you encounter issues you can:
|
|
|
|
- set `requestLocking` to `false` in the `experiments` object of your crawler options
|
|
- update all imports of `RequestQueue` to `RequestQueueV1`
|
|
- open an issue on our [GitHub repository](https://github.com/apify/crawlee)
|
|
|
|
The content below is kept for documentation purposes.
|
|
If you're interested in the changes, you can read the [blog post about the new Request Queue storage system on the Apify blog](https://blog.apify.com/new-apify-request-queue/).
|
|
|
|
:::
|
|
|
|
---
|
|
|
|
:::caution
|
|
|
|
This is an experimental feature. While we welcome testers, keep in mind that it is currently not recommended to use this in production.
|
|
|
|
The API is subject to change, and we might introduce breaking changes in the future.
|
|
|
|
Should you be using this, feel free to open issues on our [GitHub repository](https://github.com/apify/crawlee), and we'll take a look.
|
|
|
|
:::
|
|
|
|
Starting with `crawlee` version `3.5.5`, we have introduced a new crawler option that lets you enable using a new request locking
|
|
API. With this API, you will be able to pass a `RequestQueue` to multiple crawlers to parallelize the crawling process.
|
|
|
|
:::info Keep in mind
|
|
|
|
The request queue that supports request locking is currently exported via the `RequestQueueV2` class. Once the experiment is over, this class will replace
|
|
the current `RequestQueue` class
|
|
|
|
:::
|
|
|
|
## How to enable the experiment
|
|
|
|
### In crawlers
|
|
|
|
:::note
|
|
|
|
This example shows how to enable the experiment in the <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>,
|
|
but you can apply this to any crawler type.
|
|
|
|
:::
|
|
|
|
```ts
|
|
import { CheerioCrawler } from 'crawlee';
|
|
|
|
const crawler = new CheerioCrawler({
|
|
// highlight-next-line
|
|
experiments: {
|
|
// highlight-next-line
|
|
requestLocking: true,
|
|
// highlight-next-line
|
|
},
|
|
async requestHandler({ $, request }) {
|
|
const title = $('title').text();
|
|
console.log(`The title of "${request.url}" is: ${title}.`);
|
|
},
|
|
});
|
|
|
|
await crawler.run(['https://crawlee.dev']);
|
|
```
|
|
|
|
### Outside crawlers (to setup your own request queue that supports locking)
|
|
|
|
Previously, you would import `RequestQueue` from `crawlee`. To switch to the queue that supports locking, you need to import `RequestQueueV2` instead.
|
|
|
|
```ts
|
|
// highlight-next-line
|
|
import { RequestQueueV2 } from 'crawlee';
|
|
|
|
const queue = await RequestQueueV2.open('my-locking-queue');
|
|
await queue.addRequests([
|
|
{ url: 'https://crawlee.dev' },
|
|
{ url: 'https://crawlee.dev/js/docs' },
|
|
{ url: 'https://crawlee.dev/js/api' },
|
|
]);
|
|
```
|
|
|
|
### Using the new request queue in crawlers
|
|
|
|
If you make your own request queue that supports locking, you will also need to enable the experiment in your crawlers.
|
|
|
|
:::danger
|
|
|
|
If you do not enable the experiment, you will receive a runtime error and the crawler will not start.
|
|
|
|
:::
|
|
|
|
```ts
|
|
import { CheerioCrawler, RequestQueueV2 } from 'crawlee';
|
|
|
|
// highlight-next-line
|
|
const queue = await RequestQueueV2.open('my-locking-queue');
|
|
|
|
const crawler = new CheerioCrawler({
|
|
// highlight-next-line
|
|
experiments: {
|
|
// highlight-next-line
|
|
requestLocking: true,
|
|
// highlight-next-line
|
|
},
|
|
// highlight-next-line
|
|
requestQueue: queue,
|
|
async requestHandler({ $, request }) {
|
|
const title = $('title').text();
|
|
console.log(`The title of "${request.url}" is: ${title}.`);
|
|
},
|
|
});
|
|
|
|
await crawler.run();
|
|
```
|
|
|
|
## Other changes
|
|
|
|
:::info
|
|
|
|
This section is only useful if you're a tinkerer and want to see what's going on under the hood.
|
|
|
|
:::
|
|
|
|
In order to facilitate the new request locking API, as well as keep both the current request queue logic and the new, locking based request queue
|
|
logic, we have implemented a common starting point called `RequestProvider`.
|
|
|
|
This class implements almost all functions by default, but expects you, the developer, to implement the following methods:
|
|
`fetchNextRequest` and `ensureHeadIsNotEmpty`. These methods are responsible for loading and returning requests to process,
|
|
and tell crawlers if there are more requests to process.
|
|
|
|
You can use this base class to implement your own request providers if you need to fetch requests from a different source.
|
|
|
|
:::tip
|
|
|
|
We recommend you use TypeScript when implementing your own request provider, as it comes with suggestions for the abstract methods, as well as
|
|
giving you the exact types you need to return.
|
|
|
|
:::
|