wehub-resource-sync
24 lines
1.1 KiB
Plaintext
24 lines
1.1 KiB
Plaintext
---
|
|
id: skip-navigation
|
|
title: Skipping navigations for certain requests
|
|
---
|
|
|
|
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
|
|
import ApiLink from '@site/src/components/ApiLink';
|
|
import SkipNavigationSource from '!!raw-loader!roa-loader!./skip-navigation.ts';
|
|
|
|
While crawling a website, you may encounter certain resources you'd like to save, but don't need the full power of a crawler to do so (like images delivered through a CDN).
|
|
|
|
By combining the <ApiLink to="core/class/Request#skipNavigation">`Request#skipNavigation`</ApiLink> option with <ApiLink to="basic-crawler/interface/BasicCrawlingContext#sendRequest">`sendRequest`</ApiLink>, we can fetch the image from the CDN, and save it to our key-value store without needing to use the full crawler.
|
|
|
|
|
|
:::info
|
|
|
|
For this example, we are using the <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> to showcase this, but this is available on all the crawlers we provide.
|
|
|
|
:::
|
|
|
|
<RunnableCodeBlock className="language-js">
|
|
{SkipNavigationSource}
|
|
</RunnableCodeBlock>
|