Unverified Commit 69765f4d authored by Vitalie's avatar Vitalie Committed by GitHub

Extend bot list with googleweblight, Storebot-Google, Google-Inspecti… (#77728)

Co-authored-by: 's avatarJiachi Liu <inbox@huozhi.im>
parent 0329a7e2
import { HTML_LIMITED_BOT_UA_RE } from './html-bots'
// Bot crawler that will spin up a headless browser and execute JS
const HEADLESS_BROWSER_BOT_UA_RE =
/Googlebot|Google-PageRenderer|AdsBot-Google|googleweblight|Storebot-Google/i
// Bot crawler that will spin up a headless browser and execute JS.
// By default, only googlebots are considered as DOM bots. Blow is where the regex is computed from:
// x-ref: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
const HEADLESS_BROWSER_BOT_UA_RE = /google/i
export const HTML_LIMITED_BOT_UA_RE_STRING = HTML_LIMITED_BOT_UA_RE.source
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment