web-scraper

The web-scraper tool automates the process of extracting content from websites, enhancing the capability of security workflows to gather intelligence on web assets. It is designed to seamlessly integrate into automated security assessments, providing valuable data for further analysis.

Ideal Use Cases & Fit

The web-scraper shines in scenarios requiring the collection of website data for vulnerability assessments or competitive analysis. It excels when tasked with:

• Gathering structured data from multiple web pages
• Compiling information on HTML content, status codes, and responses
• Extracting complex, dynamically-loaded content that requires browser rendering
• Interacting with dynamic pages: filling forms, clicking buttons, and navigating JavaScript-heavy sites
• AI-driven browsing: performing complex, multi-step interactions using natural-language prompts

Best Practice: For basic HTML content extraction and initial reconnaissance, consider using the httpx node first, which provides efficient HTTP-based data collection. Reserve web-scraper for scenarios requiring advanced rendering capabilities or when httpx output is insufficient for your needs.

Value in Workflows

In security workflows, the web-scraper is invaluable during early reconnaissance phases, where it allows teams to quickly assess web presence and gather data for threat modeling. Its integration into workflows leads directly to informed decision-making based on real-time web data, improving the overall effectiveness of security assessments.

Input Data

The tool requires a file input specifying the target websites to scrape. This input must list the URLs to be processed. An example format could be as follows:

https://example.com
https://another-example.com

Configuration

• format: Specifies the output format of the scraper's results, with options including markdown, links, html, rawHtml, summary, and screenshot. Default is markdown.
• wait: Wait time in milliseconds to allow the target page to load fully before scraping. Aids in capturing dynamic content that may take longer to become available.
• timeout: HTTP request timeout in seconds (default: 60). Increase for interact/actions-heavy scrapes (recommended: 120–300).
• actions: JSON array of deterministic browser actions executed sequentially before scraping. See Actions Mode below.
• interact: JSON array of natural-language prompt strings executed sequentially in a stateful browser session. See Interact Mode below.

Actions Mode

The actions parameter accepts a JSON array of deterministic browser steps. Use this when you know the exact CSS selectors and sequence of interactions.

Available action types:

Type	Key Fields	Description
wait	milliseconds	Wait for a specified duration
wait	selector, milliseconds (optional)	Wait for a DOM element to appear
click	selector	Click a DOM element by CSS selector
write	text, selector (optional)	Type text into an input field
press	key	Press a keyboard key (e.g., Tab, Enter)
scroll	direction, amount	Scroll the page (up or down)
screenshot	fullPage (optional)	Capture a screenshot of the page
scrape	—	Trigger content extraction at the current page state
executeJavascript	script	Execute custom JavaScript on the page
generatePdf	—	Generate a PDF of the current page

Example — Form submission (CNMC portability lookup):

[
  {"type": "wait", "milliseconds": 2000},
  {"type": "write", "selector": "input[id='phone']", "text": "612345678"},
  {"type": "wait", "milliseconds": 1000},
  {"type": "click", "selector": "button[type='submit']"},
  {"type": "wait", "milliseconds": 5000},
  {"type": "scrape"}
]

Example — Cookie banner dismissal:

[
  {"type": "wait", "milliseconds": 1000},
  {"type": "click", "selector": "#cookie-accept"},
  {"type": "wait", "milliseconds": 2000},
  {"type": "scrape"}
]

Interact Mode

The interact parameter accepts a JSON array of natural-language prompt strings. Each prompt is executed sequentially in a persistent browser session with AI interpretation. Use this for complex flows, CAPTCHA handling, or when CSS selectors are unknown.

Example — AI-driven portability lookup:

[
  "Wait for the page to fully load",
  "Type 612345678 in the phone number input field",
  "Click the search button",
  "Wait for the results to appear",
  "Extract the operator name, phone number, and date as JSON"
]

Example — Multi-page product search:

[
  "Accept the cookie banner if present",
  "Search for 'wireless headphones' in the search bar",
  "Filter results to show only items under $200",
  "Extract the name and price of the top 5 results as JSON"
]

Tip: Prefer actions over interact when selectors are known — it is more deterministic and reliable. Use interact when the page structure is unpredictable or requires AI reasoning.

Updated: 2026-05-20