OpenClaw CLI: The Ultimate Web Scraping & Data Automation Cheat Sheet

Initiate a scraping job on a specified URL, extracting data based on a CSS or XPath selector and saving it to a JSON file.

Use specific selectors for precise data targeting. Combine with `--ai-parse` for smart content recognition.

Perform advanced data extraction using an AI model to intelligently identify and structure data according to a predefined JSON schema.

Define a robust schema to guide the AI for accurate and consistent data output.

Fetch a webpage using custom proxy settings and a specific user agent to mimic different browser environments or bypass IP blocks.

Rotate proxies and user agents for large-scale scraping to avoid detection and rate limiting.

Open a URL in a headless browser, wait for a specific element to load, and then capture a full-page screenshot.

Crucial for scraping dynamic, JavaScript-rendered websites. Adjust wait times for complex pages.

Parse local HTML files using regular expressions to extract specific data fields without making a new web request.

Ideal for reprocessing previously saved HTML or for complex pattern matching on local content.

Transform JSON data using a JQ query to reshape, filter, or reformat the output structure.

Master JQ syntax for powerful and flexible data manipulation directly from the command line.

Clean a CSV file by removing duplicate rows based on a specified unique identifier field.

Apply cleaning steps early in your data pipeline to ensure data quality before further processing.

Standardize data in a specified field, for example, converting all currency values to a common format.

Use this command to ensure consistency across diverse datasets, improving analysis reliability.

Merge two JSON files based on a common key, combining their data into a single output file.

Ensure the 'on' key exists and is consistent across both files for successful merging.

Schedule a scraping job defined in a YAML configuration file to run at a specific time using cron syntax.

Use a dedicated configuration file for complex jobs to manage parameters and targets efficiently.

Set up continuous monitoring for changes on a webpage, triggering notifications if significant updates are detected.

Configure notification channels (email, webhook, Slack) in OpenClaw settings for seamless alerts.

Execute a predefined data pipeline, orchestrating multiple scraping, transformation, and storage steps.

Pipelines are essential for complex workflows, ensuring steps run in the correct sequence with error handling.

Export extracted data from a JSON file into various formats like CSV, specifying delimiters as needed.

OpenClaw supports multiple export formats, including XML and Excel, for versatile data integration.

Integrate OpenClaw with external services by sending extracted data to a specified webhook endpoint.

Automate data delivery to dashboards, databases, or other applications using custom webhooks.

Apply an AI model to classify text data within a specified field, such as sentiment analysis or topic categorization.

Leverage OpenClaw's built-in AI models for quick insights or integrate custom models for specialized tasks.

Generate a concise summary of long text content using a specified large language model, limiting the output tokens.

Useful for condensing lengthy articles or reviews into actionable summaries post-extraction.

Compare two HTML files to detect structural or content changes, reporting differences above a set threshold.

Ideal for monitoring competitor price changes or content updates on critical web pages.

Automatically infer a JSON schema from a sample HTML page, aiding in rapid schema development for extraction.

Start with an inferred schema and refine it for more precise and robust data extraction.

Perform a semantic search within a local JSON data corpus, returning the top K most relevant results based on meaning.

Pre-process your data corpus for optimal semantic search performance and relevance.

Set a global configuration parameter, for example, defining a default proxy server for all operations.

Manage frequently used settings centrally to streamline command execution and maintain consistency.

Retrieve the current value of a specific configuration parameter, such as the logging level.

Review configuration settings to troubleshoot behavior or confirm active parameters.

Create a named profile with predefined settings like headless mode and user agent for specific scraping scenarios.

Use profiles to quickly switch between different scraping configurations without retyping parameters.

Activate a previously created named profile, applying its settings to subsequent OpenClaw commands.

Simplify complex command lines by encapsulating common options within profiles.

Authenticate OpenClaw with external services or APIs using various authentication providers like API keys.

Securely manage your credentials using OpenClaw's built-in authentication system.

Run a scraping operation in debug mode, providing verbose output and logging all activities to a specified file.

Essential for diagnosing issues with selectors, network requests, or AI model behavior.

Execute a job configuration in dry-run mode to preview the actions OpenClaw would take without actually performing them.

Verify complex pipeline configurations and command sequences before live execution.

Validate a JSON data file against a specified JSON schema to ensure data integrity and conformity.

Integrate schema validation into your pipelines to catch data formatting errors early.

Inspect a specific HTML element on a live URL, revealing its attributes, text content, and computed styles.

Use this to quickly test and refine CSS or XPath selectors directly from the CLI.

Display the OpenClaw system logs in real-time, following new entries as they are generated.

Monitor background processes and scheduled jobs for status updates and error messages.