desearch-crawlCrawl/scrape and extract content from any webpage URL. Returns the page content as clean text or raw HTML. Use this when you need to read the full contents o...
Install via ClawdBot CLI:
clawdbot install okradze/desearch-crawlGrade Fair — based on market validation, documentation quality, package completeness, maintenance status, and authenticity signals.
Calls external URL not in known-safe list
https://desearch.aiAudited Apr 17, 2026 · audit v1.0
Generated Mar 1, 2026
Businesses can use this skill to scrape competitor websites for product details, pricing, and content strategies. It helps in gathering data for benchmarking and identifying market trends efficiently.
Media companies can crawl news websites to aggregate articles, summaries, or raw HTML for content curation and analysis. This supports creating news feeds, monitoring coverage, and extracting insights from various sources.
Researchers and academics can extract text from online journals, databases, or educational websites to compile data for studies. It aids in literature reviews, data mining, and analyzing web-based information.
Digital marketing agencies can crawl client or competitor sites to analyze HTML structure, meta tags, and content for SEO optimization. This helps in identifying issues and improving search engine rankings.
Law firms or compliance teams can scrape regulatory websites or public records to track updates, extract legal texts, and ensure adherence to laws. It streamlines monitoring changes in policies or regulations.
Offer the crawling functionality as a paid API service, charging based on usage volume or subscription tiers. This model targets developers and businesses needing scalable web scraping without infrastructure management.
Collect and process web data using this skill, then sell aggregated datasets or analytical reports to clients. This adds value by transforming raw content into actionable business intelligence.
Incorporate the crawling skill into existing software platforms, such as CRM or marketing tools, as an add-on feature. This enhances product offerings and generates revenue through upselling or premium integrations.
💬 Integration Tip
Ensure the DESEARCH_API_KEY is securely stored as an environment variable to avoid exposure in code, and test with sample URLs to verify format outputs before full deployment.
Scored Apr 22, 2026
Summarize URLs or files with the summarize CLI (web, PDFs, images, audio, YouTube).
AI-optimized web search via Tavily API. Returns concise, relevant results for AI agents.
Multi search engine integration with 16 engines (7 CN + 9 Global). Supports advanced search operators, time filters, site search, privacy engines, and Wolfra...
Web search via Tavily API (alternative to Brave). Use when the user asks to search the web / look up sources / find links and Brave web_search is unavailable...
Web search and content extraction via Brave Search API. Use for searching documentation, facts, or any web content. Lightweight, no browser required.
Search the web using Baidu AI Search Engine (BDSE). Use for live information, documentation, or research topics.