-
Notifications
You must be signed in to change notification settings - Fork 136
Description
Problem Statement
Currently, Strands Agents have limited options when it comes to real-time access to external web data from dynamic, public websites-especially those with bot protection, CAPTCHA, or anti-scraping mechanisms.
This makes it difficult for agents to gather structured content from the live web at scale or with reliability.
Proposed Solution
The proposed solution is a new tool integration based on Bright Data’s Web Unlocker ,SERP API and Data Feeds, which allows Strands agents to:
Perform search engine queries and retrieve results in structured format.
Crawl any public webpage and extract content as Markdown or raw HTML, even if protected by CAPTCHA or bot detection.
The tool has been fully implemented and tested, and it follows the structure and conventions used in other tools within this repo.
The corresponding PR is ready for review.
Use Case
This tool would enable agents to:
-
Search Google/Bing/other engines and fetch live results.
-
Access and summarize real-time news or product data from protected websites.
-
Scrape content for research, market monitoring, or price comparison use cases.
-
Avoid static data sources and rely on fresh, up-to-date public web content.
Alternatives Solutions
No response
Additional Context
The tool is fully implemented in this PR: #21
The design was based on existing tools in this repository to ensure consistency.
Looking forward to your feedback and happy to make adjustments if needed.
Thanks!
Meir