user_agent_mode = "random" couldn't pass chewy's bot check

[xuchentan]

Hello here is my browser config and crawler config. Try my best to avoid bot check, but still couldn't open chewy's link through the crawler.

Browser config
`from crawl4ai import BrowserConfig

browser_config = BrowserConfig(
browser_type="chromium",
headless=False, # For visual debugging
viewport_width=1440,
viewport_height=900,
user_agent_mode = "random",#avoid bot check by randomize user agent can pass initial test
use_managed_browser=True, # Enables persistent browser sessions, avoid bot check for following test
user_data_dir="./chewy_user_profile" # Stores session & cookies

)crawler configfrom crawl4ai import CrawlerRunConfig, CacheMode

chewy_crawler_config = CrawlerRunConfig(
wait_until="networkidle",
wait_for="css:div.kib-product-card__content",
page_timeout=60000,
scan_full_page=True,
scroll_delay=0.5,
delay_before_return_html=0.6,
simulate_user=True,
override_navigator=True,
remove_overlay_elements=True,
adjust_viewport_to_content=True,
session_id="chewy_category_session",
stream=True,
verbose=True
)
`