Google Chrome or Mozilla Firefox?

[Snegovik777]

Hi Uncleсode. Is it possible to use regular browsers instead of Chromium, Nightly or Webkit? For example Google Chrome or Mozilla Firefox?
I may be wrong, but some sites seem to use protection from browsers like Chromium...

For example, these settings.

# Initialize browser config
browser_config = BrowserConfig(
    browser_type="firefox",
    chrome_channel="firefox",
    headless=False,
    viewport_width=1280,
    viewport_height=1024,
    user_agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36",
)

Unfortunately, the site does not open... For Chromium and Webkit - the same...

[prxshetty]

You could try using https://github.com/browser-use/browser-use to scrape it. It supports all kinds of api's and browser and i think should be easy to plug this into a framework until @unclecode gets a fix for this.

[Snegovik777]

Hello @unclecode ! Maybe there is news on my question?

[unclecode]

@Snegovik777 Hi 👋 let's try one of your link and I help you fix it. Share a link you have problem with. @aravindkarnam

[Snegovik777]

@Snegovik777 Hi 👋 let's try one of your link and I help you fix it. Share a link you have problem with. @aravindkarnam

https://lenta.com/

There is my code.

[Snegovik777]

@aravindkarnam Hi!! Not all sites can be parsed. I know. Apparently mine is one of them 😇 No big deal. We'll look for other ways. Anyway, thanks for crawl4ai!!

[aravindkarnam]

@Snegovik777 Yes. I did try Crawl4AI with your site. It was throwing 403 with firefox, but worked fine with chrome. I forgot to update you here.

I think the backend was detecting as a bot and returning 403. I tried with different useragents as well, but no luck.

[rohitsathish]

@aravindkarnam could you provide the instructions to use chrome as the browser?