Bypass Bot Detection with Puppeteer Stealth: A ProxyTee Guide

Web scraping with Puppeteer is powerful, but bot detection can be a significant hurdle. Many websites use anti-bot measures to identify and block automated scripts, particularly when they detect headless browsers. This article explains how to use ProxyTee, specifically its Unlimited Residential Proxies combined with Puppeteer Stealth to evade detection. Let’s dive into how you can make your Puppeteer scripts virtually undetectable, ensuring smooth web data collection.

Understanding Bot Detection and Its Challenges for Puppeteer

Puppeteer, a popular JavaScript library for browser automation, lets you control browsers via the DevTools Protocol. This is fantastic for web scraping and automated testing. However, default settings in Puppeteer can unintentionally expose your automation as a bot. For example, the navigator.webdriver: true property is a telltale sign that an automated browser is being used.

Websites check for these kinds of browser configurations to block suspicious activity. By using default settings, you could trigger anti-bot systems that might flag and block your bot activity. This is particularly true for headless browsers, which is the case in the example below:

import puppeteer from "puppeteer"

(async () => {
    // set up the browser and launch it
    const browser = await puppeteer.launch()

    // open a new blank page
    const page = await browser.newPage()

    // navigate the page to the target page
    await page.goto("https://arh.antoinevastel.com/bots/areyouheadless")

    // extract the message of the test result
    const resultElement = await page.$("#res")
    const message = await resultElement.evaluate(e => e.textContent)

    // print the resulting message
    console.log(`The result of the test is \"%s\"`, message);

    // close the current browser session
    await browser.close()
})()

When run, this script would likely show “You are Chrome headless”, indicating it failed the bot detection test. However, with ProxyTee and a plugin called Puppeteer Stealth, you can hide these default indicators and bypass these issues.

Enter Puppeteer Extra and Stealth Plugin

To enhance Puppeteer’s capability to avoid detection, ProxyTee suggests using a plugin called Puppeteer Extra, which allows the implementation of multiple functionalities as plugins, notably the Stealth plugin. puppeteer-extra is a drop-in replacement for the puppeteer, and adds functionalities. The Stealth plugin modifies various settings to evade bot detection. With this powerful combo, the headless browser will be harder to be detected.

The Puppeteer Stealth plugin targets common bot-detection mechanisms by manipulating several aspects of the browser. It removes properties that betray it as a bot, such as setting navigator.webdriver to false, and removing “HeadlessChrome” from the User-Agent header.

It is very efficient at making a headless Chromium instance bypass common bot detection tests. The goal is to make the process as difficult as possible to detect your browser controlled via Puppeteer.

Steps to Use Puppeteer Stealth with ProxyTee

Now, let’s see how to integrate this in your code and boost it using ProxyTee residential proxies:

Step 1️⃣: Install Puppeteer Extra and Stealth Plugin

First, you will have to install Puppeteer Extra and the stealth plugin with the following command:

npm install puppeteer-extra puppeteer-extra-plugin-stealth

Step 2️⃣: Configure Puppeteer Extra and Stealth Plugin

In your javascript script, you need to modify the way you import the puppeteer library to use the Extra version:

import puppeteer from "puppeteer-extra";
import StealthPlugin from "puppeteer-extra-plugin-stealth";

And if you are using CommonJS, here’s what to use instead:

const puppeteer = require("puppeteer-extra");
const StealthPlugin = require("puppeteer-extra-plugin-stealth");

After the import section, register the stealth plugin like this:

puppeteer.use(StealthPlugin());

The Stealth plugin applies several methods to bypass detection, making the browser harder to detect as an automation tool.

Optionally, you could choose what strategies are enabled or disabled using this approach:

// enable only a few evasion techniques
puppeteer.use(StealthPlugin({
    enabledEvasions: new Set(["chrome.app", "chrome.csi", "defaultArgs", "navigator.plugins"])
}));

const stealthPlugin = StealthPlugin();
puppeteer.use(stealthPlugin);

// ...

// remove the "user-agent-override" evasion method
pluginStealth.enabledEvasions.delete("user-agent-override");

Step 3️⃣: Final Code

Below is the full script including the bot bypass using Puppeteer Stealth plugin:

import puppeteer from "puppeteer-extra";
import StealthPlugin from "puppeteer-extra-plugin-stealth";

(async () => {
    // configure the stealth plugin
    puppeteer.use(StealthPlugin());
    // set up the browser and launch it
    const browser = await puppeteer.launch();

    // open a new blank page
    const page = await browser.newPage();

    // navigate the page to the target page
    await page.goto("https://arh.antoinevastel.com/bots/areyouheadless");

    // extract the message of the test result
    const resultElement = await page.$("#res");
    const message = await resultElement.evaluate(e => e.textContent);

    // print the resulting message
    console.log(`The result of the test is \"%s\"`, message);

    // close the current browser session
    await browser.close();
})();

Running the above script should return “You are not Chrome headless”, demonstrating that your browser is no longer detected by basic detection techniques.

Maximize Stealth and Success with ProxyTee’s Advanced Proxy Power

Bot detection is challenging, and avoiding it entirely is tough. However, using ProxyTee with Puppeteer Extra and the Stealth plugin provides an effective way to enhance your browser anonymity and bypass these checks. By implementing the plugin’s set of evasion configurations, your headless browser should appear much more like a real user.

However, it’s good to know that more sophisticated detection systems like Cloudflare will likely be able to detect your browser. In that case, using a more advanced system of proxies like ProxyTee’s Unlimited Residential Proxies is recommended.

Why ProxyTee? Unlike competitors, ProxyTee is able to offer unlimited bandwidth, rotating residential IPs, geo-targeting, and very competitive prices. Get started today and experience reliable scraping results!