Web Scraping with Crawlee: A Comprehensive Guide Using ProxyTee

If you're involved in web scraping, ProxyTee can be an essential tool for efficient data collection. As a provider of rotating residential proxies, ProxyTee supports various internet activities, such as web scraping, streaming, and tasks that require anonymity and IP rotation. Known for its affordability and reliability, ProxyTee provides solutions with unlimited bandwidth, a wide pool of IP addresses, and easy-to-use tools.

In this guide, we will explore how to leverage the Crawlee library for web scraping, enhanced by ProxyTee's robust infrastructure. You’ll begin with a basic web scraping example and gradually delve into more advanced topics such as session management and handling dynamic pages.

Getting Started with Web Scraping and ProxyTee

Before we dive into the code, ensure you have the following prerequisites set up on your machine:

• Node.js: Make sure you have Node.js installed.
• npm: This comes with Node.js. Verify by running node -v or npm -v.
• Code Editor: Use your preferred code editor; this example uses Visual Studio Code.

Basic Web Scraping with Crawlee

Let’s begin by scraping the Books to Scrape website, ideal for beginners with its straightforward HTML structure.

First, initialize a Node.js project by executing the following commands in your terminal:

mkdir crawlee-tutorial
cd crawlee-tutorial
npm init -y

Next, install the Crawlee library with:

npm install crawlee

Before scraping, inspect the page you want to scrape to identify the necessary HTML tags. Open the website in your browser and go to Developer Tools by right-clicking anywhere on the page and selecting "Inspect" or "Inspect Element".

The "Elements" tab shows the HTML structure. In this instance, every book is within an article tag with the class product_pod. Inside each article, the title is in an h3 tag, the book title in the title attribute of an a tag within the h3, and the price is in a p tag with the class price_color.

Now, create a file named scrape.js in your root directory and insert the following:

const { CheerioCrawler } = require('crawlee');

const crawler = new CheerioCrawler({
    async requestHandler({ request, $ }) {
        const books = [];
        $('article.product_pod').each((index, element) => {
            const title = $(element).find('h3 a').attr('title');
            const price = $(element).find('.price_color').text();
            books.push({ title, price });
        });
        console.log(books);
    },
});

crawler.run(['https://books.toscrape.com/']);

This code uses CheerioCrawler to fetch HTML, extract data from <article class="product_pod"> using a syntax similar to jQuery, and output the data to the console. You can run the code with the following command:

node scrape.js

You should see an array of book titles and prices displayed in your console.

Enhancing Scraping with ProxyTee

A proxy acts as an intermediary between your computer and the internet. When you utilize a proxy, all your web requests pass through a proxy server, masking your actual IP address. This can prevent rate-limiting and IP bans, particularly with web scraping. ProxyTee makes integrating proxies simple with features like built-in proxy handling that effectively manages errors and retries.

ProxyTee's infrastructure offers several key features:

• Unlimited Bandwidth: ProxyTee provides proxies with unlimited bandwidth, which means you don't have to worry about data limits, making it perfect for large scraping jobs. (Unlimited Bandwidth)
• Global IP Coverage: With over 30 million IP addresses from more than 100 countries, ProxyTee lets you access geographically restricted content, enabling more targeted data collection.(Global Coverage)
• Multiple Protocol Support:ProxyTee is compatible with both HTTP and SOCKS5 protocols, ensuring compatibility with all your scraping tools and applications. (Multiple Proxy Protocols)
• Auto-Rotation: The auto-rotation feature changes your IP address every 3 to 60 minutes, reducing the risk of IP bans. This is configurable to your requirements.(Auto Rotation)
• Simple API: ProxyTee has a simple API, allowing seamless integration with diverse applications and automation of your proxy tasks. (Simple API)
• User-Friendly Interface: With a simple, intuitive GUI, ProxyTee ensures users can start quickly and efficiently with no technical skills needed.(Simple & Clean GUI)

Setting Up ProxyTee

To leverage ProxyTee for your scraping tasks, you will need to set up a proxy. Here's how to get started:

1. Navigate to the ProxyTee website and create a free account.
2. Access the proxy section to set up a new Residential Proxy. Residential Proxies is the flagship product. We also offer Unlimited Residential Proxies, Static Residential Proxies, Mobile Proxies and Datacenter Proxies for different needs.
3. Note your proxy credentials: Host, Port, Username, and Password.

Next, install the axios library by running the following command in your terminal:

npm install axios

Create a new file called scrapeWithProxy.js in the root directory of your project, and copy the code below. Remember to replace HOST, PORT, USERNAME, and pass with your actual ProxyTee credentials.

const { CheerioCrawler } = require("crawlee");
const { ProxyConfiguration } = require("crawlee");
const axios = require("axios");

const proxyConfiguration = new ProxyConfiguration({
    proxyUrls: ["http://user:pass@HOST:PORT"],
});

const crawler = new CheerioCrawler({
    proxyConfiguration,
    async requestHandler({ request, $, response, proxies }) {
        // Get proxy info using lumtest.com
        try {
            const proxyInfo = await axios.get("http://lumtest.com/myip.json", {
                proxy: {
                    host: "HOST",
                    port: PORT,
                    auth: {
                        username: "USERNAME",
                        password: proxyPass,
                    },
                },
            });
            console.log("Proxy Information:", proxyInfo.data);
        } catch (error) {
            console.error("Error fetching proxy information:", error.message);
        }

        const books = [];
        $("article.product_pod").each((index, element) => {
            const title = $(element).find("h3 a").attr("title");
            const price = $(element).find(".price_color").text();
            books.push({ title, price });
        });
        console.log(books);
    },
});

crawler.run(["https://books.toscrape.com/"]);

This code initiates a CheerioCrawler to scrape data using the specified proxy. The proxy’s details are fetched from http://lumtest.com/myip.json to verify its use. Run this code with:

node scrapeWithProxy.js

You’ll see your results, now through a ProxyTee IP, along with details of the used IP address. When running this script again, a different IP address should be used, indicating the successful rotation of ProxyTee.

Session Management

Sessions are critical for maintaining continuity when you are scraping websites using cookies and login sessions. Crawlee's SessionPool effectively handles this.

Create scrapeWithSessions.js in the root and insert the code below:

const { CheerioCrawler, SessionPool } = require("crawlee");

(async () => {
    // Open a session pool
    const sessionPool = await SessionPool.open();

    // Ensure there is a session in the pool
    let session = await sessionPool.getSession();
    if (!session) {
        session = await sessionPool.createSession();
    }

    const crawler = new CheerioCrawler({
        useSessionPool: true, // Enable session pool
        async requestHandler({ request, $, response, session }) {
            // Log the session information
            console.log(`Using session: ${session.id}`);

            // Extract book data and log it (for demonstration)
            const books = [];
            $("article.product_pod").each((index, element) => {
                const title = $(element).find("h3 a").attr("title");
                const price = $(element).find(".price_color").text();
                books.push({ title, price });
            });
            console.log(books);
        },
    });

    // First run
    await crawler.run(["https://books.toscrape.com/"]);
    console.log("First run completed.");

    // Second run
    await crawler.run(["https://books.toscrape.com/"]);
    console.log("Second run completed.");
})();

The above script utilizes CheerioCrawler with a SessionPool. It initializes a session pool, then configures the crawler to use these sessions to log each session’s ID and extract book data for verification purposes. You can validate by running:

node scrapeWithSessions.js

When running the code multiple times you should see a unique session id on each run.

Dynamic Content Handling

Handling content loaded with JavaScript is tricky. Crawlee integrates with Puppeteer, enabling a browser environment for handling Javascript on websites.

For the example, let’s scrape YouTube comments. Start by ensuring that you adhere to their terms of service.

Create a file named scrapeDynamicContent.js in the root directory, and use the following script:

const { PuppeteerCrawler } = require("crawlee");

async function scrapeYouTube() {
    const crawler = new PuppeteerCrawler({
        async requestHandler({ page, request, enqueueLinks, log }) {
            const { url } = request;
            await page.goto(url, { waitUntil: "networkidle2" });

            // Scraping first 10 comments
            const comments = await page.evaluate(() => {
                return Array.from(document.querySelectorAll("#comments #content-text"))
                    .slice(0, 10)
                    .map((el) => el.innerText);
            });

            log.info(`Comments: ${comments.join("\n")}`);
        },

        launchContext: {
            launchOptions: {
                headless: true,
            },
        },
    });

    // Add the URL of the YouTube video you want to scrape
    await crawler.run(["https://www.youtube.com/watch?v=wZ6cST5pexo"]);
}

scrapeYouTube();

This script uses PuppeteerCrawler to navigate to a YouTube video URL and to wait until the page is loaded. Then, it scrapes the first ten comments from that page using JavaScript and outputs them in the console. Run by:

node scrapeDynamicContent.js

The output should now show the first ten comments on the provided YouTube link.

Conclusion

This tutorial has shown how Crawlee is a capable tool for web scraping, particularly with enhanced performance when integrated with the capabilities of ProxyTee, such as IP rotation, global IP coverage and its reliable network.

Always review the robots.txt file and the terms of service of the target website before scraping.

Ready to enhance your web scraping capabilities with professional tools? Visit ProxyTee to see how our affordable pricing plans can significantly improve your data collection needs, making large web scraping projects more manageable. ProxyTee ensures reliability, speed, and global access to streamline your data needs.

To scale this workflow in production without bandwidth throttling, explore our Automated Web Crawling Solutions, Playwright & Crawlee Integration, Australia Residential Proxies, and ProxyTee vs. Nimble (Nimbleway) Benchmark.