Web Scraping with Cheerio and Node.js: A Practical ProxyTee Guide

Web scraping is an invaluable technique for gathering data from the internet, with applications ranging from academic research to market analysis and content aggregation. This guide will demonstrate how to set up a web scraping project using Cheerio and Axios in Node.js, and we’ll highlight how ProxyTee can enhance your scraping efforts.

Cheerio, a powerful HTML parsing library, combined with Axios, a versatile HTTP client, makes web scraping simpler. Cheerio’s jQuery-like API offers intuitive DOM manipulation, while Axios efficiently handles HTTP requests, including automatic JSON transformation and promise-based asynchronous operations. ProxyTee's rotating residential proxies can further enhance this setup by providing a vast pool of IPs and ensuring that your requests look like they are coming from legitimate users, reducing the risk of being blocked.

In this tutorial, you'll learn how to use Axios to make HTTP requests, parse HTML data with Cheerio, and save extracted data in CSV or JSON. We'll also explore how to use caching for improved performance, particularly useful when paired with ProxyTee services.

You Will Learn:

• Understanding web scraping with Cheerio
• Setting up a Node.js project for web scraping
• Making HTTP requests using Axios
• Extracting and manipulating HTML with Cheerio
• Saving scraped data in CSV and JSON formats
• Using caching to improve scraping performance

What is Cheerio?

Cheerio is a fast and flexible JavaScript library, built on htmlparser2. It provides a jQuery-like API for manipulating DOM elements, making it perfect for server-side web scraping. Cheerio is efficient and straightforward, with robust APIs for data extraction and HTML parsing, a perfect complement to ProxyTee's infrastructure.

Key uses of Cheerio include:

• Traversing and parsing HTML documents
• Extracting content-specific elements
• Manipulating DOM elements on the server side

What is Axios?

Axios is a promise-based HTTP client for Node.js and browsers. It simplifies making HTTP requests in JavaScript, making it easy to fetch web pages with XMLHttpRequest. Axios supports modern browsers and offers additional features for handling responses and processing requests. Paired with ProxyTee's rotating IPs, your scraping tasks become seamless.

Axios is commonly used for:

• Making GET requests
• Handling responses and errors
• Intercepting requests and responses
• Automatic JSON transformations

Prerequisites

Before you begin, it's crucial to respect web scraping policies and remain ethical. Always adhere to a website's terms of service.

To follow this tutorial, you’ll need:

• Basic knowledge of Node.js and JavaScript.
• Node.js (v14+) and npm installed on your system. Verify using:
  • `node -v`
  • `npm -v`

• An IDE or text editor such as VS Code or Atom.
• Familiarity with basic CLI commands.

Step-by-Step Tutorial

Let's get started.

Step 1. Environment Setup

Create a Node.js Project:

Open your terminal and use the following commands to create a new directory, navigate into it, and initialize a Node.js project:

mkdir web-scraper-cheerio
cd web-scraper-cheerio
npm init -y

Install Required Packages

Use the following command to install Cheerio and Axios:

npm install cheerio axios

These libraries will be essential for parsing HTML and making requests. Consider pairing it with ProxyTee’s Unlimited Residential Proxies for reliable and efficient data collection.

Step 2. Code in Action

Create a file named `index.js` in your project directory.

Import Required Modules:

const axios = require('axios');
const cheerio = require('cheerio');
const fs = require('fs');
const path = require('path');

Here, the file system (fs) module provides file manipulation functionalities, while the path module handles file paths consistently across different operating systems.

Fetch HTML Content using Axios:

async function fetchHTML(url) {
  try {
    const { data } = await axios.get(url);
    console.log(`Fetched from network: ${url}`);
    return data;
  } catch (error) {
    console.error('Error fetching the HTML:', error);
    throw error;
  }
}

This `fetchHTML()` function makes an HTTP GET request to the given URL. The use of `async` ensures efficient handling of asynchronous operations. Remember that pairing with ProxyTee’s residential proxies can reduce the risk of bans due to repeated requests.

Extract Data using Cheerio:

function extractProducts(html) {
  const $ = cheerio.load(html);
  const products = [];

  $('.product-card').each((index, element) => {
    const name = $(element).find('.title').text().trim();
    const price = $(element).find('.price-wrapper').text().trim();
    products.push({ name, price });
  });

  return products;
}

The `extractProducts()` function utilizes Cheerio’s `load()` to parse HTML and extract data based on CSS selectors. The `products` array will hold the titles and prices for each product.

Save the Data to CSV and JSON Files:

const jsonFilePath = path.join(__dirname, 'products.json');
const csvFilePath = path.join(__dirname, 'products.csv');

function saveToJSON(data) {
  fs.writeFileSync(jsonFilePath, JSON.stringify(data, null, 2));
  console.log(`Data saved to products.json`);
}

function saveToCSV(data) {
  const csvHeader = 'Name,Price\n';
  const csvRows = data.map(product => `${product.name.replace(/,/g, '')},${product.price}`).join('\n');
  fs.writeFileSync(csvFilePath, csvHeader + csvRows);
  console.log(`Data saved to products.csv`);
}

These functions save the scraped data to JSON and CSV files, facilitating analysis. If you need high volumes of data quickly and reliably, consider pairing this with ProxyTee's unlimited bandwidth offering.

Implementing Caching:

const cache = {};

async function fetchHTML(url) {
  if (cache[url]) {
    console.log(`Fetching from cache: ${url}`);
    return cache[url];
  }

  try {
    const { data } = await axios.get(url);
    cache[url] = data;
    console.log(`Fetched from network: ${url}`);
    return data;
  } catch (error) {
    console.error('Error fetching the HTML:', error);
    throw error;
  }
}

A basic caching mechanism using the `cache` object stores and reuses previously retrieved HTML content. This will greatly improve performance and minimize network requests. By integrating ProxyTee into this process, you will also reduce server load as your requests are efficiently distributed over a wide IP pool.

Handling Pagination:

async function scrapeProducts(url, allProducts = []) {
  try {
    const html = await fetchHTML(url);
    const products = extractProducts(html);
    allProducts = allProducts.concat(products);

    saveToJSON(allProducts);
    saveToCSV(allProducts);

    const $ = cheerio.load(html);
    const nextPage = $('.next > a').attr('href');

    if (nextPage && nextPage.indexOf('#') === -1) {
      const nextUrl = new URL(nextPage, url).href;
      await scrapeProducts(nextUrl, allProducts);
    }
  } catch (error) {
    console.error('Failed to scrape the website:', error);
  }
}

The `scrapeProducts()` function is used to recursively scrape multiple pages. It uses `fetchHTML()` and `extractProducts()`, and checks for a next button to traverse through pages, efficiently scraping multi-page content.

Step 3. Compile and Run the Code:

The complete `index.js` file is:

const axios = require('axios');
const cheerio = require('cheerio');
const fs = require('fs');
const path = require('path');

const jsonFilePath = path.join(__dirname, 'products.json');
const csvFilePath = path.join(__dirname, 'products.csv');

const cache = {};

async function fetchHTML(url) {
  if (cache[url]) {
    console.log(`Fetching from cache: ${url}`);
    return cache[url];
  }

  try {
    const { data } = await axios.get(url);
    cache[url] = data;
    console.log(`Fetched from network: ${url}`);
    return data;
  } catch (error) {
    console.error('Error fetching the HTML:', error);
    throw error;
  }
}

function extractProducts(html) {
  const $ = cheerio.load(html);
  const products = [];

  $('.product-card').each((index, element) => {
    const name = $(element).find('.title').text().trim();
    const price = $(element).find('.price-wrapper').text().trim();
    products.push({ name, price });
  });

  return products;
}

function saveToJSON(data) {
  fs.writeFileSync(jsonFilePath, JSON.stringify(data, null, 2));
  console.log(`Data saved to products.json`);
}

function saveToCSV(data) {
  const csvHeader = 'Name,Price\n';
  const csvRows = data.map(product => `${product.name.replace(/,/g, '')},${product.price}`).join('\n');
  fs.writeFileSync(csvFilePath, csvHeader + csvRows);
  console.log(`Data saved to products.csv`);
}

async function scrapeProducts(url, allProducts = []) {
  try {
    const html = await fetchHTML(url);
    const products = extractProducts(html);
    allProducts = allProducts.concat(products);

    saveToJSON(allProducts);
    saveToCSV(allProducts);

    const $ = cheerio.load(html);
    const nextPage = $('.next > a').attr('href');

    if (nextPage && nextPage.indexOf('#') === -1) {
      const nextUrl = new URL(nextPage, url).href;
      await scrapeProducts(nextUrl, allProducts);
    }
  } catch (error) {
    console.error('Failed to scrape the website:', error);
  }
}

(async () => {
  const baseURL = 'https://sandbox.oxylabs.io/products';
  await scrapeProducts(baseURL);
})();

Run the code using:

node index.js

Output

Check the console for progress updates and your CSV and JSON files.

Conclusion

This guide demonstrates the process of using Cheerio and Axios for web scraping in Node.js. You have now learned to set up a Node.js project, fetch HTML content, extract and manipulate data, save data to CSV/JSON, and use caching to improve scraper performance.

By combining this knowledge with a provider like ProxyTee, you can significantly enhance the efficiency and effectiveness of your web scraping projects. Whether you need Residential Proxies for improved anonymity or Unlimited Bandwidth for large tasks, ProxyTee has tools to assist your web scraping needs. Begin with your current skills, experiment with new websites, improve techniques and take on further challenges such as handling dynamic web pages or bypassing anti-scraping technologies, all while integrating ProxyTee's features to create a solid scraping strategy.

ProxyTee offers affordable and user-friendly Unlimited Residential Proxies with features including:

• Unlimited Bandwidth: Ensures that you are not limited by data restrictions.
• Global IP Coverage: Offers over 30 million IPs across 100+ countries,
• Multiple Protocol Support: Supports HTTP and SOCKS5 protocols for flexibility with tools and applications.
• Simple & Clean GUI: User-friendly platform with minimal setup needed.
• Auto Rotation: Provides IP address changes from 3 to 60-minute intervals for enhanced anonymity, crucial for web scraping and offered at ProxyTee.
• API Integration: Seamless API integration that works with all services making automation tasks easier.

To scale this workflow in production without bandwidth throttling, explore our Shopify & Storefront Monitoring Proxies, Node.js & Puppeteer Proxy Integration, Singapore Residential Proxies, and ProxyTee vs. MarsProxies Benchmark.