Mastering HTTP Requests in Node.js with Fetch API Using ProxyTee

The internet has evolved dramatically since the first website. Today, websites rely on numerous resources, and JavaScript plays a pivotal role in client-server communication. The Fetch API, now a standard in Node.js, offers a modern approach to making HTTP requests. In this post, we’ll explore how to use the Fetch API in Node.js effectively for sending HTTP requests, particularly for web scraping, and how it integrates with ProxyTee for enhanced performance and anonymity.

Understanding the Fetch API

The Fetch API is a powerful interface for fetching network resources. It simplifies making HTTP requests like GET and POST and uses modern standards like Promises, resulting in cleaner code. Native support for Fetch API is included in Node.js version 18 and above, eliminating the need for third-party libraries for basic HTTP requests. If you’re using Node.js below version 17.5, you can install ‘node-fetch’ to achieve the same functionality.

Basic Usage of Fetch API

Let’s dive into practical examples. We’ll use a demo site to fetch product data to understand the basics of sending HTTP requests using the Fetch API.

fetch('https://sandbox.oxylabs.io/products/1')
    .then(response => response.text())
    .then(body => {
        console.log(body);
    });

This code sends an HTTP GET request and prints the HTML content of the response. For those familiar with asynchronous operations, the ‘async-await’ syntax is also supported by fetch API, offering a more synchronous and intuitive syntax:

(async () => {
    const response = await fetch('https://sandbox.oxylabs.io/products/1');
    const body = await response.text();
    console.log(body);
})();

Enhancing Web Scraping with Fetch and Cheerio

To effectively scrape and parse data, use a library like Cheerio to extract specific elements. First, install Cheerio:

npm install cheerio

Now, use Cheerio to extract the product title:

const cheerio = require('cheerio');

fetch('https://sandbox.oxylabs.io/products/1')
    .then(response => response.text())
    .then(body => {
        const $ = cheerio.load(body);
        console.log($('h2').text());
    });

This is a simple yet powerful combination for data extraction.

Managing Headers with Fetch

When interacting with servers, headers play a vital role. You can access the headers using the response.headers property.

fetch('https://sandbox.oxylabs.io/products/1')
    .then(response => {
        response.headers.forEach((value, key) => {
            console.log(`${key}: ${value}`);
        });
    });

Also, to add custom headers to your request you can specify them in the second parameter:

const headers = {
    headers: {
        'User-Agent': 'My Custom User Agent'
    }
};

fetch('https://ip.oxylabs.io/headers', headers)
    .then(response => response.text())
    .then(body => console.log(body));

Making POST Requests

The default method used by Fetch API is GET, but you can send data to servers using the POST method:

const url = 'https://httpbin.org/post';
const data = { x: 1920, y: 1080 };
const headers = { 'Content-Type': 'application/json' };

fetch(url, {
    method: 'POST',
    headers: headers,
    body: JSON.stringify(data)
})
    .then(response => response.json())
    .then(data => console.log(data));

Handling Errors

Fetch API uses Promises, so use the ‘fetch-then-catch’ approach to manage any issues with your requests.

fetch('https://invalid_url')
    .then(response => response.text())
    .then(body => console.log(body))
    .catch(error => console.error(error));

Or if you’re using an ‘async’ function:

(async () => {
    try {
        const response = await fetch('https://invalid_url');
        const body = await response.text();
        console.log(body);
    } catch (error) {
        console.error(error);
    }
})();

Fetch API vs. Axios

While the Fetch API is robust, there are alternatives, such as Axios. However, Fetch is included in Node.js. Let’s review how Fetch API works against Axios in POST request:

Using Fetch:

const url = 'https://httpbin.org/post';
const customData = { x: 1920, y: 1080 };
const headers = { 'Content-Type': 'application/json' };

fetch(url, {
    method: 'POST',
    headers: headers,
    body: JSON.stringify(customData)
})
    .then(response => response.json())
    .then(data => console.log(data));

Compared to Axios:

const axios = require('axios');
const url = 'https://httpbin.org/post';
const customData = { x: 1920, y: 1080 };
const headers = { 'Content-Type': 'application/json' };

axios.post(url, customData, { headers })
    .then(({ data }) => console.log(data))
    .catch(error => console.error(error));

Here are some of key difference:

• Fetch uses ‘body’, whereas Axios uses ‘data’ for request body.
• With Axios, JSON data is sent directly, while fetch require data conversion to a string first
• Axios handles JSON response automatically, where Fetch needs to parse the data by calling method `response.json()` first.
• Fetch allows the streaming of a response, while Axios does not.

Common Fetch Errors and Mistakes

• The Fetch API does not automatically throw an error for HTTP response codes like 404 or 500. Check the response.ok property to determine if a request is successful.
• The Fetch API does not natively support request timeouts. You must implement timeouts manually, for example using the AbortController.
• When working with the response.body, forgetting to completely consume or close the stream could result in memory leaks. Ensure the stream is fully consumed using .json() or .text()
• Fetch does not include cookies automatically in a request. To include cookies explicitly, specify the `credentials` option.
• CORS errors might cause Fetch API fails silently. Be sure to configure proper server header for cross-origin request.

Leveraging ProxyTee for Enhanced Requests

While Fetch API handles the HTTP requests, ProxyTee enhances this process by providing reliable residential proxies, making your web scraping tasks more effective. ProxyTee offers several key advantages:

• Unlimited Bandwidth: ProxyTee ensures that your high-traffic tasks will not be interrupted by bandwidth concerns.
• Global IP Coverage: Access to over 20 million IPs across 100+ countries with ProxyTee’s extensive global network for precise targeting and local operations.
• Multiple Protocol Support: Supporting both HTTP and SOCKS5 protocols, ProxyTee ensures maximum compatibility with a range of tools and applications.
• Auto Rotation: Benefit from IP auto-rotation which changes your IP address at intervals from 3-60 minutes to avoid IP blocks and restrictions from websites, and can customize this based on need.
• User-Friendly Interface: Start immediately without technical skills, thanks to a clean and easy-to-navigate GUI available in the tool.
• Simple API: Simplify automation for proxy-related tasks by using ProxyTee’s simple API for a seemless experience when incorporating your proxy usage into applications.
• Affordable Pricing: Compared to competitors, ProxyTee’s unlimited residential proxies offer savings as high as 50%, while not compromising quality

Combining Fetch API with ProxyTee, your requests not only function smoothly but also remain anonymous, secure, and cost-effective.