Using Proxies with node-fetch: A Guide by ProxyTee

Mike

• 4 min read
Using Proxies with node-fetch: A Guide by ProxyTee
Photo by Nubelson Fernandes / Unsplash

In today's digital landscape, accessing data from remote APIs and websites is a common requirement for businesses and developers. However, many online platforms enforce strict access controls, including IP-based restrictions, geo-blocking, and bot detection mechanisms. A web proxy server serves as an essential intermediary, helping users bypass these limitations while maintaining anonymity.

Node.js, a popular JavaScript runtime environment, is widely used for server-side applications, including data scraping and automation. While Node.js includes a built-in Fetch API for making HTTP requests, it does not natively support proxies. To address this limitation, developers often turn to node-fetch and https-proxy-agent, which provide seamless integration with proxy servers.

This guide explores how to use these libraries to send HTTP requests through a proxy in Node.js. Additionally, we introduce ProxyTee, a premium proxy provider offering advanced proxy solutions that enhance anonymity, evade detection, and provide seamless access to geo-restricted content.

Getting Started with Proxies in node-fetch

Before diving in, ensure you have:

  • A working installation of Node.js.
  • Access to a web proxy server, or you can create one for testing.

Setting Up Your Node.js Project

Begin by creating and initializing a new Node.js project with the following commands:

mkdir node-fetch-proxy
cd node-fetch-proxy
npm init -y

This will generate a folder named node-fetch-proxy and create a default package.json file.

Installing node-fetch and https-proxy-agent

Install the necessary libraries using npm:

npm install --save node-fetch https-proxy-agent

These libraries are now added to your project's node_modules folder and package.json.

Making HTTP Calls Through a Proxy

Now, use the HttpsProxyAgent class and fetch function to make HTTP calls through a proxy. Create a file named proxy.mjs and add this code:

import fetch from 'node-fetch';
import { HttpsProxyAgent } from 'https-proxy-agent';

// Replace <proxy_url> with your actual proxy URL
const agent = new HttpsProxyAgent('<proxy_url>');

// Use fetch with the agent option to make an HTTP request through the proxy
// Replace <target_url> with the URL you want to request
fetch('<target_url>', { agent })
  .then((response) => response.text())
  .then((text) => console.log(text))
  .catch((error) => console.error(error));

This script does the following:

  • Imports the fetch function from node-fetch for Node.js Fetch API.
  • Imports HttpsProxyAgent for managing HTTPS proxy connections.
  • Creates an instance of HttpsProxyAgent using your proxy URL (format: http://username:password@host:port).
  • Uses fetch with the agent option to make an HTTP request through the specified proxy.
  • Handles responses, logging any errors or output to your terminal.

Execute the code with:

node proxy.mjs

The HTTP response or any errors will be visible in your terminal.

Introducing ProxyTee: Your Proxy Solution

While this setup is useful, real-world use cases such as web scraping require more sophisticated measures. Many websites employ IP blocking techniques for well-known proxies, and geographical restrictions can limit access. Thankfully, ProxyTee addresses these issues by providing advanced proxy types and features

ProxyTee provides affordable, reliable, and user-friendly rotating residential proxies. Our proxies are designed to support various tasks requiring anonymity and IP rotation, such as web scraping, streaming, and data gathering. Key features include:

  • Unlimited Bandwidth: Perform data-intensive tasks without worrying about bandwidth limitations.
  • Global IP Coverage: Choose from a vast pool of 20 million+ IPs across 100+ countries to reach geo-specific content.
  • Multiple Protocol Support: Our proxies support both HTTP and SOCKS5 for enhanced compatibility.
  • Auto Rotation: IP addresses rotate automatically at customizable intervals from 3 to 60 minutes, helping to evade detection
  • User-Friendly Interface: Get up and running quickly with our easy-to-use platform, requiring minimal setup time.
  • Simple API: Automate your proxy workflows with our simple API, supporting all our service features.

ProxyTee's Product Range

ProxyTee offers diverse solutions tailored to different needs. Let's explore each product in our lineup.

Unlimited Residential Proxies

Our flagship Unlimited Residential Proxies provide exactly that—unlimited bandwidth and rotating residential IPs—for truly unthrottled access. With precise geo-targeting capabilities and competitive pricing—50% cheaper than the competition—we empower users with high performance at the best price point.

Residential Proxies

The foundation of our service, Residential Proxies route your requests via genuine residential IPs. This method mirrors regular user traffic, making it more difficult for sites to block. They are suitable for all web scraping and data gathering tasks requiring residential anonymity.

Datacenter Proxies

Datacenter Proxies employ IP addresses located within a server facility. These proxies provide robust data access at high speeds, ideal for projects needing reliable connections. Even with a lower anonymity profile, they serve valuable purposes in various scraping scenarios.

Static Residential Proxies

Static Residential Proxies grant users fixed, dependable IPs that are sourced from genuine ISPs, not data centers, allowing you to maintain a constant IP address over extended periods. This reduces the risk of flagging.

Mobile Proxies

Mobile Proxies route traffic via mobile devices over 3G/4G/5G networks. These are virtually undetectable, perfect when interacting with mobile-specific APIs or scraping mobile websites.

Conclusion

This post has demonstrated how to use node-fetch with https-proxy-agent to create HTTP requests using proxies. While this technique is foundational, advanced tasks demand robust solutions like ProxyTee.

With a wide array of residential proxies, our infrastructure equips you with the tools necessary to navigate the web seamlessly. Our offerings ensure you avoid blocks and bypass geographical restrictions. For a deeper look at our features and options, visit our pricing page to get started and find a solution tailored for your specific requirements.

Tags:
Blog NodeJS Web Scraping API