Setting Up Proxy Servers with Go: A Comprehensive Guide

All internet interactions rely on IP addresses. Websites use these addresses to identify users, determine their location, and gather other metadata like ISP, time zone, and device type. This information is often used to tailor content or restrict access. When web scraping, websites can block requests from your IP address if the traffic pattern or behavior seems unusual or malicious. That’s where proxy servers come into play. ProxyTee offers a variety of proxy solutions to help you browse and scrape discreetly, enhancing security, privacy, and anonymity.

A proxy server is an intermediary server acting as a gateway between a user and the internet. It receives user requests, forwards them to web resources, and then returns the fetched data to the users. This process helps you browse and scrape discreetly by hiding your real IP address, enhancing security, privacy, and anonymity.

Proxy servers also assist in circumventing IP bans by changing your IP address, making it appear as if requests are coming from different users. Furthermore, proxy servers in different regions enable you to access geo-specific content, like movies or news, bypassing geoblocking.

In this article, you'll learn how to set up a proxy server for web scraping in Go. We’ll also introduce you to ProxyTee's offerings and how they can simplify this process.

Set Up a Proxy Server

In this tutorial, you'll learn how to modify a web scraper application written in Go to interact with a website via a local or self-hosted proxy server. This tutorial assumes that you already have your Go development environment set up.

To begin, you need to set up your proxy server using Squid, an open-source proxy server software. If you're more comfortable with another proxy server software, you may use that as well. This article will be using Squid on a Fedora 39 Linux box. For most Linux distributions, Squid is included in the default repositories. You can also check the documentation to download the necessary packages for your operating system.

From your terminal, execute the following command to install Squid:

dnf install squid -y

Once completed, start the service by executing the following command:

sudo systemctl enable --now squid

Check the status of the service using this command:

sudo systemctl status squid

Your output should look something like the following:

● squid.service - Squid caching proxy
     Loaded: loaded (/usr/lib/systemd/system/squid.service; enabled; preset: disabled)
     Active: active (running) since Fri 2025-06-07 09:36:33 UTC; 22min ago
       Docs: man:squid(8)
   Main PID: 18210 (squid)
      Tasks: 3 (limit: 4899)
     Memory: 21.7M (peak: 26.2M)
        CPU: 1.171s
     CGroup: /system.slice/squid.service
             └─18210 /usr/sbin/squid -YC -f /etc/squid/squid.conf

This indicates that the service is active and running. By default, Squid runs and listens to requests on Port 3128. Use the following curl command to test communication via the proxy server:

curl --proxy 127.0.0.1:3128 "http://lumtest.com/myip.json"

Your response should include your public IP address, country, and organization. This confirms that you have successfully installed a working proxy server.

Set Up the Demo Scraper

For demonstration, a simple Go web scraper application is available in this GitHub repository. The scraper captures the title, excerpt, and categories of blog posts. It also uses Lumtest to get info about the IP address. The same logic is implemented using three different Go packages: Colly, goquery, and Selenium. In the next section, you will learn how to modify each implementation to use a proxy server.

Start by cloning the repository using the following command:

$ git clone https://github.com/rexfordnyrk/go_scrap_proxy.git

This repository has two branches: main, containing completed code, and basic, containing the initial code to modify. Use the command to checkout the basic branch:

$ git checkout basic

This branch includes three .go files for each library implementation without a proxy, and a chromedriver executable file required by the Selenium implementation.

You can run them using go run with a specific filename, like below:

go run ./colly.go

This will output all scraped article information. At the bottom of the output, you’ll find the IP information, telling you about the current connection used by the scraper. Running each should yield similar responses. Once tested, you can start scraping with a local proxy.

Implementing Scrapers with Local Proxy

In this section, you'll learn how to modify the scraper implementations to use your proxy server. Each .go file contains the main() function and the ScrapeWithLibrary() function.

Using goquery with a Local Proxy

goquery is a Go library for parsing and manipulating HTML documents. It doesn't handle network requests, meaning you must get the HTML page to parse. To use it with a local proxy, you will modify the HTTP client's transport with your HTTP proxy server’s URL, which is in the format http://HOST:PORT. Don't forget to import the net/url package.

Inside the ScrapeWithGoquery() function, replace the HTTP client definition with this snippet:

func ScrapeWithGoquery() {
    // Define the URL of the proxy server
    proxyStr := "http://127.0.0.1:3128"

    // Parse the proxy URL
    proxyURL, err := url.Parse(proxyStr)
    if err != nil {
        fmt.Println("Error parsing proxy URL:", err)
        return
    }

    //Create an http.Transport that uses the proxy
    transport := &http.Transport{
        Proxy: http.ProxyURL(proxyURL),
    }

    // Create an HTTP client with the transport
    client := &http.Client{
        Transport: transport,
    }
    

Run it by:

go run ./goquery.go

Using Colly with a Local Proxy

Colly is a web scraping framework known for its user-friendly API. It provides an API to handle network-related behavior. It also supports proxy usage with switching and rotation. Open the colly.go file and paste these lines right after initializing a new collector in the ScrapeWithColly() function:

   // Define the URL of the proxy server
    proxyStr := "http://127.0.0.1:3128"
    // SetProxy sets a proxy for the collector
    if err := c.SetProxy(proxyStr); err != nil {
        log.Fatalf("Error setting proxy configuration: %v", err)
    }

This code uses Colly's SetProxy() to define the proxy for network requests.

Then run by:

go run ./colly.go

Using Selenium with a Local Proxy

Selenium is a tool for automating web browser interactions. It is useful for mimicking real user interactions. You can use it with proxy servers via selelium.Capabilities{}. Open the selenium.go file and replace the selelium.Capabilities{} definition inside ScrapeWithSelenium() with the code:

    // Define proxy settings
    proxy := selenium.Proxy{
        Type: selenium.Manual,
        HTTP: "127.0.0.1:3128", // Replace with your proxy settings
        SSL:  "127.0.0.1:3128", // Replace with your proxy settings
    }

    // Configuring the WebDriver instance with the proxy
    caps := selenium.Capabilities{
        "browserName": "chrome",
        "proxy":       proxy,
    }

This sets up proxy parameters for Selenium's WebDriver. Now run using the command below:

go run ./selenium.go

While you can manage your proxy server, you may be limited by the ability to set up new servers in different regions and do all the related maintenance and security issues.

ProxyTee Proxy Servers

ProxyTee offers a reliable and affordable proxy service for web scraping and other internet activities. With a vast global network, including Residential Proxies and Datacenter Proxies, ProxyTee can provide various options with benefits like anonymity, speed, and reliability.

The auto-rotation feature of ProxyTee also ensures high anonymity, minimizing detection from web servers. ProxyTee's Unlimited Residential Proxies are a great solution for data-gathering tasks.

Here are a few features of ProxyTee:

1. Unlimited Bandwidth: ProxyTee provides proxies with unlimited bandwidth, eliminating data overage concerns during high-traffic tasks like web scraping or streaming.
2. Global IP Coverage: With a vast pool of IP addresses from more than 100 countries, ProxyTee allows users to access a wide range of geographic locations, which is useful for performing location-based tasks or accessing regional content.
3. Multiple Protocol Support: ProxyTee supports both HTTP and SOCKS5 protocols, ensuring compatibility with different applications and tools.
4. User-Friendly Interface: The ProxyTee platform has a clean and intuitive GUI that helps users get started quickly. It requires minimal setup and no technical expertise.
5. Auto Rotation: The auto-rotation feature automatically changes IP addresses, which is ideal for web scraping, and can be customized to suit your needs.
6. API Integration: ProxyTee offers a simple API that helps with seamless integration into various applications. This helps developers automate tasks.

Setting Up a Residential Proxy with ProxyTee

It is easy to set up a residential proxy with ProxyTee. Visit the website, and choose your residential proxy plan. ProxyTee provides various options with affordable prices. Next, you’ll be prompted to set up your billing information and will receive a credential after activation.

Now you need to integrate your ProxyTee residential proxy with your code.

Using goquery with a ProxyTee Residential Proxy

Inside the ScrapeWithGoquery() function, you need to modify the proxyStr to include authentication credentials in the proxy URL format http://USERNAME:password@HOST:PORT. Replace the definition with this code:

func ScrapeWithGoquery() {
    // Define the proxy server with username and password
    proxyUsername := "username" //Your residential proxy username 
    proxyPassword := "your_password" //Your Residential Proxy password here
    proxyHost := "server_host" //Your Residential Proxy Host
    proxyPort := "server_port"  //Your Port here
    
    proxyStr := fmt.Sprintf("http://%s:%s@%s:%s", url.QueryEscape(proxyUsername), url.QueryEscape(proxyPassword), proxyHost, proxyPort)
    
    // Parse the proxy URL

You’ll also need to ignore verifying SSL/TLS certificates. Add the crypto/tls package to imports and replace the http.Transport definition after parsing the proxy URL with the code:

    //Create an http.Transport that uses the proxy
    transport := &http.Transport{
        Proxy: http.ProxyURL(proxyURL),
        TLSClientConfig: &tls.Config{
            InsecureSkipVerify: true, // Disable SSL certificate verification
        },
    }

    // Create an HTTP client with the transport

Run the code using this command

go run brightdata/goquery.go

Using Colly with a ProxyTee Residential Proxy

To use Colly with a ProxyTee residential proxy, you need to supply a custom HTTP transport in order to ignore SSL verification. Modify ScrapeWithColly(), adding the following after initializing a new collector. Don’t forget to import net/url and net/http:

func ScrapeWithColly() {
    ...
    
    //Create an http.Transport that uses the proxy
    transport := &http.Transport{
        TLSClientConfig: &tls.Config{
            InsecureSkipVerify: true, // Disable SSL certificate verification
        },
    }
    
    // Set the collector instance to use the configured transport
    c.WithTransport(transport)

Next, change the proxyStr to contain the residential proxy credentials:

    // Define the proxy server with username and password
    proxyUsername := "username" //Your residential proxy username 
    proxyPassword := "your_password" //Your Residential Proxy password here
    proxyHost := "server_host" //Your Residential Proxy Host
    proxyPort := "server_port"  //Your Port here

    proxyStr := fmt.Sprintf("http://%s:%s@%s:%s", url.QueryEscape(proxyUsername), url.QueryEscape(proxyPassword), proxyHost, proxyPort)

Then, run the code using the command

go run brightdata/colly.go

Using Selenium with a ProxyTee Residential Proxy

When working with Selenium, modify selenium.Proxy{} to include the proxy URL string and the authentication credentials. Replace the current proxy definition with:

    // Define the proxy server with username and password
    proxyUsername := "username"      //Your residential proxy username
    proxyPassword := "your_password" //Your Residential Proxy password here
    proxyHost := "server_host"       //Your Residential Proxy Host
    proxyPort := "server_port"       //Your Port here

    proxyStr := fmt.Sprintf("http://%s:%s@%s:%s", url.QueryEscape(proxyUsername), url.QueryEscape(proxyPassword), proxyHost, proxyPort)

    // Define proxy settings
    proxy := selenium.Proxy{
        Type: selenium.Manual,
        HTTP: proxyStr,
        SSL:  proxyStr,
    }
    

Don't forget to import the net/url package.

To disable SSL certificate verification, modify the chromeCaps definition:

    caps.AddChrome(chrome.Capabilities{Args: []string{
        "--headless=new", // Start browser without UI as a background process
        "--ignore-certificate-errors", // // Disable SSL certificate verification
    }})

Finally, because Selenium doesn’t support authenticated proxy configuration natively, we will use an extension. Add the proxyauth package:

go get https://github.com/rexfordnyrk/proxyauth

Then, import the package into the brightdata/selenium.go file and paste this snippet after the chromeCaps definition, but before the caps.AddChrome(chromeCaps) line:

    //Building proxy auth extension using BrightData Proxy credentials
    extension, err := proxyauth.BuildExtention(proxyHost, proxyPort, proxyUsername, proxyPassword)
    if err != nil {
        log.Fatal("BuildProxyExtension Error:", err)
    }

    //including the extension to allow proxy authentication in chrome
    if err := chromeCaps.AddExtension(extension); err != nil {
        log.Fatal("Error adding Extension:", err)
    }

Run with the command:

go run brightdata/selenium.go

With ProxyTee, setting up your proxy is straightforward. Create your residential proxy, obtain your credentials, and then modify your code to use the proxy for web scraping.

Conclusion

Web proxy servers are critical for tailored internet interactions. In this article, you learned about proxy servers, how to set up one using Squid, and how to integrate proxies into Go applications, specifically for web scraping. You also learned how ProxyTee provides state-of-the-art proxy infrastructure. ProxyTee helps you collect data efficiently without the need for additional infrastructure and maintenance. Check out ProxyTee pricing for different use cases.