Web Scraping with Go: A Practical Guide Using ProxyTee

In this tutorial, you'll discover why Go is an excellent language for efficient web scraping and learn how to build a Go scraper from the ground up, enhanced by the power of ProxyTee.

This post will cover:

• Can you perform web scraping with Go?
• Best Go web scraping libraries
• Build a web scraper in Go with ProxyTee integration

Can You Perform Web Scraping With Go?

Go, or Golang, is a statically-typed programming language developed by Google. It's known for its efficiency, concurrency, and ease of maintenance, making it increasingly popular for web scraping tasks. The language's features make it ideal for high-performance scraping applications. This efficiency is crucial for dealing with the demands of modern web scraping, which often involves handling multiple requests and parsing large datasets. With ProxyTee's robust proxy network, you can leverage these advantages without worrying about IP blocks or geo-restrictions, ensuring your scrapers operate effectively.

Key benefits of using Go for web scraping include its built-in concurrency model that allows multiple web requests to be handled at the same time. Go’s standard library has all the tools you need, such as an HTTP client and HTML parsing, to get web pages, read HTML and get data. ProxyTee’s auto-rotation ensures your IP is constantly changing to avoid detection. For these default packages, there are a variety of web scraping libraries to choose from, here’s a look at some of the most popular:

Best Go Web Scraping Libraries

Here are some of the best web scraping libraries for Go:

• Colly: A powerful web scraping and crawling framework for Go, offering a functional API for making HTTP requests, managing headers, and parsing the DOM. It supports parallel scraping, rate limiting, and automatic cookie handling.
• Goquery: This library, using a jQuery-like syntax for HTML parsing, enables the selection of HTML elements through CSS selectors, DOM manipulation, and data extraction.
• Selenium: A client for the popular web testing framework that automates web browsers for web scraping. Selenium controls a web browser and simulates human interaction with pages and can perform scraping on JavaScript-heavy sites.

Combine these tools with ProxyTee’s Unlimited Residential Proxies and you get a potent setup for reliable and fast web scraping.

Prerequisites

Before you begin, you'll need to install Go on your machine. The installation process may vary depending on your operating system:

Set up Go on macOS

• Download Go.
• Open the downloaded file and follow the instructions in the installation prompts. The package will be installed to /usr/local/go and /usr/local/go/bin will be added to your PATH environment variable.
• Restart any open Terminal sessions.

Set up Go on Windows

• Download Go.
• Run the downloaded MSI file and follow the prompts in the installation wizard. The installer will install Go in C:/Program Files or C:/Program Files (x86) and will add the bin folder to the PATH environment variable.
• Close and reopen any command prompts.

Set up Go on Linux

• Download Go.
• Ensure your system doesn’t have a /usr/local/go folder. If it exists, delete it using rm -rf /usr/local/go.
• Extract the downloaded archive to /usr/local: tar -C /usr/local -xzf goX.Y.Z.linux-amd64.tar.gz, replacing X.Y.Z with the Go package version you downloaded.
• Add /usr/local/go/bin to the PATH environment variable with export PATH=$PATH:/usr/local/go/bin.
• Reload your PC.

Verify your Go installation with go version, which will return your Go version, like go version go1.20.3.

Build a Web Scraper in Go

This guide shows you how to create a Go web scraper. This program will get information from the ProxyTee home page. The scraping program will get HTML elements from the page and convert the info into a useful format.

Step 1: Set up a Go project

Let's get started setting up your Go web scraper project by opening the terminal and running:

mkdir go-web-scraper

This directory will hold your Go project. Then, run the init command to initialize the project module.

go mod init web-scraper

This command will create a web-scraper module in your root folder.

The go-web-scraper folder now contains a go.mod file:

module web-scraper

go 1.20

The last line depends on your Go version.

Time to add some Go code in your IDE. This guide uses Visual Studio Code. As VS Code doesn't support Go out of the box, install the Go extension by selecting the “Extensions” icon on the left sidebar and searching for “Go.” Click on “Install” for the “Go for Visual Studio Code” extension. Next, click on “File,” select “Open Folder…,” and navigate to the go-web-scraper directory. Then, in the “Explorer” section, right-click, choose “New File…,” and create a scraper.go file:

// scraper.go

package main

import (
	"fmt"
)

func main() {
	fmt.Println("Hello, World!")
}

The main() function is the starting point for your Go application. VS Code will ask to install packages to work with Go, so install them all. Now, run the Go script from the VS Terminal with the command below.

go run scraper.go

This will print Hello, World!

Step 2: Get Started With Colly

For easier web scraping in Go, use one of the packages described earlier. Check your target website by right-clicking the background, then choosing “Inspect.” This will bring up your browser’s DevTools. Go to “Network,” then to the “Fetch/XHR” section.

The site doesn’t do many AJAX calls. The data is included in the HTML document returned by the server. For sites like this that have static content, an HTML parser such as Colly is a great choice. Use the following command to add Colly to your project dependencies:

go get github.com/gocolly/colly

This generates a go.sum file and modifies the go.mod file.

Some of the key Colly concepts:

The main part of Colly is the Collector. This lets you do HTTP requests and web scraping by callbacks:

• OnRequest(): Called before any HTTP request with Visit().
• OnError(): Called if there is an HTTP request error.
• OnResponse(): Called after a response from the server.
• OnHTML(): Called after OnResponse() when the server returns valid HTML.
• OnScraped(): Called after the end of all OnHTML() calls.

Each function takes a callback as a parameter. Colly will use this when its associated event happens. To use a Colly scraper you must follow this functional approach.

The function NewCollector() is used to create a Collector object:

c := colly.NewCollector()

Import Colly and make a Collector by modifying scraper.go as below:

// scraper.go

package main

import (
  // import Colly
  "github.com/gocolly/colly"
)

func main() {
  c := colly.NewCollector()
  // scraping logic...
}

Step 3: Connect to the Target Website

Connect to the target page using Colly:

c.Visit("https://proxytee.com/")

Visit() carries out an HTTP GET request and downloads the HTML file from the server. It will trigger the onRequest event and start Colly’s lifecycle. Visit() is called after registering the other callbacks.

Sometimes the HTTP request fails, and if it does Colly triggers the OnError event. A server outage, bad URL or even a website blocking your scraping bot could cause this. Many websites adopt anti-bot measures that check for valid User-Agent HTTP headers. Find out more about ProxyTee’s anti-bot solutions and get a valid User-Agent HTTP header:

c.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"

Any Visit() call will now do a request using that HTTP header. Here is the updated scraper.go file:

// scraper.go

package main

import (
  // import Colly
  "github.com/gocolly/colly"
)

func main() {
  // initialize the Collector
  c := colly.NewCollector()
  // set a valid User-Agent header
  c.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
  // connect to the target site
  c.Visit("https://proxytee.com/")
  // scraping logic...
}

Step 4: Inspect the HTML Page

Let’s analyze the DOM of ProxyTee’s home page for an effective way to get data. Visit the homepage, where you will notice cards showcasing ProxyTee’s services with related industries, for example the Residential Proxies, Datacenter Proxies, and so on. This information is good to scrape. If you right click on the card and click Inspect you will see the card's HTML code in the DOM.

Each service card is an <a> HTML element with the following child elements:

• A <figure> with the service image.
• A <div> with the name of the service field.

Using the CSS classes of the HTML elements and their parents, you’ll have the right CSS selectors to get the elements you need.

The cards use the class section_cases__item which is inside the <div class="elementor-element-6b05593c">. So you can use the CSS selector below to get all service cards:

.elementor-element-6b05593c .section_cases__item

For each card, get the child elements with these selectors:

.elementor-image-box-img img

.elementor-image-box-content .elementor-image-box-title

The goal here is to get the URL, image, and service name from each card.

Step 5: Select HTML Elements with Colly

You can apply CSS or XPath selectors with Colly as below:

c.OnHTML(".your-css-selector", func(e *colly.HTMLElement) {
    // data extraction logic...
})

Colly will use the function you give it on each matching HTML element, automatically looping through each selected element.

A Collector has many OnHTML() callbacks which will run in the same order the onHTML() instructions are in your code.

Step 6: Scrape Data from a Webpage With Colly

Use Colly to extract data from the HTML webpage.

Create a Struct called Service to store your scraped data:

type Service struct {
    Url, Image, Name string
}

The struct sets a group of fields which can be made into objects. A Go slice can store the struct you create:

var services []Service

The slice works like an array or list.

The OnHTML() function helps set up the scraping logic:

// iterating over the list of service card
// HTML elements
c.OnHTML(".elementor-element-6b05593c .section_cases__item", func(e *colly.HTMLElement) {
    url := e.Attr("href")
    image := e.ChildAttr(".elementor-image-box-img img", "data-lazy-src")
    name := e.ChildText(".elementor-image-box-content .elementor-image-box-title")
    // filter out unwanted data
    if url != "" || image != "" || name != "" {
        // initialize a new Service instance
        service := Service{
            Url:   url,
            Image: image,
            Name:  name,
        }
        // add the service instance to the list
        // of scraped services
        services = append(services, service)
    }
})

This snippet selects all the cards on the ProxyTee homepage and loops through each one getting the URL, image, and name for the card. It then creates an object of the type Service and appends it to the slice.

Running a Colly scraper is quite easy. The method Attr() is for getting a specific HTML attribute from the current element. For getting attribute values or text, ChildAttr() and ChildText() are what you need, they use CSS selectors to find child HTML elements.

This means that you can also get data from service details pages. All you need is to follow the links you find on your current page, and add new scraping logic to suit the need. This is how web crawling and web scraping work together!

Step 7: Export the Extracted Data

Once the OnHTML() instruction has finished, the slice services stores all the data in Go objects. For easier data access you'll want to change the data format. Below are steps for converting your data to CSV and JSON files.

Go has great standard data exporting features so external packages aren't required for CSV or JSON conversions. Add the following import statements to your Go script.

For CSV export:

import (
    "encoding/csv"
    "log"
    "os"
)

For JSON export:

import (
   "encoding/json"
   "log"
   "os"
)

To export the services slice to a CSV, use the snippet below:

// open the output CSV file
file, err := os.Create("services.csv")
// if the file creation fails
if err != nil {
    log.Fatalln("Failed to create the output CSV file", err)
}
// release the resource allocated to handle
// the file before ending the execution
defer file.Close()

// create a CSV file writer
writer := csv.NewWriter(file)
// release the resources associated with the 
// file writer before ending the execution
defer writer.Flush()

// add the header row to the CSV
headers := []string{
    "url",
    "image",
    "name",
}
writer.Write(headers)

// store each Service product in the
// output CSV file
for _, service := range services {
    // convert the Service instance to
    // a slice of strings
    record := []string{
        service.Url,
        service.Image,
        service.Name,
    }
    
    // add a new CSV record
    writer.Write(record)
}

This snippet creates a new CSV, writes the headers, and then steps through each Service converting each element to strings before writing them to the output file. Go's CSV Writer will turn the list of strings into CSV records. To execute the script use the following command.

go run scraper.go

A new services.csv file should be in the root folder of your project. You will see data similar to the below.

url,image,name
https://proxytee.com/products/residential-proxies//images/products/22038e906687.png,Residential Proxies
https://proxytee.com/products/unlimited-residential-proxies//images/products/22038e906687.png,Unlimited Residential Proxies
https://proxytee.com/products/static-residential-proxies//images/products/22038e906687.png,Static Residential Proxies
https://proxytee.com/products/proxy-proxies//images/products/494eea37589a.png,Mobile Proxies
https://proxytee.com/products/datacenter-proxies//images/products/8c433ac212f2.png,Datacenter Proxies

Similarly, export to a JSON format with the snippet below:

file, err := os.Create("services.json")

if err != nil {
    log.Fatalln("Failed to create the output JSON file", err)
}

defer file.Close()

// convert services to an indented JSON string
jsonString, _ := json.MarshalIndent(services, " ", " ")

// write the JSON string to file
file.Write(jsonString)

The JSON file will be similar to the following:

[
  {
    "Url": "https://proxytee.com/products/residential-proxies/",
    "Image": "/images/products/22038e906687.png",
    "Name": "Residential Proxies"
  },
    {
        "Url": "https://proxytee.com/products/unlimited-residential-proxies/",
        "Image": "/images/products/22038e906687.png",
        "Name": "Unlimited Residential Proxies"
    },
    {
        "Url": "https://proxytee.com/products/static-residential-proxies/",
        "Image": "/images/products/22038e906687.png",
        "Name": "Static Residential Proxies"
    },
    {
        "Url": "https://proxytee.com/products/proxy-proxies/",
        "Image": "/images/products/494eea37589a.png",
        "Name": "Mobile Proxies"
    },
  {
    "Url": "https://proxytee.com/products/datacenter-proxies/",
    "Image": "/images/products/8c433ac212f2.png",
    "Name": "Datacenter Proxies"
  }
]

Your data is now in an easily accessible format!

Step 8: Put it All Together

The complete Golang scraper code is shown below:

// scraper.go
package main

import (
  "encoding/csv"
  "encoding/json"
  "log"
  "os"
  // import Colly
  "github.com/gocolly/colly"
)

// define data structures to store the scraped data
type Service struct {
  Url, Image, Name string
}

func main() {
  // initialize the struct slices
  var services []Service

  // initialize the Collector
  c := colly.NewCollector()

  // set a valid User-Agent header
  c.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
  
  // iterating over the list of service card
  // HTML elements
  c.OnHTML(".elementor-element-6b05593c .section_cases__item", func(e *colly.HTMLElement) {
    url := e.Attr("href")
    image := e.ChildAttr(".elementor-image-box-img img", "data-lazy-src")
    name := e.ChildText(".elementor-image-box-content .elementor-image-box-title")
    // filter out unwanted data
    if url != "" && image != "" && name != "" {
      // initialize a new Service instance
      service := Service{
        Url:   url,
        Image: image,
        Name:  name,
      }
      // add the service instance to the list
      // of scraped services
      services = append(services, service)
    }
  })

  // connect to the target site
  c.Visit("https://proxytee.com/")

  // --- export to CSV ---

  // open the output CSV file
  csvFile, csvErr := os.Create("services.csv")
  // if the file creation fails
  if csvErr != nil {
    log.Fatalln("Failed to create the output CSV file", csvErr)
  }
  // release the resource allocated to handle
  // the file before ending the execution
  defer csvFile.Close()

  // create a CSV file writer
  writer := csv.NewWriter(csvFile)
  // release the resources associated with the
  // file writer before ending the execution
  defer writer.Flush()

  // add the header row to the CSV
  headers := []string{
    "url",
    "image",
    "name",
  }
  writer.Write(headers)

  // store each Service product in the
  // output CSV file
  for _, service := range services {
    // convert the Service instance to
    // a slice of strings
    record := []string{
      service.Url,
      service.Image,
      service.Name,
    }
    // add a new CSV record
    writer.Write(record)
  }

  // --- export to JSON ---

  // open the output JSON file
  jsonFile, jsonErr := os.Create("services.json")
  if jsonErr != nil {
    log.Fatalln("Failed to create the output JSON file", jsonErr)
  }
  defer jsonFile.Close()
  // convert services to an indented JSON string
  jsonString, _ := json.MarshalIndent(services, " ", " ")

  // write the JSON string to file
  jsonFile.Write(jsonString)
}

This Go program creates a functional web scraper in a very short amount of code.

Conclusion

This post has explored using Go for web scraping, with an emphasis on ease of use, and the best scraping libraries it has to offer. It has also shown you the power of combining ProxyTee with Go to build powerful data harvesting solutions. The practical scraper that you built in this guide, combined with ProxyTee's features such as unlimited bandwidth, global IP coverage, multiple protocol support, and auto-rotation, makes for an efficient tool. With less than 100 lines of code, Go combined with the correct proxy solutions makes web scraping a simple, practical, and quick to implement task.

Websites adopt anti-scraping techniques and anti-bot solutions to block scraping bots. ProxyTee delivers Unlimited Residential Proxies with global coverage, along with multiple proxy protocols for every need. Contact us to see which solutions fit your use case best, including affordable pricing plans.