Web Scraping with C#: A Comprehensive Guide Using ProxyTee

When embarking on a web scraping project, one of the initial key decisions revolves around selecting the programming language. C# stands out as a robust, general-purpose language equipped with a plethora of tools suitable for addressing a diverse range of web scraping challenges. C#, while being part of the C family, distinguishes itself with its efficiency and scalability while maintaining simplicity and ease of use. In this guide, we'll explore how to leverage C# for web scraping, demonstrating with real-world examples. Keep in mind that the techniques discussed are transferable to other languages within the .NET ecosystem, such as VB.NET and F#.

C# Web Scraping Tools

Before diving into the coding phase, it's crucial to select the appropriate C# library or package. These packages equip you with the essential functionality needed to download HTML content, parse it, and efficiently extract the desired data. Here are some of the top C# packages for web scraping:

• Html Agility Pack: This is one of the most popular C# packages with nearly 50 million downloads from NuGet alone. Its popularity stems from its ability to handle malformed HTML and support for XPath. It also offers the flexibility to parse local HTML files, and directly download web pages.
• ScrapySharp: Enhancing the capabilities of C#, ScrapySharp includes support for CSS Selectors and browser simulation. However, it's worth noting that ScrapySharp isn’t as actively maintained as other tools in the C# web scraping ecosystem.
• Puppeteer Sharp: This package is a .NET port of the popular Puppeteer project, designed for Node.js. Puppeteer Sharp utilizes the Chromium browser to load pages and employs asynchronous code behavior with async-await patterns, making it a good option for those familiar with its architecture or those needing a browser to render dynamic content.

Building a web scraper with C#

For our demonstration, we will use Html Agility Pack within a .NET 5 SDK environment in Visual Studio Code. This method is compatible with .NET Core 3 and should be effective across other versions of .NET.

We'll mimic scraping data from a hypothetical bookstore, aiming to collect the names and prices of books. Before that, let's set up our development environment.

Setup Development environment

First, ensure you have Visual Studio Code installed. Be aware that Visual Studio and Visual Studio Code are different. Then, download and install .NET 5.0 or a newer version. Once installation is complete, check the command line interface by using this command:

dotnet --version

The terminal should display your installed .NET version.

Project Structure and Dependencies

The scraping script will reside within a .NET project. To begin, create a console application by opening your terminal in the directory and typing:

dotnet new console

The console will confirm that the application has been successfully created. Now, install the necessary packages to proceed. Use the following command to install the HtmlAgilityPack package for the project.

dotnet add package HtmlAgilityPack

For CSV exports, use the command:

dotnet add package CsvHelper

If using Visual Studio, go to Project > Manage Project Dependencies to install those packages by searching on the NuGet Packages window.

With the packages set up, we can move on to writing code.

Download and Parse Web Pages

The foundation of any scraping project is fetching and processing the HTML of the desired pages. This is handled by Html Agility Pack's HtmlWeb class. It offers the convenient Load function to retrieve HTML from URLs and convert it into an HtmlDocument object.

To start, add necessary imports in the Program.cs file:

using HtmlAgilityPack;

Next, add the following GetDocument function:

static HtmlDocument GetDocument(string url)
{
    HtmlWeb web = new HtmlWeb();
    HtmlDocument doc = web.Load(url);
    return doc;
}

This completes the downloading aspect; next is parsing.

Parsing the HTML: Getting Book Links

Here, we extract data using HtmlDocument methods to select elements by XPath. The two available methods are SelectNodes and SelectSingleNode:

public HtmlNodeCollection SelectNodes(string xpath);
public HtmlNode SelectSingleNode(string xpath);

We'll use SelectNodes to capture all book details from our target page, parsing it to extract each book link. Inspecting a page can be done using the browser’s developer tools to grab the desired XPath for the elements you wish to extract. We can use //h3/a for book links in our example.

HtmlDocument doc = GetDocument(url);
HtmlNodeCollection linkNodes = doc.DocumentNode.SelectNodes("//h3/a");

The linkNodes will return a collection of HtmlNode. Because the links on a web page are typically relative paths, converting to an absolute URL is required. Here's the updated function to fetch the URLs of all book links:

static List<string> GetBookLinks(string url)
{
    var bookLinks = new List<string>();
    HtmlDocument doc = GetDocument(url);
    HtmlNodeCollection linkNodes = doc.DocumentNode.SelectNodes("//h3/a");
    var baseUri = new Uri(url);
    foreach (var link in linkNodes)
    {
        string href = link.Attributes["href"].Value;
        bookLinks.Add(new Uri(baseUri, href).AbsoluteUri);
    }
    return bookLinks;
}

In our Main function, it is good practice to confirm the code is operating properly.

static void Main(string[] args)
{
    var bookLinks = GetBookLinks("http://books.toscrape.com/catalogue/category/books/mystery_3/index.html");
    Console.WriteLine("Found {0} links", bookLinks.Count);
}

Run dotnet run on your terminal to see the result. It should output the number of links extracted.

Parsing the HTML: Getting Book Details

Now with the book links, the next step involves parsing each book page to extract the book name and price, for which a class is created first to hold the required data.

public class Book
{
    public string Title { get; set; }
    public string Price { get; set; }
}

The XPath for book titles on our example page will be //h1. Finding the price on the page is a bit complex so we need the XPath //div[contains(@class,"product_main")]/p[@class="price_color"].

To acquire these details using SelectSingleNode, the function will look like the following.

static List<Book> GetBookDetails(List<string> urls)
{
    var books = new List<Book>();
    foreach (var url in urls)
    {
        HtmlDocument document = GetDocument(url);
        var titleXPath = "//h1";
        var priceXPath = "//div[contains(@class,\"product_main\")]/p[@class=\"price_color\"]";
        var book = new Book();
        book.Title = document.DocumentNode.SelectSingleNode(titleXPath).InnerText;
        book.Price = document.DocumentNode.SelectSingleNode(priceXPath).InnerText;
        books.Add(book);
    }
    return books;
}

The Main function can be modified to also verify that these steps also complete correctly by using the following code.

static void Main(string[] args)
{
    var bookLinks = GetBookLinks("http://books.toscrape.com/catalogue/category/books/mystery_3/index.html");
    Console.WriteLine("Found {0} links", bookLinks.Count);
    var books = GetBookDetails(bookLinks);
}

Next is the part where we write out the extracted information into a CSV file.

Exporting Data

Install the CsvHelper library as outlined in the previous setup section and use the following statement.

using System.Globalization;
using CsvHelper;

For the function to perform CSV exports:

static void exportToCSV(List<Book> books)
{
    using (var writer = new StreamWriter("./books.csv"))
    using (var csv = new CsvWriter(writer, CultureInfo.InvariantCulture))
    {
        csv.WriteRecords(books);
    }
}

Modify the Main() function so that it also triggers this function.

static void Main(string[] args)
{
    var bookLinks = GetBookLinks("http://books.toscrape.com/catalogue/category/books/mystery_3/index.html");
    var books = GetBookDetails(bookLinks);
    exportToCSV(books);
}

To bring everything together, below is the complete code so far.

using System.Globalization;
using CsvHelper;
using HtmlAgilityPack;

namespace webscraping
{
    class Program
    {
        static HtmlDocument GetDocument(string url)
        {
            HtmlWeb web = new HtmlWeb();
            HtmlDocument doc = web.Load(url);
            return doc;
        }
        static List<string> GetBookLinks(string url)
        {
            var bookLinks = new List<string>();
            HtmlDocument doc = GetDocument(url);
            HtmlNodeCollection linkNodes = doc.DocumentNode.SelectNodes("//h3/a");
            var baseUri = new Uri(url);
            foreach (var link in linkNodes)
            {
                string href = link.Attributes["href"].Value;
                bookLinks.Add(new Uri(baseUri, href).AbsoluteUri);
            }
            return bookLinks;
        }
        static List<Book> GetBookDetails(List<string> urls)
        {
            var books = new List<Book>();
            foreach (var url in urls)
            {
                HtmlDocument document = GetDocument(url);
                var titleXPath = "//h1";
                var priceXPath = "//div[contains(@class,\"product_main\")]/p[@class=\"price_color\"]";
                var book = new Book();
                book.Title = document.DocumentNode.SelectSingleNode(titleXPath).InnerText;
                book.Price = document.DocumentNode.SelectSingleNode(priceXPath).InnerText;
                books.Add(book);
            }
            return books;
        }
        static void exportToCSV(List<Book> books)
        {
            using (var writer = new StreamWriter("books.csv"))
            using (var csv = new CsvWriter(writer, CultureInfo.InvariantCulture))
            {
                csv.WriteRecords(books);
            }
        }
        static void Main(string[] args)
        {
            var bookLinks = GetBookLinks("http://books.toscrape.com/catalogue/category/books/mystery_3/index.html");
            Console.WriteLine("Found {0} links", bookLinks.Count);
            var books = GetBookDetails(bookLinks);
            exportToCSV(books);
        }
    }
}

Running dotnet run, you will create a CSV file with the extracted book details.

Scraping dynamic web pages with Selenium

When dealing with pages that load content dynamically using JavaScript, libraries like Html Agility Pack become inadequate since the required data isn't included in the initial HTTP GET response. For these cases, Selenium is a solid solution. Selenium allows for the automation of web browsers, mimicking user actions like button clicks and form filling, making it possible to retrieve data just like a normal user, essential for dealing with these modern pages.

Setting up Selenium

In addition to a browser itself, you will also need its respective web driver as well as the language bindings for use of its API with C# code. This will facilitate controlling a browser programmatically using a small executable file called a web driver and the instructions coded in your selected language. Web drivers can be downloaded from their official websites or via the WebDriverManager Nuget package, which greatly simplifies the setup process. With the latter method, you don't have to keep tabs on the browser versions, update binaries or worry about environment variables.

For installation using Visual Studio, go to the NuGet Package Manager and install Selenium.Webdriver and WebDriverManager, then move onto Program.cs.

If you are using VSCode, use these commands in the terminal:

dotnet add package WebDriverManager --version 2.17.1
dotnet add package Selenium.WebDriver --version 4.15.0

Once installed, the code is ready to be used with selenium.

Loading a webpage

Modify Program.cs and include these libraries for use.

using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using WebDriverManager;
using WebDriverManager.DriverConfigs.Impl;

Next, set up the browser using the following method.

static void Main(string[] args)
{
    // setting up the driver
    new DriverManager().SetUpDriver(new ChromeConfig());
    var driver = new ChromeDriver();
    // load the webpage
    driver.Navigate().GoToUrl("http://quotes.toscrape.com/js/");
    // scraping code here
    // closing the browser 
    driver.Quit();
}

This basic skeleton code initiates the Chrome browser, navigates to the specified URL and quits it once completed.

Class to hold the data

For best practices in our code, create a class to hold the extracted data of our web page. As an example, our target web page will extract quotes, so let's name the class accordingly.

public class Quote
{
    public string? Text { get; set; }
    public string? Author { get; set; }
    public override string ToString()
    {
        return Author + " says, " + Text;
    }
}

The ToString override is purely for formatting and readable outputs in the console.

Locating the elements

Selenium enables elements on a web page to be found using CSS selectors and XPath. These selectors can be targeted using methods like driver.FindElement(), which will return a single object, or driver.FindElements(), which can return a collection of them.

Here's an example of locating an h1 tag using the code:

element = driver.FindElement(By.CssSelector("h1"));
Console.WriteLine(element.Text);

For our scraping case, each quote container can be found using the following:

var quoteContainers = driver.FindElements(By.CssSelector("div.quote"));

Individual quote elements inside each quote container will be scraped using this loop:

foreach (var item in quoteContainers)
{
    Console.WriteLine(item.FindElement(By.CssSelector(".author")).Text);
}

Finally, update the above logic to use our data class as so.

Quote quote = new()
{
    Text = item.FindElement(By.CssSelector("span.text")).Text,
    Author = item.FindElement(By.CssSelector(".author")).Text
};
Console.WriteLine(quote.ToString());

Exporting data to CSV

Using CsvHelper, implement a way to extract the above data into a CSV. This will involve the installation of CsvHelper Nuget package, as described in the beginning of the guide. First, start with a list variable to hold the quotes. Then, add all the extracted data into that list and trigger the write function at the end of all scrapes.

var quotes = new List<Quote>();

//add each quote as you scrape to the quotes list above
quotes.Add(quote)

//outside the for loop
using (var writer = new StreamWriter("~/quotes.csv"))
using (var csv = new CsvWriter(writer, CultureInfo.InvariantCulture))
{
    csv.WriteRecords(quotes);
}

Complete Code

Below is the finalized code using everything in this section for web scraping dynamic pages using selenium.

using System.Globalization;
using CsvHelper;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using WebDriverManager;
using WebDriverManager.DriverConfigs.Impl;

namespace webscraping
{
    public class Quote
    {
        public string? Text { get; set; }
        public string? Author { get; set; }
        public override string ToString()
        {
            return Author + " says, " + Text;
        }
    }
    class Program
    {
        static void Main(string[] args)
        {
            new DriverManager().SetUpDriver(new ChromeConfig());
            var driver = new ChromeDriver();
            driver.Navigate().GoToUrl("http://quotes.toscrape.com/js/");
            var quotes = new List<Quote>();
            var quoteContainers = driver.FindElements(By.CssSelector("div.quote"));
            foreach (var item in quoteContainers)
            {
                Quote quote = new()
                {
                    Text = item.FindElement(By.CssSelector("span.text")).Text,
                    Author = item.FindElement(By.CssSelector(".author")).Text
                };
                quotes.Add(quote);
                Console.WriteLine(quote.ToString());
            }
            using (var writer = new StreamWriter("./quotes.csv"))
            using (var csv = new CsvWriter(writer, CultureInfo.InvariantCulture))
            {
                csv.WriteRecords(quotes);
            }
        		// Close the driver
        		driver.Quit();
        }
    }
}

Run this code and you will generate the quotes.csv file.

Conclusion

C# offers multiple excellent libraries and tools to complete web scraping projects. This guide provides insight into Html Agility Pack for use with static content and Selenium for use with dynamic pages. These two tools provide coverage for a range of web pages out there on the web. It is advised to modify the provided code for use with multiple pages or modify the logic with specific project goals.

When developing your own scraping tools, it's important to use ethical practices and avoid aggressive scraping behaviors that could strain the website servers. That's where ProxyTee comes into play. ProxyTee offers unlimited residential proxies with automatic IP rotation that help make your scrapers look like regular user traffic to avoid getting blocked. You'll have access to over 20 million IPs in 100+ locations that also allows you to target specific geo locations with your project. With a simple API and easy to use GUI, setting it up is easy and fast for any user. Plus ProxyTee offers plans up to 50% cheaper than the competition with all these features included, including unlimited bandwidth.

People also ask

Is C# good for web scraping?

C# is commonly used in web scraping, similar to Python. If you're picking a language for a scraping project, going with the one that you're more familiar with can be a good place to start, but with a large amount of helpful guides available, it is easy to pick either one.

Is web scraping legal?

Web scraping can be permissible if all laws are followed. Be aware that if proxies are used for your project, seek out professional legal advice regarding the activity that it is being applied to.

Which language is best for web scraping?

Many programming languages are capable of web scraping, but the one to choose depends on various factors, like proficiency of the user, the frameworks and the ease and flexibility offered with each language. Some of the more popular choices for web scraping are Python, C#, Node.js, Ruby and Rust.

Can web scraping be detected?

Websites can definitely detect the activity of scrapers with indicators such as browser parameters, IP addresses, user behaviors and other details. To stay undetectable, employ practices like IP rotations with a service like ProxyTee and customize request headers.

What is the best C# web scraping library?

Some top C# libraries for scraping are Selenium, HttpClient, Scrapy Sharp, and Puppeteer Sharp.