Web Scraping with Kotlin: A Practical Guide Using ProxyTee

Kotlin, with its concise syntax and JVM interoperability, has emerged as a strong contender for web scraping. This guide will walk you through how to use Kotlin for effective web scraping, enhanced with the power and reliability of ProxyTee. You’ll learn how to set up your environment, explore suitable libraries, and create a practical scraping script.

Why Choose Kotlin for Web Scraping?

Kotlin is a statically-typed, cross-platform language known for its conciseness and safety. It is also fully interoperable with Java and its extensive ecosystem of libraries. This interoperability makes it particularly attractive for web scraping, allowing developers to leverage mature Java libraries with the benefit of Kotlin’s more expressive syntax. With its adoption by Google for Android development, Kotlin’s future in various software domains, including web scraping, is bright.

When you choose Kotlin for web scraping and pair it with ProxyTee, you benefit from a high-performance, low-cost solution. ProxyTee offers rotating residential proxies, crucial for avoiding detection and blocks from target websites, alongside features such as unlimited bandwidth, and a vast IP pool for truly global data collection. This allows you to focus on the data without getting caught in anti-scraping measures. We ensure high availability and optimal performance for all your web scraping needs.

Kotlin Web Scraping Libraries

Here are a few of the most popular Kotlin libraries that help you with your web scraping journey:

• skrape{it}: A Kotlin library specifically designed for HTML/XML parsing, and can even work as a headless browser, rendering Javascript pages as required.
• chrome-reactive-kotlin: This is useful for programmatically controlling Chromium-based browsers.
• ksoup: Inspired by Jsoup, this lightweight library can be used for parsing HTML and extracting tags and attributes.

Since Kotlin can be integrated with Java, you have access to the myriad of options available for the Java environment as well, including the popular Jsoup library, if required. This opens up more potential options and gives the developer greater freedom.

Setting up your Kotlin Environment

Let’s get our hands dirty and set up the needed environment!

1. Install JDK: You need to install the Java Development Kit to get Kotlin working correctly. Download the latest version, or check if there are other specific instructions from the Java side, install the Java and move to the next step.
2. Choose your build tool: There are some good options for a good build tool, but you should choose either Maven or Gradle to set up your dependencies. Since Gradle has first class support for Kotlin we will use this one. You need a version greater than 8.5 if your system runs Java 21. Download Gradle here, but the version must match with the Java version, check the docs before proceed. If you are on Maven this guide might still help you out if you make some minor adjustments.
3. Pick an IDE: Use a good Kotlin editor, like Visual Studio Code with the Kotlin language extension or IntelliJ IDEA.

Your environment is ready!

Create a Kotlin Project

Use your command prompt to run the following commands to generate the initial Kotlin project structure:

mkdir ProxyTeeKotlinScraper
cd ProxyTeeKotlinScraper

gradle init --type kotlin-application

When prompted, select Kotlin for the DSL and give a suitable name to your package, like com.proxytee.scraper. Choose defaults for the rest of the questions.

Open the directory on your chosen IDE and look for the App.kt in your main directory.

Build a Web Scraper

Now let’s create our scraping application! Our goal is to extract the information from this website.

Step 1: Setup the scraping library

For simplicity, we will use the skrape{it} library.

Add the dependency on the build.gradle.kts:

implementation("it.skrape:skrapeit:1.2.2")

After adding the new dependency, we should add also the following imports at the top of our App.kt file:

import it.skrape.core.*
import it.skrape.fetcher.*

These imports provide all of the data fetching capabilities we’ll need to get started with this guide.

Step 2: Download the page

Now add the following code to your main function:

skrape(HttpFetcher) {
    request {
        url = "https://quotes.toscrape.com/"
    }

   response {
       htmlDocument {
           print(html)
       }
    }
}

This is what this script does: calls the url defined on the url attribute with the default http get method and gets the full document in the body. In our code the HTML data gets printed, if your IDE runs the application you will be able to verify the page contents from its output, this confirms the request works as expected.

Step 3: Inspect the page Structure

When we’re working with web scrapers we should always make sure that our selectors are valid and also keep the code tidy, therefore a page inspection would be needed to retrieve the HTML class names and attributes used in the data of interest, open your DevTools panel on any modern browser, right click on one of the quote cards, inspect it and try to understand it’s structure to get the required CSS selectors. Our quotes data will be within the .quote, the quote text inside the .text class and the author on the .author class. The tag’s are nested in the .tag class inside a container div. After this step, let’s create a Quote object and make a list of it where all data can be stored. On top of our file add the class definition:

class Quote(var text: String, var author: String, tags: List<String>?) {
    var tags: MutableList<String> = ArrayList()
    init {
        if (tags != null) {
            this.tags.addAll(tags)
        }
    }
}

Inside the main function add this:

val quotes: MutableList<Quote> = ArrayList()

This will hold our scraped data from the HTML that’s been returned.

Step 4: Implement Scraping

Use the class selector .quote from the skrape{it} api to filter the wanted elements inside the HTML:

skrape(HttpFetcher) {
    request {
        url = "https://quotes.toscrape.com/"
    }
    response {
        htmlDocument {
            ".quote" {
                findAll {
                    forEach { quoteElement ->
                        val text = quoteElement.findFirst(".text").text
                        val author = quoteElement.findFirst(".author").text

                        val tags = try {
                            quoteElement.findAll(".tag").map { tagElement -> tagElement.text }
                        } catch (e: it.skrape.selects.ElementNotFoundException) {
                            null
                        }
                        val quote = Quote(
                            text = text,
                            author = author,
                            tags = tags
                        )
                        quotes.add(quote)
                    }
                }
            }
        }
    }
}

This section goes through each .quote node, extract the text, author, and tags, creates a quote object and stores the resulting object in the quote variable. It does all the needed error handling with try catch expressions as tags may not be always present.

Also import the exception

import it.skrape.selects.ElementNotFoundException

Now, after executing this script, we are ready to print the result:

for (quote in quotes) {
        println("Text: ${quote.text}")
        println("Author: ${quote.author}")
        println("Tags: ${quote.tags.joinToString("; ")}")
        println()
 }

Step 5: Add the crawling Logic

Our scraping application now works only with the first page. Let's make a modification to make our script go through all of the available pages. Modify your main block to resemble the code below:

var nextUrl: String? = "https://quotes.toscrape.com/"
while (nextUrl != null) {
        skrape(HttpFetcher) {
           request {
               url = nextUrl!!
           }
           response {
              htmlDocument {
                   ".quote" {
                      findAll {
                         forEach { quoteElement ->
                                  val text = quoteElement.findFirst(".text").text
                                  val author = quoteElement.findFirst(".author").text
                                  val tags = try {
                                      quoteElement.findAll(".tag").map { tagElement -> tagElement.text }
                                  } catch (e: ElementNotFoundException) {
                                      null
                                  }
                                  val quote = Quote(
                                      text = text,
                                      author = author,
                                      tags = tags
                                 )
                             quotes.add(quote)
                          }
                      }
                    }
                try {
                       ".next a" {
                         findFirst {
                                nextUrl = "https://quotes.toscrape.com" + attribute("href")
                           }
                     }
                   } catch (e: ElementNotFoundException) {
                       nextUrl = null
                   }
              }
          }
      }
   }

We wrapped our code inside a while loop that continues until a valid next element can’t be found in the dom. When this situation appears our application terminates by setting the nextUrl variable to null, triggering the while loop exit.

Step 6: Export to CSV

Kotlin doesn’t provide anything out of the box to create CSV’s but fortunately some open source libraries can be used for this job. Let’s include this dependency to make our job easier:

implementation("com.github.doyaaaaaken:kotlin-csv-jvm:1.9.3")

Remember to also include the import directive

import com.github.doyaaaaaken.kotlincsv.dsl.*

And the data exporting part:

val header = listOf("quote", "author", "tags")
val csvContent: List<List<String>> = quotes.map { quote ->
    listOf(
        quote.text,
        quote.author,
        quote.tags.joinToString("; ")
    )
}
csvWriter().open("quotes.csv") {
    writeRow(header)
    writeRows(csvContent)
}

The above snippet does the following: declares our column header values as a List, transforms each of our quote objects to string values to match our columns, initiates the CSV write process, generates the header and adds all records into the csv.

Step 7: Full code

Our full code must be similar to this:

package com.proxytee.scraper

import it.skrape.core.*
import it.skrape.fetcher.*
import it.skrape.selects.ElementNotFoundException
import com.github.doyaaaaaken.kotlincsv.dsl.*

// define a class to represent the scraped data in Kotlin
class Quote(var text: String, var author: String, tags: List<String>?) {
    var tags: MutableList<String> = ArrayList()
    init {
        if (tags != null) {
            this.tags.addAll(tags)
        }
    }
}

fun main() {
    // where to store the scraped data
    val quotes: MutableList<Quote> = ArrayList()

   // the URL of the next page to visit
    var nextUrl: String? = "https://quotes.toscrape.com/"

    // until there is a page to visit
   while (nextUrl != null) {
        skrape(HttpFetcher) {
            // make an HTTP GET request to the specified URL
            request {
               url = nextUrl!!
           }
            response {
               htmlDocument {
                    // select all ".quote" HTML elements on the page
                   ".quote" {
                       findAll {
                            forEach { quoteElement ->
                                 // scraping logic on a single quote element
                                  val text = quoteElement.findFirst(".text").text
                                   val author = quoteElement.findFirst(".author").text
                                    val tags = try {
                                      quoteElement.findAll(".tag").map { tag -> tag.text }
                                    } catch (e: ElementNotFoundException) {
                                          null
                                    }
                                   // create a Quote object and add it to the list
                                   val quote = Quote(
                                      text = text,
                                      author = author,
                                     tags = tags
                                  )
                                quotes.add(quote)
                            }
                       }
                    }
                  // crawling logic
                  try {
                         ".next a" {
                            findFirst {
                              nextUrl = "https://quotes.toscrape.com" + attribute("href")
                         }
                       }
                  } catch (e: ElementNotFoundException) {
                        nextUrl = null
                  }
              }
            }
        }
   }
   // create a "quotes.csv" file and populate it
   // with the scraped data
  val header = listOf("quote", "author", "tags")
  val csvContent: List<List<String>> = quotes.map { quote ->
    listOf(
          quote.text,
         quote.author,
           quote.tags.joinToString("; ")
        )
   }
   csvWriter().open("quotes.csv") {
     writeRow(header)
    writeRows(csvContent)
  }
}

When you execute your script with

./gradlew run

a CSV file will be generated after all of the crawling work is completed. This makes it super easy for other team members or programs to perform additional processing.

Avoid IP Bans with ProxyTee

To fully complete our guide, you must know how to make sure your scraper runs as smoothly as possible without causing harm or issues, as anti bot measures are one of the biggest headaches for anyone looking to extract public data, here enters our trusted and robust friend: ProxyTee. With ProxyTee you don't have to worry about complex mechanisms for rotating ips, all happens automatically by enabling your subscription on our page.

ProxyTee offers reliable residential proxies to enhance web scraping tasks by providing:

• Unlimited Bandwidth: No limits on the amount of data that can be processed
• Global Coverage: Over 20 million IPs from various geographic locations.
• Multiple Proxy Protocols: Support for both HTTP and SOCKS5.
• Auto-Rotation: Automatically rotate IP address at predefined intervals between 3 to 60 minutes.

Integrate ProxyTee in Kotlin

Here is how you should be integrating ProxyTee in our application:

skrape(HttpFetcher) {
    request {
        url = "https://quotes.toscrape.com/"
       proxy = proxyBuilder {
            type = Proxy.Type.HTTP
           host = "proxy.proxytee.com"
           port = 12345 // Replace with the correct port
      }
      authentication = basic {
          username = "your_username" // Replace with your username
          password = "your_password" // Replace with your password
      }
   }
   // ... rest of your code
}

Use the proxy, the basic authentication properties and you can run your program with ProxyTee configured with a few lines of code.

Ethical Web Scraping

Ethical scraping is something we must be aware of when working in this space:

• Scrape only public data: Make sure the target pages don’t contain any kind of private data or something protected with authentication mechanisms.
• Check robots.txt: Adhere to guidelines provided on the target webiste’s robots.txt file.
• Limit your request frequency: Too much load can crash servers. Make sure you send the requests accordingly with a rate limiter mechanism, so you don't flood their services and affect other users.
• Be aware of the Terms of service: Always verify what the TOS says, if you are unsure seek additional information.
• Use reputable scraping tools: Keep in mind if tools you choose, get the needed maintenance and they work with latest security and ethical standards.

Conclusion

Kotlin’s versatility and ease of use makes it an excellent option for web scraping tasks. With the integration of a reliable proxy service such as ProxyTee, you can carry out complex tasks more easily. We have shown how you can utilize these technologies together, making sure data collection operations are done efficiently and ethically.

ProxyTee offers multiple products that can be very helpful for this type of operation:

• Residential Proxies: Offers millions of real IPs for web scraping. The main feature of our product is to have unlimited bandwidth, and a huge ip pool that provides worldwide country level coverage, unlike other providers which might not have enough resources, or have their IPs in a random way or even on a continent scope. Also our products can be priced as low as 50% less compared to other vendors in the space.
• Unlimited Residential Proxies: It has all of our Residential Proxies but with no usage restriction.
• Datacenter Proxies: Provides faster and stable scraping experience.
• Static Residential Proxies: For tasks that require a fixed IP. You will get the IP’s you paid without changes.
• Mobile Proxies: Our Mobile proxy products are the best if you want to access data from mobile devices and applications.

Feel free to look into our pricing options and our other use-cases for more detailed scenarios. Don't forget to compare us to other solutions on the market.