Paging Through the Web: Basic and Advanced Selenium Techniques in Java for Handling Pagination in Various Scenarios-Part 2

Hope you have found the part 1 of this blog informative. This blog is the continuation of my previous blog, where we talked about what is Pagination, why is it useful and how we can automate the pagination logic to scrape or retrieve large quantity of data in fraction of seconds.

Here is the link to part 1 of this blog, where I have shown two different scenarios, implemented pagination logic using selenium java, to scrape large data.

Part 1 Link: https://www.numpyninja.com/post/paging-through-the-web-basic-and-advanced-selenium-techniques-in-java-for-handling-pagination-in-va

In this blog Let’s dig little deeper and implement pagination logic by making use of While loop and further optimized and reduced the code to one line using Java Stream.

Scenario#3 When there is a next button or forward page icon is present in the pagination body and page number keep changing until forward icon or next button is not clickable

Sometimes when we are not sure how many pages we need to iterate, in that case we cannot use for loop. That time we have to make use of while loop, and based on condition we need to stop the iteration. So here we will iterate until next button or forward Page icon is disabled. See below,

Or

The approach to handle pagination will be the same for both the pagination tab shown above. I’ll show how we can develop pagination logic for example#1 using while loop.

Step1. Navigate to https://www.audible.com/

Step2: Here I want to grab all the book names related to Harry Potter displayed on each page. There are total of 415 items when I type Harry potter in the search box of home page.

Code snippet:

Step3: Get the web Element for forward Page icon and Total item count (total books displayed in all pages=415). The search result shows 1-21 pages and a backward and forward Page icon.

(Note: sometimes all the page numbers are not displayed and it get changed until the forward page icon is disabled, in this case we cannot make use of ‘for loop’ since we do not know how many times it will execute)

Step4: Store all the books names on each page in a List. Create an empty array List to add the book names. Using advanced for loop iterate through list to get the book name using b.getText and add to empty bookNames arrayList.This piece of code will retrieve all the book names from page 1.

To move to next page and perform the same actions we will use while loop.

Step 5: We will use while loop to iterate through remaining pages until the forward page icon is disabled. Once it reaches the last page, the condition will be met ,it will come out of loop and execution stops.

Step6: Also we need to  handle exception not found using try and catch block for forwardPage icon WebElement when it’s no more clickable  on last page.

Step7: We can display and compare the results of total book count displayed vs retrieved through code.

Scenario#4 When there is a next button or forward page icon is present in the pagination body and page number keep changing until forward icon or next button is not clickable(Using Java Stream)

Now I will show you how pagination logic can be implemented for the (above scenario#3) using Java Stream. Introduced in Java 8, Java Stream is a powerful and expressive way to process collections of data in a functional programming style. It's operated on collections of objects like List or an array. It helps us to perform operations like filtering, mapping, reducing and sorting.

Step1: Follow step 1,2 and 3 above in Scenario#3

Step2: Store all the books names on each page in a List. Instead of creating another ArrayList to store bookNames and iterate it using advanced for loop , we can simply reduced the number of coding steps to one line using JavaStream (as you can see in code snippet below and compare it with the code in scenario#3).

Code explanation: JavaStream works on list. So first we need to create a List of type <WebElement> BookList to locate and get all the books names. Using Lambda expression stream().map(b-> b.getText()) will get all text (i.e. book names). collect(Collectors.toList()) will collect these book names in new list(bookNames) of type <string>. bookNames.forEach(b->System.out.println(b)) will print each book names in the current page.

Step3: Now to iterate through other pages, we will again use While loop and iterate until forward Page Icon is disabled. Instead of using advanced for loop we can again make use of JavaStream.

Step4: Handle the not found exceptions in try catch block for Forward Page Icon.

Output:

Conclusion:

Testing on websites featuring paginated content becomes remarkably efficient with the automation capabilities of Selenium in Java. By automating pagination user can easily navigate through different pages in a fraction of seconds. They are useful in searching or scraping data on ecommerce websites, forum and discussion board, job portal and many more.

I’ve tried to explained the fundamental concept of pagination and demonstrated how to automate using Selenium in Java. We have covered various scenarios, including basic pagination with static page number, “Next" or ‘>’ buttons to dynamic loading page number and optimized & reduced the code using JavaStream.

With this we came to an end of this blog, I hope by now you have fair understanding about What is pagination, how to automate it using selenium in Java in various situations to streamlined the testing process, making it easier to navigate through pages, retrieve or scrape data as per the requirement.

Happy Automating!!!!