Pagination using Selenium Python

PAGINATION : Pagination is basically navigating from one page to another page.

Till now I have gone through 2 types of Pagination. I have used this pagination to extract data from different pages. I will show you by taking an example from two websites.

Lets start with 1st website and website link is : https://covidtelangana.com

In this website next page button is like following which is in text format.

Steps to navigate each Page is mentioned below:

1) First we need to inspect above Green color Load next 20 button by right clicking and then inspect. After inspecting we can find element by using any locator like xpath, id, class. So code should be like following:

Button=driver.find_element_by_xpath('/html[1]/body[1]/div[1]/div[1]/div[1]/div[2]/div[3]/div[1]/button[1]')

2) In while loop we can use Try and except statement like below:

while(True):
    try:
        Button. Click()
    except:
        break

Basically in above code while loop will keep on running as long as the button is present and clickable. If button is not found It will go to exception and come out of the loop.

Now I want to introduce you with 2nd type of pagination and website I used is: https://www.glassdoor.com/Job/index.htm

In this website next page button format is like below:

Steps to navigate each page:

1) First step is to get count of pages which we can extract by following methods:

# Extracting text of string "Page 1 of 30"
Element = driver.find_element_by_xpath("//[@id='MainCol']/div[2]/div[1]").text
print(Element)

# finding index of "of" from string "Page 1 of 30" 
a = Element. Index('of')

# using substring method finding number "30" which is total pages count
s = Element[a+3:]
print(s)

# convert string 30 into data type integer
total_pages = int(s)

Above code shows that, to get number of pages count from the string which is given, first we need to use index method to find index of adjacent elements. Note in above text which is Page 1 of 30 adjacent elements of 30 is "of" but if the text will be like "Page 1 of 30 Pages" then we have to find index of adjacent elements like "of" and "pages" both. Then using substring method in python we need to find element which is pages count as we are doing in above code. Finally our text 30 will come as string so to convert into integer we are using int data type method of python.

2) Now to click on each and every page we will use for loop.

for p in range (total_pages):
    driver.find_element_by_xpath("//ul//li//a[@data-test='pagination-next']").click()
    time.sleep(3)

In the above code, I am finding first the next page button using xpath and then clicking on it inside a for loop. After that giving a sleep time of 3 seconds to give some time in case any pop ups come after clicking the button.