How to get network call requests in Selenium?

Capturing network requests is essential in understanding how a website works and what data is transmitted. Selenium, being a web automation tool, can be used to capture network requests. This article will discuss capturing network requests and retrieving data from a website using Selenium in Java.

We can easily capture network requests or logs using the browser developer tools. Most modern web browsers have built-in developer tools that allow us to view and capture network requests. We can access the developer tools by right-clicking on the webpage and selecting the “inspect” option.

Clicking on the “Inspect” option will open the developer tools, where there are a bunch of tabs. We can click the “Network” tab to capture the network calls.

Now, we will capture the same network calls using Selenium.

Capturing network call requests using Selenium

Selenium 4 introduced a new feature using which we can access the developer tools, and it becomes very easy to get the network calls or console logs and many other things. That feature is “DevTools“

So, let’s see how we can get the network calls using DevTools in Selenium.

Note: We are using Selenium 4.7.0 version with chrome version 109.

Create a session with DevTools

		DevTools devTools = ((ChromeDriver) driver).getDevTools();
		devTools.createSession();

Enable Network tracking

devTools.send(Network.enable(Optional.of(1000000), Optional.empty(), Optional.empty()));

Network.enable method enables the network tracking. It accepts three parameters – maxTotalBufferSize, maxResourceBufferSize and maxPostDataSize.

All the parameters are to be sent in Bytes.

Add listener

We can add the listener to capture the network requests.

devTools.addListener(Network.requestWillBeSent(), request -> { });

Now, we can access the network requests and get the HTTP methods, URLs, or request body of the requests.

Q – How to get the HTTP methods of the network requests?

We can use the getRequest() to get a Request instance and then the getMethod() function to get the HTTP method for the specific request.

devTools.addListener(Network.requestWillBeSent(), request -> { 
   System.out.println("Request Method : " + request.getRequest().getMethod());
});

Q – How to get the request URLs of the network requests?

getUrl() method returns the request URL of a network call

devTools.addListener(Network.requestWillBeSent(), request -> { 
   System.out.println("Request URL : " + request.getRequest().getUrl());
});

Q – How to get the request headers of the network requests?

getHeaders() can be used to get the headers of a request.

devTools.addListener(Network.requestWillBeSent(), request -> { 
     System.out.println("Request headers: "+request.getRequest().getHeaders().toString());
});

Q – How to get the request payload of the network requests?

getPostData() returns the request payload as Optional<String>.

devTools.addListener(Network.requestWillBeSent(), request -> { 
      System.out.println("Request body: "+request.getRequest().getPostData().toString());
});

Whole Code

Now, let’s write the whole code fetching the details mentioned for the URL – “https://www.makemytrip.com”.

import java.util.Optional;

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.devtools.DevTools;
import org.openqa.selenium.devtools.v108.network.Network;

public class CodekruTest {

	public static void main(String[] args) {

		// pass the path of the chromedriver location in the second argument
		System.setProperty("webdriver.chrome.driver", "E:\\chromedriver.exe");
		WebDriver driver = new ChromeDriver();

		DevTools devTools = ((ChromeDriver) driver).getDevTools();
		devTools.createSession();
		devTools.send(Network.enable(Optional.of(1000000), Optional.empty(), Optional.empty()));
		devTools.addListener(Network.requestWillBeSent(), request -> {
			System.out.println("Request Method : " + request.getRequest().getMethod());
			System.out.println("Request URL : " + request.getRequest().getUrl());
			System.out.println("Request headers: " + request.getRequest().getHeaders().toString());
			System.out.println("Request body: " + request.getRequest().getPostData().toString());
		});

		driver.get("https://www.makemytrip.com");

	}

}

We have highlighted line 6, which shows v108. We have used v108 as it is compatible with our chrome version 109. Please use the version which is compatible with your chrome version. Otherwise, you might face errors while executing automation.

We hope that you have liked the article. If you have any doubts or concerns, please write to us in the comments or mail us at admin@codekru.com.

Related Articles –

• How to get network call responses in Selenium?
• Network throttling in Chrome with Selenium
• How to capture the console logs in Selenium Java?