A network call is a request made to a server to retrieve or send data. When a user interacts with a website, various network calls are made in the background to retrieve information from the server. For example, when a user clicks on a button, a network call is made to retrieve the data associated with that button. So, it is important to know what requests were made and the responses that we got for the same.
We already discussed getting the network call requests using Selenium, and this post will focus on getting the network call responses.
Note: We have also attached a video at the end of this article. In case you are more comfortable with the video version, then please feel free to have a look.
We can capture the network calls using the browser developer tools. Most modern web browsers have built-in developer tools that allow us to view and capture network calls.
We can access the developer tools by right-clicking on the webpage and selecting the “inspect” option.
Clicking on the “Inspect” option will open the developer tools, where there are a bunch of tabs. We can click the “Network” tab to capture the network calls.
Capturing network call responses using Selenium
Now, let’s see how we can get the network call responses using selenium.
Selenium 4 has introduced a new feature called DevTools, which allows us to interact with the browser developer tools. We will also use the same to get the network call responses.
Note: In our code, you’ll see an import “org.openqa.selenium.devtools.v108.network.Network” showing a version of 108. This is because we are currently using chrome version 109, and v108 import is compatible with it. Please use the version that is compatible with your chrome browser.
Create a session with DevTools
DevTools devTools = ((ChromeDriver) driver).getDevTools();devTools.createSession();
Enable Network tracking
devTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));
Network.enable method enables the network tracking. It accepts three parameters – maxTotalBufferSize, maxResourceBufferSize and maxPostDataSize.
All the parameters are to be sent in Bytes.
Capturing response information of a network call
We can easily capture the response of network calls using the below code –
devTools.addListener(Network.responseReceived(), responseReceived -> {
}
Network.responseReceived is fired whenever an HTTP response is received, and the value will be stored in the responseReceived variable. We can then use the stored value to get the network call’s response headers, response status, or response body.
responseReceived.getResponse() will return a Response object which provides many helpful methods to get information about an HTTP response.
How to get the response headers of a Network call using Selenium?
responseReceived.getResponse().getHeaders() gets the headers of a response.
devTools.addListener(Network.responseReceived(), responseReceived -> {
System.out.println("Response headers: " + responseReceived.getResponse().getHeaders().toString());
});
Whole Code
import java.util.Optional;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.devtools.DevTools;
import org.openqa.selenium.devtools.v108.network.Network;
public class CodekruTest {
public static void main(String[] args) {
// pass the path of the chromedriver location in the second argument
System.setProperty("webdriver.chrome.driver", "E:\\chromedriver.exe");
WebDriver driver = new ChromeDriver();
DevTools devTools = ((ChromeDriver) driver).getDevTools();
devTools.createSession();
devTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));
devTools.addListener(Network.responseReceived(), responseReceived -> {
String responseUrl = responseReceived.getResponse().getUrl();
if (responseUrl.contains("makemytrip")) {
System.out.println("Url: " + responseUrl);
System.out.println("Response headers: " + responseReceived.getResponse().getHeaders().toString());
}
});
driver.get("https://www.makemytrip.com");
}
}
How to get the response status of a Network call using Selenium?
responseReceived.getResponse().getStatus() gets the response status of an HTTP response.
devTools.addListener(Network.responseReceived(), responseReceived -> {
System.out.println("Response status: " + responseReceived.getResponse().getStatus());
});
Whole Code
import java.util.Optional;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.devtools.DevTools;
import org.openqa.selenium.devtools.v108.network.Network;
public class CodekruTest {
public static void main(String[] args) {
// pass the path of the chromedriver location in the second argument
System.setProperty("webdriver.chrome.driver", "E:\\chromedriver.exe");
WebDriver driver = new ChromeDriver();
DevTools devTools = ((ChromeDriver) driver).getDevTools();
devTools.createSession();
devTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));
devTools.addListener(Network.responseReceived(), responseReceived -> {
String responseUrl = responseReceived.getResponse().getUrl();
if (responseUrl.contains("makemytrip")) {
System.out.println("Url: " + responseUrl);
System.out.println("Response status: " + responseReceived.getResponse().getStatus());
}
});
driver.get("https://www.makemytrip.com");
}
}
How to get the response body of a Network call using Selenium?
It is a two-step process –
- First, get the request id of a network call using
responseReceived.getRequestId()method. - And then, we can use the request id to get the response body of a network call by using
devTools.send(Network.getResponseBody(requestId)).getBody().
devTools.addListener(Network.responseReceived(), responseReceived -> {
RequestId requestId = responseReceived.getRequestId();
System.out.println("Response body: "+ devTools.send(Network.getResponseBody(requestId)).getBody());
});
Whole Code
import java.util.Optional;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.devtools.DevTools;
import org.openqa.selenium.devtools.v108.network.Network;
import org.openqa.selenium.devtools.v108.network.model.RequestId;
public class CodekruTest {
public static void main(String[] args) {
// pass the path of the chromedriver location in the second argument
System.setProperty("webdriver.chrome.driver", "E:\\chromedriver.exe");
WebDriver driver = new ChromeDriver();
DevTools devTools = ((ChromeDriver) driver).getDevTools();
devTools.createSession();
devTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));
devTools.addListener(Network.responseReceived(), responseReceived -> {
String responseUrl = responseReceived.getResponse().getUrl();
RequestId requestId = responseReceived.getRequestId();
if (responseUrl.contains("makemytrip")) {
System.out.println("Url: " + responseUrl);
System.out.println("Response body: " + devTools.send(Network.getResponseBody(requestId)).getBody());
}
});
driver.get("https://www.makemytrip.com");
}
}
How to get the response time of a Network call using Selenium?
To determine the response time for a call, we must monitor and record two events.
- requestWillBeSent event – This event is triggered when the page is about to send an HTTP request. It will help us get the start time of a request.
- loadingFinished event – The “loadingFinished” event is triggered once the HTTP request has completed loading. This event can be used to determine the time at which the request was finished.
Getting the start time of a request
In order to keep track of requests, we will utilize two maps. The first map will store the starting time of each request, while the second map will store the corresponding URL string, with the requestId serving as the key for both.
requestWillBeSent.getRequestId()will get us the requestId of the network requestrequestWillBeSent.getTimestamp().toJson().doubleValue()will get us the start time of the request.requestWillBeSent.getRequest().getUrl()will get us the request URL.
Map<String, Double> timingMap = new HashMap<>();
Map<String, String> requestIdMap = new HashMap<>();
devTools.addListener(Network.requestWillBeSent(), requestWillBeSent -> {
RequestId requestId = requestWillBeSent.getRequestId();
double startTime = requestWillBeSent.getTimestamp().toJson().doubleValue();
String requestUrl = requestWillBeSent.getRequest().getUrl();
timingMap.put(requestId.toString(), startTime);
requestIdMap.put(requestId.toString(), requestUrl);
});
Getting the end time of a request
loadingFinshed.getRequestId()will get us the requestId of the network requestloadingFinshed.getTimestamp().toJson().doubleValue()will provide us with the completion time of the request, indicating when it was finished.
devTools.addListener(Network.loadingFinished(), loadingFinshed -> {
RequestId requestId = loadingFinshed.getRequestId();
double endTime = loadingFinshed.getTimestamp().toJson().doubleValue();
});
Getting response time of every call ( end time – start time )
The response time of a network call can be determined by calculating the difference between the end time and the start time.
devTools.addListener(Network.loadingFinished(), loadingFinshed -> {
RequestId requestId = loadingFinshed.getRequestId();
double endTime = loadingFinshed.getTimestamp().toJson().doubleValue();
if (timingMap.get(requestId.toString()) != null) {
String url = requestIdMap.get(requestId.toString());
double responseTime = endTime - timingMap.get(requestId.toString());
System.out.println("Url: " + url + " , response time: "+ responseTime + " seconds");
}
});
And below is our whole code.
import java.util.HashMap;
import java.util.Map;
import java.util.Optional;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.devtools.DevTools;
import org.openqa.selenium.devtools.v109.network.Network;
import org.openqa.selenium.devtools.v109.network.model.RequestId;
public class CodekruTest {
public static void main(String[] args) {
// pass the path of the chromedriver location in the second argument
System.setProperty("webdriver.chrome.driver", "E:\\chromedriver.exe");
WebDriver driver = new ChromeDriver();
DevTools devTools = ((ChromeDriver) driver).getDevTools();
devTools.createSession();
devTools.send(Network.enable(Optional.of(1000000), Optional.empty(), Optional.empty()));
Map<String, Double> timingMap = new HashMap<>();
Map<String, String> requestIdMap = new HashMap<>();
devTools.addListener(Network.requestWillBeSent(), requestWillBeSent -> {
RequestId requestId = requestWillBeSent.getRequestId();
double startTime = requestWillBeSent.getTimestamp().toJson().doubleValue();
String requestUrl = requestWillBeSent.getRequest().getUrl();
timingMap.put(requestId.toString(), startTime);
requestIdMap.put(requestId.toString(), requestUrl);
});
devTools.addListener(Network.loadingFinished(), loadingFinshed -> {
RequestId requestId = loadingFinshed.getRequestId();
double endTime = loadingFinshed.getTimestamp().toJson().doubleValue();
if (timingMap.get(requestId.toString()) != null) {
String url = requestIdMap.get(requestId.toString());
double responseTime = endTime - timingMap.get(requestId.toString());
System.out.println("Url: " + url + " , response time: " + responseTime + " seconds");
}
});
driver.get("https://www.makemytrip.com");
}
}
The code above will provide you with the response time for each network call.
How to get network call responses after performing an operation
Sometimes, we may be required to monitor network calls after performing an operation like submitting a form or clicking on a button.
To simplify the process, we will assume that we only need the response body of network calls triggered by clicking a button. We will follow the below steps to get the network calls.
- Store network calls in a map with requestId’s as keys and response bodies as values.
Map<String, String> map = new HashMap<>();
devTools.addListener(Network.responseReceived(), responseReceived -> {
String responseUrl = responseReceived.getResponse().getUrl();
RequestId requestId = responseReceived.getRequestId();
if (responseUrl.contains("makemytrip")) {
String responseURL = responseUrl;
// put in map only if there is a response body
try {
String responseBody = devTools.send(Network.getResponseBody(requestId)).getBody();
map.put(requestId.toString(), responseBody);
} catch (Exception e) {
}
}
});
- Clone the map just before performing the click operation
// cloning the map before performing the operation
Map<String, String> beforeClickResponses = new HashMap<>();
beforeClickResponses.putAll(map);
- Then, perform the click operation and again store the actual map value in a third map.
Map<String, String> beforeClickResponses = new HashMap<>();
beforeClickResponses.putAll(map);
// write your code here
// to perform any operation like submitting a form
// Again cloning the map after performing operation
Map<String, String> afterClickResponses = new HashMap<>();
afterClickResponses.putAll(map);
- After clicking, remove keys from afterClickResponses that are also in beforeClickResponses map to get the final network calls made after the click operation.
afterClickResponses.keySet().removeAll(beforeClickResponses.keySet());
Note: We used the requestId as the key and the response body as the value, but depending on the test case requirement, you can choose anything else.
Whole code –
import java.util.HashMap;
import java.util.Map;
import java.util.Optional;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.devtools.DevTools;
import org.openqa.selenium.devtools.v114.network.Network;
import org.openqa.selenium.devtools.v114.network.model.RequestId;
public class CodekruTest {
public static void main(String[] args) throws InterruptedException {
// pass the path of the chromedriver location in the second argument
System.setProperty("webdriver.chrome.driver", "E:\\chromedriver.exe");
WebDriver driver = new ChromeDriver();
Map<String, String> map = new HashMap<>();
DevTools devTools = ((ChromeDriver) driver).getDevTools();
devTools.createSession();
devTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));
devTools.addListener(Network.responseReceived(), responseReceived -> {
String responseUrl = responseReceived.getResponse().getUrl();
RequestId requestId = responseReceived.getRequestId();
if (responseUrl.contains("demoqa")) {
String responseURL = responseUrl;
// put in map only if there is a response body
try {
String responseBody = devTools.send(Network.getResponseBody(requestId)).getBody();
map.put(requestId.toString(), responseBody);
} catch (Exception e) {
}
}
});
driver.get("https://www.makemytrip.com");
// cloning the map before performing the operation
Map<String, String> beforeClickResponses = new HashMap<>();
beforeClickResponses.putAll(map);
// write your code here
// to perform any operation like submitting a form
// Again cloning the map after performing operation
Map<String, String> afterClickResponses = new HashMap<>();
afterClickResponses.putAll(map);
// remove keys from "afterClickResponses" that are also present in
// "beforeClickResponses" map
afterClickResponses.keySet().removeAll(beforeClickResponses.keySet());
}
}
We hope that you have liked the article. You can also refer to the video below, which shows how we can capture network call responses. f you have any doubts or concerns, please write to us in the comments or mail us at admin@codekru.com.
how to get response time ? using chromdevtool ion selenium java
Hi Nitesh, we have incorporated your question in the article itself. Let us know if you still have any doubts.
COuld you please paste your POM.xml?
Sure, here is our pom.xml file.
Is It possible to run this code in python?
Hi Ani, this article’s code is written using Java, so it won’t be able to run in Python. However, there might also be something similar in Python, using which you can run your code.
Hi Ani, We have also written the same article in Python.
Article Link – https://www.codekru.com/selenium-python/get-network-call-responses-in-selenium-python
how to capture the network response post clicking a button/link in an application?
Hey Arpita, we can capture the network calls by storing them in a map before and after the click operation. We have included this update in our post and hope that this solution helps you. Please feel free to let us know if you have any other queries or concerns.
Hi! Thank you for this article. Is it possible to capture size values from network tab?