Headless Chrome and regular Chrome have the same capabilities, and running them with Selenium is a very similar process. The difference is that Headless Chrome does not generate any sort of user interface. In other words, no browser is visibly launched.
If you happen to be web scraping with Selenium, it’s often helpful to see what exactly the browser is doing in real time for development and debugging purposes. However, using headless mode can be great if your script is working and you don’t want to be bothered with an open browser. And even better, a headless browser should generally run faster than its headed counterpart, given that it doesn’t require the extra resources normally needed to visually render everything happening in the browser.
To run Headless Chrome, you’ll first need to set up Selenium.
Once you’ve got Selenium working, using Headless Chrome is a breeze. For example, let’s see if we can get to DuckDuckGo‘s home page.
Since Headless Chrome has no visible browser, we’ll take a screenshot to confirm what the browser is doing.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
from selenium import webdriver chromedriver = 'C:\\Users\\grayson\\Downloads\\chromedriver.exe' options = webdriver.ChromeOptions() options.add_argument('headless') options.add_argument('window-size=1200x600') # optional browser = webdriver.Chrome(executable_path=chromedriver, chrome_options=options) browser.get('https://www.duckduckgo.com') browser.save_screenshot('C:\\Users\\grayson\\Downloads\\headless_chrome_test.png') browser.quit() |
If the script finishes and your screenshot shows DuckDuckGo’s home page, you’re all set!
If something’s not working, make sure Selenium, Chrome, and chrome driver are all up-to-date and try again. If all else fails, leave a comment and we’ll figure it out.
Aadil says
Hi, how would I add a username and password if authentication is required?
I am trying to access an Intranet page, and a box pops up asking me for a username and password.
So currently I get a 401 error code response.
Thank you
Grayson Stanton says
Hi Aadil. First you’ll need to locate the username and password input elements with xpath or css selectors. It’ll probably look something like username_element = browser.find_element_by_xpath(‘//input[@name=”username”]’)
Then you’ll need to send the login credentials to their respective elements with the send_keys() method, which will probably look something like username_element.send_keys(“Aadil”)
Finally, you’ll need to either simulate pressing the Enter key or go through a similar process as above to locate and click the login button using the click() method.
Let me know if this doesn’t work for you.
Aadil says
Hi, thanks for the response. Sorry – I wasn’t too clear in my question.
This is for proxy authentication, so not really credentials on the page but rather a popup from the browser.
Grayson Stanton says
Oh I see, so you need to switch to a different window. In that case, you’ll probably want to use the browser.switch_to_window() method and browser.window_handles. Check out the links below for more details:
https://selenium-python.readthedocs.io/navigating.html
https://stackoverflow.com/questions/10629815/handle-multiple-window-in-python
https://stackoverflow.com/questions/8631500/click-the-javascript-popup-through-webdriver
https://stackoverflow.com/questions/17676036/python-webdriver-to-handle-pop-up-browser-windows-which-is-not-an-alert