Headless Firefox and regular Firefox have the same capabilities, and running them with Selenium is a very similar process. The difference is that Headless Firefox does not generate any sort of user interface. In other words, no browser is visibly launched.
If you happen to be web scraping with Selenium, it’s often helpful to see what exactly the browser is doing in real time for development and debugging purposes. However, using headless mode can be great if your script is working and you don’t want to be bothered with an open browser. And even better, a headless browser should generally run faster than its headed counterpart, given that it doesn’t require the extra resources normally needed to visually render everything happening in the browser.
To run Headless Firefox, you’ll first need to set up Selenium.
Once you’ve got Selenium working, using Headless Firefox is a breeze. For example, let’s see if we can get to DuckDuckGo‘s home page.
Since Headless Firefox has no visible browser, we’ll take a screenshot to confirm what the browser is doing.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
from selenium import webdriver geckodriver = 'C:\\Users\\grayson\\Downloads\\geckodriver.exe' options = webdriver.FirefoxOptions() options.add_argument('-headless') browser = webdriver.Firefox(executable_path=geckodriver, firefox_options=options) browser.get('https://www.duckduckgo.com') browser.save_screenshot('C:\\Users\\grayson\\Downloads\\headless_firefox_test.png') browser.quit() |
If the script finishes and your screenshot shows DuckDuckGo’s home page, you’re all set!
If something’s not working, make sure Selenium, Firefox, and gecko driver are all up-to-date and try again. If all else fails, leave a comment and we’ll figure it out.
Luis says
January 17, 2019 at 8:59 pmThis was exactly what I was looking for, thank you so much for this. Now I just need to make my web scraper run on “headless mode” and find a way to host it somewhere so it is continuously working.
Grayson Stanton says
January 17, 2019 at 9:33 pmHey thanks Luis, glad it was helpful. For hosting you might give Python Anywhere a look, I’ve used them in the past and had a good experience.