Browser automation¶
Kibitzr uses Firefox browser and Selenium Python library for browser automation.
Installing Firefox can be cumbersome, please refer to FireFox installation guide.
Simple HTML forms can be filled using form
key,
but complex scenarios require full power of Python (Selenium) scripting.
Filling simple forms¶
Imagine, for example, that you need to authorize on a site before fetching content. For common case the check will look like:
checks:
- name: Bank account balance
url: https://bank.com
form:
- id: login
creds: bank.login
- id: password
creds: bank.password
... (transform and notify) ...
Key id: login
means that HTML element fill be found using ID selector login
.
Key creds: bank.login
means that input’s value will be taken from creds
dictionary
using bank.login
as a hierarchy path.
Check assumes that kibitzr-creds.yml
contains:
bank:
login: mr.robot
password: 123&dSLHj*sdfa
Available Form Selectors¶
Field can be selected using one of the three selectors: id, css, xpath. (Make sure to use lowercase).
Available Field Value Generators¶
As in example above, field can be filled from creds
dictionary.
Another option is to provide Jinja2 template in key value
.
Template will have access to conf
and creds
.
However any plain text value can be passed as well.
For example, the same value, as in creds example can be rendered by:
checks:
- name: Bank account balance
url: https://bank.com
form:
- id: login
value: "{{ creds['bank']['login'] }}"
- id: password
value: "{{ creds['bank']['password'] }}"
... (transform and notify) ...
Note: don’t forget to wrap Jinja2 template in quotes, since curly bracket is a valid YAML markup for dictionary. Please refer to Jinja2 template documentation for details.
Python scenarios with Selenium¶
For complex cases Kibitzr provides access to Selenium driver. Here is an example of filling current date into form field:
checks:
- name: Daily updates
url: https://daily.com
scenario: |
import datetime
today = datetime.date.today()
element = driver.find_element_by_id('datefield-1')
element.send_keys(today.strftime('%m/%d/%Y')
run = driver.find_element_by_id('run-button')
run.click()
... transforms and notify ...
Wait for Javascript to render contents¶
Sometimes web page uses some complex Javascript to render a page after it is loaded.
These pages don’t require form filling, or complex scenarios, simple delay will do.
To define delay add delay
key with number of seconds to wait:
checks:
- url: https://www.producthunt.com/posts/kibitzr
delay: 1
... transforms and notify ...
Working around two-factor authentication¶
Some sites require entering code sent in a SMS for logging from the new device. 2-FA can’t be automated without weakening security. But Kibitzr can use persistent Firefox profile. Start persistent Firefox session with
$ kibitzr firefox
Than authenticate on all sites, that require first-login 2-FA.
When ready, hit Return in the terminal prompt.
New profile will be saved in firefox_profile
directory.
If this directory exists, kibitzr will load it for each following run.
Note: if running kibitzr remotely through SSH, use X11 forwarding.
Debugging/Troubleshooting¶
Writing robust Selenium scenarios is no easy task, and most likely it won’t work from the first time. Kibitzr has a few options to help with debugging.
See what happens in Firefox by running in foreground mode. Just add
checks: - url: ... scenario: ... headless: false
to check dictionary.
Launch Pdb within scenario and explore step-by-step.
checks: - url: https://javascript-labyrinth.io scenario: import pdb; pdb.set_trace() ...
Experiment inside Jupyter notebook. See notebook example.