This is a tutorial to create a job offer monitoring system in Python!
Some job offers receive so many applications they get removed after only a few days. If you want to be the first one to apply, you should monitor the career page of your favourite companies! In this article, I will share how you can build a simple job monitoring system with Python. Here are the steps to follow:
- Scrap job offers
- Compare offers with those from the day before
- Create an email system and get notified every time there is a new offer
- Schedule your scraper locally with a CRON job
- Python skills (Beginner)
- HTML skills (Beginner)
- The Selenium library
- A web driver (for instance ChromeDriver, which can be downloaded here: https://chromedriver.chromium.org/downloads)
All code in this article can be found in my GitHub repository:
Before you start installing the libraries, don’t forget to create a new virtual environment. If you don’t know what I’m talking about, check this! https://realpython.com/python-virtual-environments-a-primer/
Step 1: Scrape the Job Offers
Web Scraping is the process of collecting data from internet pages. The first goal is to collect the name of the job offers from a webpage. In this example, I’m going to use the page which lists the VIE (Volontariat International en Entreprise) jobs in South Africa. VIE is a special type of contract which gives young French citizens the opportunity to find jobs overseas. Here is the URL of the page I’m going to scrape.
The webpage looks like this:
By inspecting the webpage, you can see that the offer title has the class “xt_offrelink”. We are going to use this in order to select the information we want to scrape.
The function scrapeOffer will save the array of job titles in a pickle format. The inputs are the driverPath and the URL page that is to be monitored.
This function will save the array of job offers in a pickle format. It’s a convenient way to export Python objects in order to use them in another session. You can find more information about Pickle here.
The array: “I’M A PICKLE! I TURNED MYSELF INTO A PICKLE!!”
Step 2: Compare the Job Offer with Yesterday’s One
The function check_newOffer() will simply compare the offer from the day before with the current days offer. If there is a new offer, it will trigger the function “sendEmail” (this will be defined in the next section).
This algorithm can be run every day or every hour if you want to call shotgun on the job.
Step 3: Send Yourself an Email with the New Offer
There are many tutorials to teach you how to send an email with Python (https://realpython.com/python-send-email/). The function below will do the job quite well. Personally, I will use my hotmail.com account to send an email to my school outlook account. Depending on your email provider (Gmail, Yahoo, Wanadoo…), you might change the host address and the port used. A list can be found here: https://www.arclab.com/en/kb/email/list-of-smtp-and-pop3-servers-mailserver-list.html
Now that we have all our functions, let’s wrap them in a final file!
To run this code, go to your terminal and use:
If everything went well, you should receive emails with all the current job offers! Yeah-bo!
Step 4: Schedule the Scraper
Now it’s time to schedule your script to run every day (or every hour!). Your best option would be to run your script on a server in a cloud but an easier option will be to run it locally on your computer. Sure, you’ll need to make sure you’re computer is open when the script has to run, but it’ll save you time and money to schedule your scraper locally. In order to do so, you can create a CRON job. It’s a task programmed on your computer to be run at a specific time.
source your_virtual_env/bin/activate && python your_foler_path/wrapper.py
The function below will schedule the execution of the above line in the terminal every day at 12:00. Remember this will only work if your computer is turned on.
There we go! Now you will receive emails with new job offers every day at 12:00! More importantly, you learned what is a CRON job, how to use a webdriver with Selenium and how to perform basic Web Scraping! Now let’s be honest: creating a bot to monitor job offers is an elegant way of procrastinating your job hunting. If you really want a job, you should also connect with the Head of Talent of your dream company on LinkedIn, ask your friends for feedback on your resume and write this fucking motivational letter!
I’m freelancing as a Data Scientist/Full Stack Developer. Don’t hesitate to contact me on Linkedin if you need help with this project or if you need a freelancer! https://www.linkedin.com/in/henri-terrasse-672850101/