Executing python script via crontab

Executing python script via crontab

INTRODCUTION

Crontab: Once you hear word cron you will come across the word daemon, so before proceeding further, let us define what is meaning of word daemon. Daemon is a computer program that runs as a background process. So if I say the word cron daemon, it means that it is also a program that runs the task set by you in the background. One can also say that it is the tool to automate the process. One need to add tasks to system’s crontab files using the appropriate syntax and cron will automatically run these files for you. In this tutorial, you will learn how to automate python scripts using bash as well as crontab. Before processing let us see where are the system crontab files.

Location of system crontab files: These are located at location /var/spool/cron/. If you will move to this location, then you will see the directory crontabs. If you move inside this directory using the admin privileges, you can see crontab files. Be default you will see that directory is empty and question is how to create files inside it. We will automate python script using crontab by inserting script entry in crontab file using command crontab -e. Then will look for its existence at location /var/spool/cron/crontabs. Also, we will also look for the crontab errors at location var/log.

Downloading python code for automation: I have written some of the python code at Sanpreet Github Repository. You may also follow me there for latest updates in machine learning, deep learning, python basics, docker and other latest technology. I am cloning one of repository to automate it using git clone.

git clone https://github.com/sanpreet/Text-Similarity-using-cosine-similarity.git

Creating the virtual environment to run the code: I would prefer to create the virtual environment to run the code at the local system and then make the entry of virtual environment using the bash script to crontab file. This is the approach I will follow in this tutorial. Let us create the virtual environment as below

virtualenv python --python3 crontabenv

If virtualenv is not installed in the system, please install it using the below command

sudo apt-get install virtualenv

I am providing these steps with respect to ubuntu. Steps may change if operating system in windows or mac.

How to run the code on local system: After creating the virtualenv, it is important to activate it using the below command

source crontabenv/bin/activate

As the environment is activate please go to the below folder

cd Text-Similarity-using-cosine-similarity

You will see requirements.txt file. Please install it using the below command

pip3 install -r requirements.txt

You will see the below screenshot

Installing requirements.txt in virtual environment

Putting python code in bash file: To automate the script it is important to make a bash script for this application and running such script using crontab file. So let us create a bash file using the below command

sudo touch script_file.sh

If permissions are not provided to the file, please provide them using the below command

sudo chmod 777 script_file.sh

Open the file using any editor suppose I had opened the file using nano editor using the below command

sudo nano script_file.sh

Write the below lines to it. Remember write your path there where you had cloned

#! /bin/bash
cd /home/datascience/Desktop/all_folders/Sanpreet_Sir/Github/Text-Similarity-using-cosine-similarity
source ../crontabenv/bin/activate
python3 Text_Similarity.py

Run the bash file after deactivating virtualenv using the below command

./script_file.sh

If everything goes well you will see the below result

Running bash script

Inserting the entry to the crontab file: Now it is the last step which is to insert the entry to the crontab file. Open the crontab file using the command crontab -e

* * * * * /bin/bash /home/datascience/Desktop/all_folders/Sanpreet_Sir/Github/Text-Similarity-using-cosine-similarity/script_file.sh > /dev/null 2>&1

This will run the script all the time. I will also create another post where I will explain what happens when the system shut down to the task executed by the cron daemon. Now if everything goes well, you will see the output when you write this task to the file.

crontab: installing new crontab

Let us see what happens in the location /var/spool/cron/crontabs. You will get the entry data science there.

How to see the crontab logs:

How to see crontab logs

This is a very important step because if you are able to see the logs, you can know whether the script is running properly or not. To see the log please open the below file

sudo nano /etc/rsyslog.d/50-default.conf

Let us see how the file looks like

log errors cron in 50–default.conf

Start the service rsyslog using the below command

sudo service rsyslog restart

Once you did this, logs will be formed in var/log/cron.log. Let us see newly created log file. I got output as below

Sep  8 11:47:01 datascience-MS-7B49 CRON[12221]: (datascience) CMD (/bin/bash /home/datascience/Desktop/all_folders/Sanpreet_Sir/Github/Text-Similarity-using-cosine-similarity/script_file.sh > /dev/null 2>&1 )

I think the problem is because of /dev/null 2>&1. This means the output of the file is redirected to /dev/null. Redirecting to null means there is no output. So I have edited the command written the crontab file by opening the file using the command crontab -e as below

* * * * * /bin/bash /home/datascience/Desktop/all_folders/Sanpreet_Sir/Github/Text-Similarity-using-cosine-similarity/script_file.sh >> /home/datascience/Desktop/all_folders/Sanpreet_Sir/Github/Text-Similarity-using-cosine-similarity/output.txt

This will create the file output.txt and >> will append the output to this file

Conclusion: In this tutorial, readers have read about how to automate python scripts using bash as well as crontab file. We have also gone through the location of crontab file as well how to enable cron.log so as to know whether script is running properly or not. There are some parts missing in this tutorial such as whether the script is running when the computer is shut down or how cron daemon can do the work when the system is off. I believe I would make another tutorial for such.At the last we have also solved an error where the output is discarded using /dev/null 2>&1 and is corrected by creating a file where the output is redirected. I hope at the end of the tutorial there may be some questions which is coming in your mind about executing python scripts using crontab and I would be happy if you ask me. You may also read other articles from this blog to get knowledge about deep learning, tensorflow, NLTK (Natural Language Toolkit) and basics of python. In the coming time I would also write articles on edge computing which is the demand of today. You may contact me at my email id: successindeed358@gmail.com.

References

How to use cron in linux

How to schedule task on linux: An Introduction to Crontab Files

Originally published at http://ersanpreet.wordpress.com on September 8, 2019.