A step-by-step tutorial with an illustration
You’re almost definitely accustomed to working Python scripts in your native machine. Nevertheless, that machine might not have the ability to meet the compute or storage necessities wanted to run sure operations.
For these circumstances, one can leverage digital situations to fulfill any necessities that may’t be met on premise.
Right here, we carry out a fast demo to point out how Python scripts could be run in Amazon’s EC2 situations.
Demo
What higher option to clarify easy methods to do one thing than with an illustration?
As a part of a information, we are going to intention to run a Python script named “random_number_generator.py” in an EC2 occasion.
Fairly easy, proper? The script simply prints out a random quantity utilizing the Numpy package deal.
It’s necessary to remember the model of Python in addition to the variations of the packages which might be wanted to run the code efficiently (my present setting makes use of Python 3.9.13 and makes use of simply the Numpy module).
To get the Python model utilized in a machine, merely enter the next command within the command line:
python --version
To get the checklist of packages and their variations put in with pip, use the next command within the command line.
pip checklist
You may set up packages in your occasion one after the other, however that may be time-consuming. You additionally run the danger of getting the unsuitable model of a package deal. So, it’s really helpful to retailer all packages in a “necessities.txt” file, which could be carried out with the next command.
pip freeze > necessities.txt
Step 1: Launch the EC2 Occasion
First, the EC2 occasion that the script will likely be run in must be launched.
I received’t delve into this step an excessive amount of, since there are numerous tutorials on easy methods to create an EC2 occasion on AWS (and it’s not designed to be troublesome within the first place).
Be happy to decide on the pricing and safety configurations that suit your use case. Since that is only a easy demo, we’ll use utilizing a free tier occasion with no further safety measures.
The extra necessary particulars when configuring an occasion are as follows:
1. Selecting an Amazon Machine Picture (AMI)
The very first thing you may be requested to do when launching an occasion is to decide on your AMI.
The AMI you select will decide the occasion’s username, which you’ll need to know to hook up with your occasion afterward. For these uncertain of what their occasion’s username is, the AWS documentation covers the default username assigned to situations launched with every AMI.
For our instance, we are going to launch an occasion utilizing the Amazon Linux AMI. The username for any such occasion is “ec2-user”.
The AMI of alternative may also affect the instructions that will likely be wanted to facilitate operations with the command line. As an illustration, the instructions used for an occasion launched with Amazon Linux AMI might not match an occasion launched with Ubuntu AMI.
2. Key pair
Additionally, you will have to create a key pair or use an current key (as a .pem file) for accessing your occasion.
For our instance, we are going to create a personal key referred to as “test_key.pem”
After you have accomplished all sections, click on “Launch occasion”.
Step 2: Hook up with the occasion
AWS means that you can connect with your occasion utilizing safe shell (SSH).
To seek out the ssh command wanted to hook up with the occasion, choose your occasion within the EC2 console and click on the “Join” tab on the highest. After that, choose the “SSH consumer” tab.
You can be introduced with the command you’ll need to hook up with your occasion. You might be additionally proven the general public DNS of your occasion (you’ll need this later, so maintain this in thoughts).
On this case, the command for connecting to this occasion is:
ssh -i "test_key.pem" ec2-user@ec2-34-204-68-237.compute-1.amazonaws.com
The ssh command for connecting to the occasion needs to be executed from the placement of your key file. In your terminal window, change the listing to the folder containing the non-public key and enter the offered command. Choose “sure” when prompted. If profitable, the output ought to one thing seem like the next:
Voila! You’ve got now linked to your occasion!
Step 3: Set up Python (Non-obligatory)
Your EC2 occasion ought to already provide a model of Python3, which you’ll be able to confirm your self with the python3 --version command.
If you’re on the lookout for a selected model of Python, you possibly can set up it your self (skip this step if this doesn’t apply to you).
The Amazon Linux occasion offers Python 3.7. For the sake of the demo, let’s assume that we have to match our native machine and set up Python 3.9.13.
Notice: The next instructions are meant for situations launched with Amazon Linux AMI. They might not be suited to situations launched with different AMI (e.g., Ubuntu).
First, we set up the dependencies for Python.
sudo yum -y groupinstall "Growth Instruments"
sudo yum -y set up openssl-devel bzip2-devel libffi-devel
Subsequent, we obtain the Python model of curiosity with the wget command.
sudo yum -y set up wget
wget https://www.python.org/ftp/python/3.9.13/Python-3.9.13.tgz
The occasion ought to now have a file named “Python-3.9.13.tgz”.
Notice: these instructions are used for downloading Python model 3.9.13. To get the Python model of your alternative, merely substitute the model quantity within the instructions with the one in every of your selecting.
After downloading the file, extract it.
tar xvf Python-3.9.13.tgz
The extracted Python folder ought to now be within the EC2 occasion (you possibly can verify this utilizing the ls command).
Transfer to the created Python folder and carry out the set up.
cd Python-*/
./configure --enable-optimizations
sudo make altinstall
With that, the brand new model of Python needs to be efficiently put in! To check this, we will enter the Python3.9 --version command.
Step 4: Copy information to the EC2 occasion
Now that we now have the proper model of Python, we will copy all information into the occasion. In our case, the information being copied are the “random_number_generator.py” and “necessities.txt” information.
We’ll go away the Python folder and create a brand new folder named “challenge”, which can include the 2 information. This step is non-compulsory; you possibly can place your information wherever you’d like within the occasion.
Now, open a brand new terminal window (let’s name this terminal 2). Terminal 2 will likely be used to repeat the information within the native machine to the EC2 occasion.
Recordsdata could be copied from one location to a different with the scp (secured copy) command. The command for copying information from an area machine to an EC2 occasion has the next format:
scp -i </path/key-pair-name.pem> </path/file_to_copy> <username>@<instance_public_dns>:<destination_path>
Notice: To get the EC2 occasion’s public DNS, go to the SSH consumer tab within the AWS EC2 console (see step 2).
The “random_number_generator.py” and “necessities.txt” information could be copied into the “challenge folder” within the occasion with these instructions:
scp -i test_key.pem random_number_generator.py ec2-user@ec2-34-204-68-237.compute-1.amazonaws.com:/residence/ec2-user/challengescp -i test_key.pem necessities.txt ec2-user@ec2-34-204-68-237.compute-1.amazonaws.com:/residence/ec2-user/challenge
If there are not any points, the command ought to yield the next output:
If you return to terminal 1 (i.e. terminal with the EC2 occasion), you need to have the ability to see the information within the vacation spot path you said within the scp command.
Step 5: Create a digital setting
At this stage, you is perhaps tempted to put in your packages in an effort to run your code immediately. Nevertheless, putting in packages globally may end up in conflicting conduct with the system package deal supervisor.
So, it’s higher to work in a digital setting as a substitute.
We’ll create a digital setting named “test_venv” after which enter that setting.
Python3.9 -m venv test_env
supply test_env/bin/activate
When you activate the digital setting, you need to have the ability to see its identify in parenthesis to the left of the present listing.
Now that we’re within the digital setting, we will set up all the mandatory packages wanted to run the script. In case you have your packages and variations saved in a necessities.txt file, you possibly can carry out all installations with a single command.
pip3.9 set up -r necessities.txt
As soon as once more, this command is for Python 3.9. Modify the instructions based mostly on the model of Python used.
Step 6: Run Your Code!
You’ve created your EC2 occasion, linked to the occasion, put in the proper Python model, transferred your information to the occasion, and arrange your digital setting. Lastly, you possibly can run your Python scripts in your new occasion!
As soon as you’re carried out working within the digital setting, you possibly can go away it with the deactivate command.
Pleasant Reminder: Cease Your Occasion
EC2 situations use a pay-as-you-go pricing mannequin, so at all times cease your occasion when you’re carried out utilizing them! In any other case, you’ll discover your self getting an disagreeable invoice.
Even in case you are engaged on a free-tier occasion, it’s finest to construct the behavior of stopping situations as quickly as you’re carried out with them early on.
Conclusion
With this demo, you’ve hopefully managed to run your scripts on AWS situations with little bother.
Even when you primarily write packages that may be run on native machines, it’s good to have some publicity to computing providers supplied by cloud platforms because you’ll be ready when your work has larger computational demand.
I want you the very best of luck in your knowledge science endeavors!
