Moto, Pytest, and AWS Databases: A High quality and Knowledge Engineering Crossroads | by Taylor Wagner | Jan, 2023

The hows and whys of Moto and Pytest with AWS databases

Overview

I’ve just lately labored loads with Pytest to check AWS companies utilizing Boto3. Boto3 is a really highly effective AWS SDK for working with AWS companies. My analysis and hands-on expertise with Boto3 lead me to find a complimentary device known as Moto! Moto, in partnership with Boto3, turned my go-to for take a look at technique planning in addition to sustaining clear manufacturing knowledge in my latest undertaking. On this article, I’ll share the hows and whys for mock testing AWS databases utilizing Moto (a faker Boto3) with Pytest. I’ll even take you step-by-step via the method of testing an AWS NoSQL database service — Amazon DynamoDB — with Moto and Pytest.

Please be aware: All pictures until in any other case famous are by the creator.

Moto

Moto is a Python library that has the flexibility to mock programmatic utilization of AWS companies. To put it out very merely — Boto3 is an object-oriented API for creating, configuring, and managing actual AWS companies and accounts, and Moto is a mock model of Boto3. Whereas Moto doesn’t embody each technique Boto3 provides, Moto does present an excellent proportion of Boto3 strategies with a view to run checks regionally. When working with Moto, I extremely advocate utilizing Boto3 documentation together with the Moto docs for extra detailed explanations of strategies.

The benefits of Moto:

• AWS account not required
• Keep away from the danger of adjusting precise knowledge or assets in an AWS manufacturing account
• Simply convert Moto mock set ups into Boto3 for actual world use-cases
• Quick working checks — no latency issues
• Simple to study and get began with
• Free! — is not going to incur any prices from AWS

An obstacle of Moto:

• Doesn’t embody all of Boto3 strategies — Boto3 provides extra in depth protection of AWS companies

The Moto documentation gives perception into which Boto3 strategies which are out there to be used. For instance, slightly over half of the DynamoDB strategies from Boto3 can be found to mock out utilizing Moto.

Pytest

Pytest is a Python framework used for writing small and readable checks. This framework makes use of detailed assertion introspection, making plain assert statements very user-friendly. Pytest has the highly effective functionality to be scaled to assist advanced, useful testing of functions and libraries.

Exams may be executed with a easy pytest command. The pytest command runs all checks however particular person take a look at information may be run with the filepath after pytest, like so: pytest checks/test_dynamodb.py. Operating pytest offers you correct outcomes whereas every take a look at will yield both a green-colored “.” for move, a red-colored “F” for fail, or a yellow-colored “S” for skip (if relevant). There are two tremendous useful flags that I’m going to share and extremely advocate placing to make use of when executing checks with Pytest. Please be aware: The code doesn’t change in any respect between the instances these completely different instructions have been run throughout the screenshots on this part. The one distinction is the flags.

First is the -s (stdout/stderr output) flag. Operating pytest -s will show all print statements within the terminal when checks are run. With out the addition of the -s flag, even when there are print statements within the code, no print statements will seem within the terminal. That is very helpful for debugging functions.

One other helpful flag is the -v (verbose) flag. Utilizing pytest -v will present many extra granular particulars within the terminal when checks are run. This flag will present the category and technique names of every take a look at, an gathered proportion of every take a look at’s completion within the scheme of all checks to be ran, in addition to a inexperienced “PASSED”, purple “FAILED”, or yellow “SKIPPED” indicator with every particular person take a look at. Discover that this flag doesn’t present print statements.

Professional-tip: It’s doable to mix flags when working the take a look at execution command! When working with Pytest, I usually run pytest -sv. The mixture of the -s and -v flags collectively present the small print that I’m on the lookout for with larger readability in addition to perception into print statements.

Getting Began

Pytest requires the usage of Python3.7+. You’ll be able to obtain the newest model of Python right here. Then, it would take three libraries to get began. Open your IDE of alternative and in a model new listing, run the next terminal command to obtain the wanted dependencies:

pip set up boto3 moto pytest

Subsequent, we arrange our “shopper connection”. The method for establishing a Moto connection is similar to Boto3. For Boto3, AWS credentials are required. With a purpose to mock this course of utilizing Moto, we’ll simply use faux credentials. Be at liberty to set the values to no matter you want so long as the kind is a string.

Please be aware: In an actual use-case, it will be essential to maintain confidential credential values protected and never hard-code them into an utility. Variables may be exported by way of the command line or saved in an setting file to restrict accessibility. Additionally, Any area can be utilized for the connection, however “us-east-1” is often used as that area consists of all AWS companies and choices.

Pytest makes use of fixtures in a file named conftest.py to share throughout modules. Within the root of your utility, create the file by working this command:

contact conftest.py

and add within the following content material to configure the shopper connection:

import boto3
import os
import pytest
from moto import mock_dynamodb
@pytest.fixture
def aws_credentials():
"""Mocked AWS Credentials for moto."""
os.environ["AWS_ACCESS_KEY_ID"] = "testing"
os.environ["AWS_SECRET_ACCESS_KEY"] = "testing"
os.environ["AWS_SECURITY_TOKEN"] = "testing"
os.environ["AWS_SESSION_TOKEN"] = "testing"
@pytest.fixture
def dynamodb_client(aws_credentials):
"""DynamoDB mock shopper."""
with mock_dynamodb():
conn = boto3.shopper("dynamodb", region_name="us-east-1")
yield conn

Each the AWS credentials perform and the mock DynamoDB shopper perform are implementing the Pytest fixture decorator in order that they might be leveraged in different information of the appliance. The mock DynamoDB shopper perform makes use of Moto to create a faux shopper connection to AWS, equally to how it’s carried out in Boto3. For the aim of this text, we’ll check out Moto with DynamoDB for instance AWS service. Look into the Moto docs when you have curiosity in utilizing Moto for different AWS companies.

Testing DynamoDB with Moto

Go forward and create one other file known as test_dynamodb.py within the root listing of your undertaking with the next command:

contact test_dynamodb.py

Step one within the take a look at file is to create a DynamoDB desk. Since many strategies in Moto require a desk already created, we’ll create a really primary take a look at desk that will probably be leverage when utilizing varied strategies to keep away from the necessity of making a desk in each single take a look at case. There are a number of methods for creating this take a look at desk, however I’m going to point out you methods to create the desk by implementing a context supervisor. The good information is that context managers come from Python’s native library, requiring no extra package deal downloads. The take a look at desk may be created like this:

After creating the take a look at desk in a single perform with a context supervisor, we will go forward and take a look at that this take a look at desk was certainly created. The desk title of the take a look at desk will probably be a required parameter for a lot of Moto DynamoDB strategies, so I’ll create this as a variable for your complete take a look at class to keep away from repetition and for higher readability. This step isn’t crucial, but it surely helps save time and lowers the prospect of syntax errors from having to repeatedly re-write worth of the desk title when calling completely different strategies.

In the identical test_dynamodb.py file, beneath the the context supervisor, create a take a look at class so that every one associated checks (checks for the “my-test-table”) may be grouped collectively.

As you may see above, the dynamodb_client argument is handed via the take a look at technique as an argument. This comes from the conftest.py file, the place we already established the mock connection to Amazon DynamoDB. Line 4 shows the non-obligatory declaration of the desk title variable. Then on line 9, we name the context supervisor into the test_create_table technique with a view to entry the “my-test-table”.

From there, we will write a take a look at to validate that the desk was efficiently created by calling the .describe_table() and .list_tables() strategies from Moto and asserting that the desk title exists within the response output. To run the take a look at merely execute the next command (it’s possible you’ll embody extra flags in the event you so want):

pytest

This command ought to yield the results of a passing take a look at:

Additional DynamoDB Testing with Moto

Your take a look at file is now prepared for any extra DynamoDB take a look at strategies that you simply wish to discover utilizing Moto, corresponding to including an merchandise to the “my-test-table”:

The take a look at you see above may be positioned contained in the TestDynamoDB class straight underneath the test_create_table technique. The test_put_item technique additionally makes use of the context supervisor to make use of the identical take a look at desk that was initially created on the prime of the file. At the beginning of every technique and with the decision of the context supervisor inside every take a look at suite, the “my-test-table” is in it’s authentic standing upon first creation. Please be aware: state doesn’t persist between take a look at strategies.

I’ve created a GitHub repository that encompasses the pattern checks included within the article with the addition of rather more! My instance code expands on the testing within the article to incorporate the next DynamoDB checks:

• Making a desk
• Placing an merchandise into the desk
• Deleting an merchandise from the desk
• Deleting a desk
• Including desk tags
• Eradicating desk tags

This repository additionally consists of some testing for AWS RDS. In my instance code, I organized take a look at information into its personal listing and grouped checks for consistencies in testing functions. I cherry-picked completely different strategies for every service to showcase the vary of what Moto provides. There are lots of extra strategies that Moto has out there that aren’t included in my instance code. Make sure to examine Moto’s documentation!

Ultimate Ideas

Plugging in Pytest fixtures and Python context managers, Pytest’s ease of use and read-to-scale framework coupled with Moto is a good mixture for validating the utilization of AWS’s database companies. Whereas the Moto/Pytest cocktail will not be a enough drink of alternative for testing actual AWS companies and accounts, it’s a enough possibility for training and gaining a greater understanding of the way you may take a look at your actual AWS account with out the specter of safety breaches, tampering essential knowledge, or working up an costly, pointless invoice. Simply take into account that Moto has a restricted vary of strategies as in comparison with Boto3. It has been my expertise that working with Moto’s mocked responses will present you the understanding wanted to make the most of Boto3 for actual AWS use-cases. It’s my hope that my article and instance code might help you together with your AWS database wants.