An easy Python-based strategy with the total error traceback despatched to Slack
As organizations develop, it’s common to have ELT or batch inference jobs dwelling in AWS Lambda that run periodically. There are lots of articles on the market on monitoring Lambda errors utilizing CloudWatch Alarms, however for periodic lambda jobs, that is each overly advanced and would not present a lot helpful info within the notification apart from that there was an error.
Right here I suggest a extra direct manner of monitoring these periodic Lambda capabilities. With this methodology, you’re going to get a notification containing the error message and traceback for every error instantly in your slack channel, eliminating the necessity to dig into the logs to troubleshoot the error. Whereas this methodology does contain CloudWatch, it doesn’t require CloudWatch Alarms or any message queuing Service like SQS or Kenisis.
A fast warning earlier than we start: In case your Lambda operate processes many requests per minute and begins to fail, your slack channel will blow up 💥 with error messages. Excessive-volume lambda capabilities are higher off utilizing one of many strategies described intimately within the following assets:
See the fundamental structure under.
The consumer sends a request to API Gateway, which forwards that request to a mannequin we’ve deployed with Lambda. That Lambda mannequin will return a prediction or an error to the API Gateway to ship again to the consumer and also will write a log message to Cloud Watch logs.
Up to now, we’ve a regular mannequin deployment on AWS Lambda. The magic occurs when CloudWatch triggers our Notification Lambda to ship a message to Slack every time the log comprises the textual content ERROR
.
How we’ll implement it:
- Deploy a easy linear mannequin to a lambda operate and join it to an API gateway endpoint.
- Create a Slack Webhook.
- Create a Lambda operate to ship messages to Slack.
- Configure a CloudWatch to set off our notification Lambda.
As a check case, we’ll deploy a easy mannequin that doubles a quantity and raises a ValueError
if the enter isn’t a quantity. If you have already got a lambda operate or different AWS service you need to monitor, you’ll be able to bounce forward to Create a Slack Webhook.
Create a brand new Lambda operate:
- Go to the Lambda service within the AWS console.
- Choose
Capabilities
from the sidebar. - Choose
Create operate
to — you guessed it — create a brand new operate.
4. Select Creator from Scratch
, enter a reputation in your Lambda operate, and choose the runtime. I’ll name my operate MonitorMe
, and I’m utilizing Python 3.8 because the runtime. Settle for the remainder of the default settings and click on Create operate
.
5. Use the next code for our easy Lambda mannequin.
Stepping via the code:
API Gateway forwards question string parameters to our Lambda operate via the occasion
parameter.
The above code first units the variable x
to occasion['queryStringParameters']['x']
. Each aspect in occasion['queryStringParameters']
is a string, so to make use of x
for any numeric calculations, we should forged it to a numeric knowledge kind. We do this with x = float(x)
. float
is an information kind representing a floating-point quantity, so making an attempt to forged something that isn’t a quantity to float
will trigger our Lambda to throw an exception. Lastly, we double x
and return it within the physique of our response.
Create the API Gateway Endpoint
Notifications would work with out this step, however organising an API is comparatively easy and supplies our MonitorMe
Lambda with a practical occasion
variable to course of.
- Go to the API gateway service and choose
Construct
on the HTTP API panel.
2. Join our MonitorMe
Lambda operate as an API integration.
Click on Add integration
- Choose
Lambda
as the mixing kind - Make sure that the right area is chosen (this could match the AWS area your Lambda is hosted in,) then seek for the
MonitorMe
Lambda operate we created above. - Present a reputation for the API. I am calling my API
MyAPI
. (intelligent, I do know) - Click on
Subsequent
3. Now we want a route. A route assigns an motion to a URL path. The motion we’re going to arrange forwards a request to our MonitorMe
Lambda operate.
- Enter a logical path for the Useful resource path. I’m utilizing the title of our Lambda operate
MonitorMe
, however you should utilize something - Choose
MonitorMe
because the Integration goal. - Click on Subsequent.
4. On the following display screen, settle for the defaults.
Make sure that Auto-deploy is turned on. With out it, we’ve to deploy any adjustments to our API manually. This could be helpful if we would have liked to deploy to a growth API for testing earlier than deploying into the manufacturing API. We need to hold issues easy for this instance.
The API is deployed. Let’s Take a look at it!
- Discover the
Invoke URL
on theMyAPI
web page for the$default
stage. That is our base URL. Copy it.
2. Add the route path we created in API creation step 3 to the bottom URL. Substitute {YOUR API BASE URL}
with base URL you copied within the earlier step. (The URL is case insensitive, however I’ve capitalized it for readability):
https://{YOUR API BASE URL}/MonitorMe
3. Add the question to the URL by appending a ?x=
adopted by the worth to check. That is the place we ship parameters to the MonitorMe
Lambda operate. For instance:
https://{YOUR API BASE URL}/MonitorMe?x=2
4. Plug this into your browser, and vualá. 2×2= 4.0
That is one tremendous helpful mannequin. 😉
Finally, we begin to get into the meat of the mission. We are going to ship messages to Slack by the use of a webhook.
- Create a channel in your notifications. I am calling mine
aws-notifications
2. Go to api.slack.com/apps. If you’re not signed in, you will note a message asking you to register to your Slack account to create an utility. Do it.
3. As soon as you’re signed in, you’ll as a substitute see a button labeled Create an App. Click on it.
4. We need to create our app from scratch.
5. On the following display screen, add an app title, choose your workspace, and click on Create App.
6. When you create the app, the Slack API web site conveniently suggests some options. We wish Incoming Webhooks, so discover that field and click on it.
7. On the incoming webhooks web page, click on the slider within the higher proper nook to show webhooks on. This will even broaden a settings part on the backside of the web page.
8. Click on the Add New Webhook to Workspace
button within the settings part on the backside of the web page.
9. Choose the channel we created earlier and Enable.
10. Now, you will note that you’ve created a webhook URL. The curl instance updates so we will check it out on the command line.
Merely seize the Pattern curl request to publish to channel
code and paste it into your terminal like so (If you’re on an older model of Home windows, you’ll be able to set up cURL by following the instructions in this StackOverflow reply):
It’s going to return okay
. Test your slack channel, and you will note a publish from AWS Notifier that claims Hiya, World!
.
Yay! We will now publish to our slack channel.
11. Copy the Webhook URL. We’re going to want it later.
On this part, we’ll arrange a Lambda operate that, when triggered, will ship us the Slack message with the error particulars.
Identical to earlier than, create a brand new Lambda operate:
- Go to the Lambda service within the AWS console.
- Choose
Capabilities
from the sidebar. - Choose
Create operate
to — you guessed it once more — create a brand new operate.
4. Select Creator from Scratch
, enter a reputation in your operate, after which choose the runtime. For this step, I’ll name the operate Notify
and use Python 3.8 because the runtime. Settle for the remainder of the default settings and click on Create operate
.
5. Copy and paste the next code into your new Lambda operate.
Let’s step via the code
- Set
SLACK_WEBHOOK
to the slack webhook URL you created earlier. You could replace this to the URL of the webhook you created. - AWS calls the
lambda_handler
operate when the Lambda is invoked. AWS passes inoccasion
andcontext
as parameters.occasion
comprises the encoded error message. - Subsequent, Lambda calls
decode_logpayload
to decode the error message. This operate does three issues:
1. Usebase64decode
to decode the error string.
2. Usegzip
to decompress the decoded error string.
3. Convert the ensuing JSON to a Python dictionary and return it. - Then we construct the slack message. Particulars on Slack message development can be found in Creating wealthy message layouts within the Slack API documentation.
• We add everylogEvent
to an attachments listing. This ensures we get your entire log message.
• Then, we construct the message with the primary textual content equal to the log group title, which identifies the Lambda that threw the error, and the listing oflogEvent
s, which comprises the error log. - Lastly, we create a request object with urllib’s
Request
class and ship it to theSLACK_ENDPOINT
with’sgeturl
methodology.
There’s one element right here that is essential. We create the set off on Notify
Lambda, not the Lambda throwing the error. Because of this if that you must monitor a number of lambda capabilities, you add one set off for every to the Notify
Lambda. Fortuitously, AWS doesn’t impose a restrict on the variety of triggers for a single Lambda operate.
Establishing the set off
- Whereas nonetheless in our
Notify
Lambda, chooseAdd set off
from the highest of the web page.
2. This can carry up the Set off configuration dialog.
• Choose CloudWatch Logs
because the set off kind.
• Choose the Log group comparable to the Lambda’s logs you need to monitor. If you’re following alongside, this shall be /aws/lambda/MonitorMe
.
• Title the filter. This may be something. Go wild.
• The filter sample will decide which logs set off the notification Lambda. Right here I’ll merely enter ERROR
which is able to match any logs that comprise the textual content ERROR
.
You will discover extra details about CloudWatch Filters on the Filter and sample syntax web page within the CloudWatch Documentation.
• Click on Add within the decrease proper of the dialog.
And we’re completed! 🌴
Let’s be sure that this all works as anticipated. We’ve two instances to check:
- We should always not get a notification in Slack if the
MonitorMe
Lambda logs a message that’s not an error. - We should always get a helpful message if the
MonitorMe
Lambda does log an error.
Case 1: Profitable MonitorMe
execution
To check this, we hit our API endpoint with the identical URL we used to check the endpoint initially:
https://{YOUR API BASE URL}/MonitorMe?x=2
And we confirm that no messages had been despatched to Slack:
Nothing occurred. Success!
Case 2: Unsuccessful MonitorMe
execution
Let’s modify the API name to ship a string as a substitute of a quantity. This can trigger the Lambda to throw an error.
https://{YOUR API BASE URL}/MonitorMe?x=wompwomp
Glorious. An error. Let’s take a look at Slack.
Wonderful! Not solely did we get a notification in Slack, the notification got here full with the error traceback! We will see that it was line 7 of lambda_function.py
and that the error comes from making an attempt to forged wompwomp
to drift.
There are different methods to seize and ship errors to Slack from any AWS service. I listed some articles at the start of this publish, however whenever you anticipate a low quantity of errors, this strategy is superior as a result of it’s extra easy and provides you your entire error message proper in Slack.
At Komaza, we use this strategy to observe pipelines in proof-of-concept tasks, knowledge high quality checks, and a few Lambda capabilities manually triggered by considered one of our inner net functions. We have caught errors like database schema adjustments, expired tokens, and varied different points instantly and had been in a position to deploy fixes in beneath 24 hours in all instances. This comparatively easy strategy to Lambda monitoring has improved our stakeholder expertise by guaranteeing greater high quality knowledge and serving to us reply sooner to system failures.
If you wish to assist us present life-changing financial alternatives for small-holder farmers whereas powering one of many planet’s handiest carbon sequestration engines, take a look at our careers web page. We’re rising quick and have knowledge groups in Kenya and the US.
Now go do some good.