Wednesday, July 13, 2022
HomeData ScienceMonitor Your AWS Lambda Capabilities (or Any Different AWS Service) in Slack...

Monitor Your AWS Lambda Capabilities (or Any Different AWS Service) in Slack | by Ben Bogart | Jul, 2022


Picture by Brett Jordan on Unsplash

An easy Python-based strategy with the total error traceback despatched to Slack

As organizations develop, it’s common to have ELT or batch inference jobs dwelling in AWS Lambda that run periodically. There are lots of articles on the market on monitoring Lambda errors utilizing CloudWatch Alarms, however for periodic lambda jobs, that is each overly advanced and would not present a lot helpful info within the notification apart from that there was an error.

Right here I suggest a extra direct manner of monitoring these periodic Lambda capabilities. With this methodology, you’re going to get a notification containing the error message and traceback for every error instantly in your slack channel, eliminating the necessity to dig into the logs to troubleshoot the error. Whereas this methodology does contain CloudWatch, it doesn’t require CloudWatch Alarms or any message queuing Service like SQS or Kenisis.

A fast warning earlier than we start: In case your Lambda operate processes many requests per minute and begins to fail, your slack channel will blow up 💥 with error messages. Excessive-volume lambda capabilities are higher off utilizing one of many strategies described intimately within the following assets:

See the fundamental structure under.

Picture by Creator. AWS Structure Icon and Slack Emblem utilized in accordance with printed use insurance policies (1).

The consumer sends a request to API Gateway, which forwards that request to a mannequin we’ve deployed with Lambda. That Lambda mannequin will return a prediction or an error to the API Gateway to ship again to the consumer and also will write a log message to Cloud Watch logs.

Up to now, we’ve a regular mannequin deployment on AWS Lambda. The magic occurs when CloudWatch triggers our Notification Lambda to ship a message to Slack every time the log comprises the textual content ERROR.

How we’ll implement it:

  1. Deploy a easy linear mannequin to a lambda operate and join it to an API gateway endpoint.
  2. Create a Slack Webhook.
  3. Create a Lambda operate to ship messages to Slack.
  4. Configure a CloudWatch to set off our notification Lambda.

As a check case, we’ll deploy a easy mannequin that doubles a quantity and raises a ValueError if the enter isn’t a quantity. If you have already got a lambda operate or different AWS service you need to monitor, you’ll be able to bounce forward to Create a Slack Webhook.

Create a brand new Lambda operate:

  1. Go to the Lambda service within the AWS console.
  2. Choose Capabilities from the sidebar.
  3. Choose Create operate to — you guessed it — create a brand new operate.
Picture by Creator

4. Select Creator from Scratch, enter a reputation in your Lambda operate, and choose the runtime. I’ll name my operate MonitorMe, and I’m utilizing Python 3.8 because the runtime. Settle for the remainder of the default settings and click on Create operate.

5. Use the next code for our easy Lambda mannequin.

Stepping via the code:

API Gateway forwards question string parameters to our Lambda operate via the occasion parameter.

The above code first units the variable x to occasion['queryStringParameters']['x']. Each aspect in occasion['queryStringParameters'] is a string, so to make use of x for any numeric calculations, we should forged it to a numeric knowledge kind. We do this with x = float(x). floatis an information kind representing a floating-point quantity, so making an attempt to forged something that isn’t a quantity to floatwill trigger our Lambda to throw an exception. Lastly, we double x and return it within the physique of our response.

Create the API Gateway Endpoint

Notifications would work with out this step, however organising an API is comparatively easy and supplies our MonitorMe Lambda with a practical occasion variable to course of.

  1. Go to the API gateway service and choose Construct on the HTTP API panel.
Picture by Creator

2. Join our MonitorMe Lambda operate as an API integration.

Click on Add integration

  • Choose Lambda as the mixing kind
  • Make sure that the right area is chosen (this could match the AWS area your Lambda is hosted in,) then seek for the MonitorMe Lambda operate we created above.
  • Present a reputation for the API. I am calling my API MyAPI. (intelligent, I do know)
  • Click on Subsequent
Picture by Creator

3. Now we want a route. A route assigns an motion to a URL path. The motion we’re going to arrange forwards a request to our MonitorMeLambda operate.

  • Enter a logical path for the Useful resource path. I’m utilizing the title of our Lambda operate MonitorMe, however you should utilize something
  • Choose MonitorMe because the Integration goal.
  • Click on Subsequent.
Picture by Creator

4. On the following display screen, settle for the defaults.

Make sure that Auto-deploy is turned on. With out it, we’ve to deploy any adjustments to our API manually. This could be helpful if we would have liked to deploy to a growth API for testing earlier than deploying into the manufacturing API. We need to hold issues easy for this instance.

Picture by Creator

The API is deployed. Let’s Take a look at it!

  1. Discover the Invoke URL on the MyAPI web page for the $default stage. That is our base URL. Copy it.
Picture by Creator

2. Add the route path we created in API creation step 3 to the bottom URL. Substitute {YOUR API BASE URL} with base URL you copied within the earlier step. (The URL is case insensitive, however I’ve capitalized it for readability):

https://{YOUR API BASE URL}/MonitorMe

3. Add the question to the URL by appending a ?x= adopted by the worth to check. That is the place we ship parameters to the MonitorMe Lambda operate. For instance:

https://{YOUR API BASE URL}/MonitorMe?x=2

4. Plug this into your browser, and vualá. 2×2= 4.0

Picture by Creator

That is one tremendous helpful mannequin. 😉

Finally, we begin to get into the meat of the mission. We are going to ship messages to Slack by the use of a webhook.

  1. Create a channel in your notifications. I am calling mine aws-notifications
Picture by Creator

2. Go to api.slack.com/apps. If you’re not signed in, you will note a message asking you to register to your Slack account to create an utility. Do it.

Picture by Creator

3. As soon as you’re signed in, you’ll as a substitute see a button labeled Create an App. Click on it.

Picture by Creator

4. We need to create our app from scratch.

Picture by Creator

5. On the following display screen, add an app title, choose your workspace, and click on Create App.

Picture by Creator

6. When you create the app, the Slack API web site conveniently suggests some options. We wish Incoming Webhooks, so discover that field and click on it.

Picture by Creator

7. On the incoming webhooks web page, click on the slider within the higher proper nook to show webhooks on. This will even broaden a settings part on the backside of the web page.

Picture by Creator

8. Click on the Add New Webhook to Workspacebutton within the settings part on the backside of the web page.

Picture by Creator

9. Choose the channel we created earlier and Enable.

Picture by Creator

10. Now, you will note that you’ve created a webhook URL. The curl instance updates so we will check it out on the command line.

Picture by Creator (I deleted this webhook after penning this publish 😁)

Merely seize the Pattern curl request to publish to channelcode and paste it into your terminal like so (If you’re on an older model of Home windows, you’ll be able to set up cURL by following the instructions in this StackOverflow reply):

Picture by Creator

It’s going to return okay. Test your slack channel, and you will note a publish from AWS Notifier that claims Hiya, World!.

Picture by Creator

Yay! We will now publish to our slack channel.

11. Copy the Webhook URL. We’re going to want it later.

Picture by Creator

On this part, we’ll arrange a Lambda operate that, when triggered, will ship us the Slack message with the error particulars.

Identical to earlier than, create a brand new Lambda operate:

  1. Go to the Lambda service within the AWS console.
  2. Choose Capabilities from the sidebar.
  3. Choose Create operate to — you guessed it once more — create a brand new operate.
Picture by Creator

4. Select Creator from Scratch, enter a reputation in your operate, after which choose the runtime. For this step, I’ll name the operate Notifyand use Python 3.8 because the runtime. Settle for the remainder of the default settings and click on Create operate.

Picture by Creator

5. Copy and paste the next code into your new Lambda operate.

Let’s step via the code

  • Set SLACK_WEBHOOK to the slack webhook URL you created earlier. You could replace this to the URL of the webhook you created.
  • AWS calls the lambda_handler operate when the Lambda is invoked. AWS passes in occasionandcontext as parameters. occasion comprises the encoded error message.
  • Subsequent, Lambda callsdecode_logpayload to decode the error message. This operate does three issues:
    1. Use base64decode to decode the error string.
    2. Use gzip to decompress the decoded error string.
    3. Convert the ensuing JSON to a Python dictionary and return it.
  • Then we construct the slack message. Particulars on Slack message development can be found in Creating wealthy message layouts within the Slack API documentation.
    • We add every logEvent to an attachments listing. This ensures we get your entire log message.
    • Then, we construct the message with the primary textual content equal to the log group title, which identifies the Lambda that threw the error, and the listing of logEvents, which comprises the error log.
  • Lastly, we create a request object with urllib’sRequest class and ship it to the SLACK_ENDPOINT with’s geturl methodology.

There’s one element right here that is essential. We create the set off on Notify Lambda, not the Lambda throwing the error. Because of this if that you must monitor a number of lambda capabilities, you add one set off for every to the Notify Lambda. Fortuitously, AWS doesn’t impose a restrict on the variety of triggers for a single Lambda operate.

Establishing the set off

  1. Whereas nonetheless in our Notify Lambda, choose Add set off from the highest of the web page.
Picture by Creator

2. This can carry up the Set off configuration dialog.
• Choose CloudWatch Logs because the set off kind.
• Choose the Log group comparable to the Lambda’s logs you need to monitor. If you’re following alongside, this shall be /aws/lambda/MonitorMe.
• Title the filter. This may be something. Go wild.
• The filter sample will decide which logs set off the notification Lambda. Right here I’ll merely enter ERROR which is able to match any logs that comprise the textual content ERROR.

You will discover extra details about CloudWatch Filters on the Filter and sample syntax web page within the CloudWatch Documentation.

• Click on Add within the decrease proper of the dialog.

Picture by Creator

And we’re completed! 🌴

Let’s be sure that this all works as anticipated. We’ve two instances to check:

  1. We should always not get a notification in Slack if the MonitorMe Lambda logs a message that’s not an error.
  2. We should always get a helpful message if the MonitorMe Lambda does log an error.

Case 1: Profitable MonitorMe execution

To check this, we hit our API endpoint with the identical URL we used to check the endpoint initially:

https://{YOUR API BASE URL}/MonitorMe?x=2
Picture by Creator

And we confirm that no messages had been despatched to Slack:

Picture by Creator

Nothing occurred. Success!

Case 2: Unsuccessful MonitorMe execution

Let’s modify the API name to ship a string as a substitute of a quantity. This can trigger the Lambda to throw an error.

https://{YOUR API BASE URL}/MonitorMe?x=wompwomp
Picture by Creator

Glorious. An error. Let’s take a look at Slack.

Picture by Creator

Wonderful! Not solely did we get a notification in Slack, the notification got here full with the error traceback! We will see that it was line 7 of lambda_function.pyand that the error comes from making an attempt to forged wompwomp to drift.

There are different methods to seize and ship errors to Slack from any AWS service. I listed some articles at the start of this publish, however whenever you anticipate a low quantity of errors, this strategy is superior as a result of it’s extra easy and provides you your entire error message proper in Slack.

At Komaza, we use this strategy to observe pipelines in proof-of-concept tasks, knowledge high quality checks, and a few Lambda capabilities manually triggered by considered one of our inner net functions. We have caught errors like database schema adjustments, expired tokens, and varied different points instantly and had been in a position to deploy fixes in beneath 24 hours in all instances. This comparatively easy strategy to Lambda monitoring has improved our stakeholder expertise by guaranteeing greater high quality knowledge and serving to us reply sooner to system failures.

If you wish to assist us present life-changing financial alternatives for small-holder farmers whereas powering one of many planet’s handiest carbon sequestration engines, take a look at our careers web page. We’re rising quick and have knowledge groups in Kenya and the US.

Now go do some good.

RELATED ARTICLES

LEAVE A REPLY

Please enter your comment!
Please enter your name here

- Advertisment -
Google search engine

Most Popular

Recent Comments