How Batch Jobs Can Assist Cybersecurity | by Teri Radichel | Cloud Safety | Jul, 2022

July 18, 2022

1

Batch jobs for penetration testing, safety metrics, incident response, and extra

Picture from AWS Put up on Monte Carlo Simulations

In my final submit I wrote a couple of ebook evaluation on the ebook Learn how to Measure Something in Cybersecurity Threat. A lot of the work and analysis I’m doing proper now revolves across the idea of cybersecurity metrics. How can we seize them? How can we stop errors extra holistically to cut back danger? 2nd Sight Lab’s work as a penetration testing and safety evaluation firm falls into this realm.

Usually cybersecurity steering talks in regards to the “what” you need to do and never the “the right way to do it”. In my final ebook I proposed some concepts round cybersecurity metrics and the info you may wish to seize. I talked about automating processes. However what do I imply precisely? How do you try this?

I wish to present extra concrete examples of the way you may seize and course of some information and switch it right into a report. That’s precisely what I do with my penetration assessments, however you may do it with any kind of information from any system or log and write no matter kind of report serves your goal.

I might advocate striving to seize information about safety issues that might facilitate an information breach or enhance the affect, ought to one happen. After you’ve accomplished that, transfer on to auto-remediation, enforcement, detection, predictive danger calculations, and risk searching. That may be my precedence order if I had restricted time and finances. If you happen to can work on it abruptly, even higher.

How batch jobs can assist

I’ve labored in a myriad of industries throughout my profession from oil and gasoline to retail, safety and know-how firms, healthcare, fishing, meals and beverage, actual property, enterprise capital, video games, music, promoting, public relations, publishing and printing. But when I needed to let you know the place most of my expertise exists, it might be within the monetary sector.

A part of that was working for a financial institution on again workplace methods involving information processing for billions of {dollars} of belongings underneath administration and tax methods for one of many largest banks within the US. I labored in the same capability for a fortune 500 US retail firm on a venture emigrate a tax system off by $300,000 monthly to our preliminary launch which was off by $23 — and I shortly discovered why and stuck it. In each these environments we made use of batch jobs.

Monetary System Examples: Batch jobs are used regularly in monetary methods to course of a batch of information regularly. Examples, most of which I labored on personally:

Course of tax information and price foundation at particular closing dates to arrange information for reporting.
Dividends processing to calculate and put dividend funds in investor accounts.
Implement a brand new sweep account that calculated curiosity every day for money in an funding account.
Course of transfers of cash into the corporate from exterior investing firms.
Course of trades, mutual fund investments, and 401K exercise.
Retailer stay chat information in accordance with monetary business rules.

Inputs and outputs: Usually these batch jobs had a feed that was typically a flat file (sometimes a CVS file again then) handed over from an inside or exterior system to a location on the community. The batch job would learn and course of the file and enter the info right into a database within the correct format or remodel the file into one other file. In my newest instance I’m taking inputs from two safety merchandise in XML and JSON.

Transformations: Typically the batch job would run database queries to rework the info within the course of. For instance, banking methods have reconciliation processes run on the finish of a day to make sure all of the transactions within the system add up (actually known as “Finish of Day”) and are correctly recorded previous to the beginning of the subsequent day. That finish of day processing ran as batch jobs. My newest work includes reworking inputs into a typical format for processing.

Background processing: Batch jobs execute with out human interplay. They’re examined to execute correctly and are kicked off by automated occasions or scheduling to carry out their job. Lack of human interplay, a well-tested course of, and safe implementation can assist stop safety errors and tampering.

Single goal: The fantastic thing about a batch job is that it’s scheduled to run at a sure time of day to course of a particular chunk of information and produce a particular end result. Every batch job has a single goal and will be examined and run individually from each different batch job. Organizations can purchase third-party instruments to handle batch jobs that report on their success or failure so any issues will be shortly recognized and resolved.

Discrete steps: One different factor I did with batch jobs was to separate processes into phases. As an alternative of making an attempt to run a complete complicated course of without delay (such because the processing carried out by that tax system producing the massive month-to-month discrepancy) you’ll be able to run particular person processes main as much as the ultimate end result. That lets you independently take a look at every step of a course of and discover errors alongside the way in which. It’s simpler to pinpoint issues.

Simplify testing: Breaking a batch job into steps makes it much less time consuming to construct and take a look at every subsequent step. You don’t should run the primary two steps within the course of each time you wish to take a look at the third step. You seize the output of the second step after which use that as enter once you take a look at the third step. You’ll simply must have validation in place to make sure the steps are all the time run so as and nothing is skipped.

Re-runnable: One key level about batch jobs is that they should be re-runnable, or idempotent. If you happen to run the batch job again and again on the identical information they need to produce the identical end result. Ideally they don’t have to roll again all the outcomes if the job failed midway by with the intention to begin from the start. The batch job ought to decide up the place it left off. That’s an issue I had with a safety instrument I used to be utilizing just lately. After I transformed to a batch job I had so as to add in performance to make it re-runnable. I’ll most likely clarify that additional in upcoming posts.

Occasion-Pushed Safety: One other massive advantage of batch jobs within the cloud is which you can set off a batch job to run when a sure motion happens. One instance is that after I’m performing penetration assessments, I drop recordsdata right into a bucket and that triggers processing recordsdata used to generate my closing report. I defined event-driven safety automation on this SANS white paper again in 2016:

Balancing Safety and Innovation With Occasion Pushed Automation

Simpler within the cloud: Though my work with batch jobs started years earlier than I grew to become closely concerned in cloud and you’ll positively make use of them on-premises, I discover it to be simpler in cloud environments. The cloud is ideal for batch jobs because it natively helps triggering jobs primarily based on occasions and scheduling. Every cloud supplier has a service for batch jobs, creatively known as “Batch” in every case:

Sarcastically, GCP introduced their “Batch” service on July 13, 2022, about 13 days after my final weblog submit stating that I used to be going to be writing extra about batch jobs and 5 days in the past. Good!

Lower your expenses (probably): One of many advantages of utilizing AWS Batch Jobs is which you can run them with spot situations. With spot situations, you’ll be able to bid on how a lot you’re keen to spend and your job will run each time AWS has further capability and is keen to decrease the price to your goal value level. A batch job is one thing that may run within the background. Typically it doesn’t matter when it’s run — like once you’re parsing by a bunch of logs for analysis functions. Different occasions it should matter — like when you’ll want to run finish of day funding processing earlier than the market opens within the morning. If you happen to don’t want your batch job to run at a particular time or inside a particular time interval, you’ll be able to probably avoid wasting cash processing massive quantities of information.

Cybersecurity Batch Jobs: What can you utilize a batch job for in cybersecurity? So many issues I couldn’t record all of them right here however listed here are a number of the issues I’m engaged on:

Parse information for cybersecurity metrics experiences.
Generate penetration testing experiences.
Kick off penetration testing processes.
Kick off setup of a brand new penetration atmosphere together with creating a brand new account as I did on this prior weblog submit:

If you happen to work in incident response and risk searching maybe you may use a batch job for the next:

Run automated incident response processes to seize incident information.
Course of and weed out false positives from incident findings, preserving the unique information however narrowing it right down to a smaller set of information for evaluation.
Automated remediation of findings in cloud accounts.
Automated evaluation of coverage violations to ship emails to offenders to repair their non-compliant assets and experiences to their managers.
Automated risk searching to seek for anomalies in logs.

If you happen to wished to course of information utilizing the strategies within the ebook I discussed firstly of this text, AWS has an instance of utilizing AWS Batch for Monte Carlo Simulations:

There are such a lot of methods you may use a batch job that you may most likely consider a large number of different use instances.

Suits and begins

As I launched into my journey to maneuver batch jobs to the cloud, I hit just a few points and nonetheless resolving a few of them. There’s the method of writing the code that you just wish to run, testing it, after which safe implementation of the batch job and storing the info securely. Because it seems, all of that could be a bit sophisticated, particularly once you attempt to introduce segregation of duties within the method wherein I’m trying to take action. I’m nonetheless investigating points I discovered equivalent to not having the ability to discover cross-account KMS key logging. It should be there someplace I simply want to search out it and can replace this submit after I do.

It doesn’t appear logical to make use of MFA with a batch job since they’re run with out human interplay however that doesn’t imply I don’t wish to do it. 🙂 Perhaps I wish to know if and when delicate operations get executed or I’m required to make use of MFA with a selected function. I’m engaged on a lot of associated objects proper now — and I nonetheless wish to get again to my dwelling networking weblog posts and safe code ebook. It’s all associated. Keep tuned when you’re .

Teri Radichel

If you happen to appreciated this story please clap and observe:

Medium: Teri Radichel or Electronic mail Checklist: Teri Radichel
Twitter: @teriradichel or @2ndSightLab
Requests providers by way of LinkedIn: Teri Radichel or IANS Analysis