ACM.96 Gotchas when attempting to create zero-trust insurance policies after which later deleting and recreating the insurance policies
This can be a continuation of my sequence of posts on Automating Cybersecurity Metrics.
Dependencies between CloudFormation stacks
What occurred after I tried to delete some stacks is that I couldn’t delete them as a result of different CloudFormation stacks reference that stack. I’ve to delete the factor referencing that stack earlier than I can delete the associated stack.
Then, after I tried to redeploy the stacks, I ended up with errors the place I couldn’t create a coverage as a result of one other useful resource didn’t exist, however I couldn’t create the useful resource with out the coverage. That’s a design challenge. Right here’s an instance:
- I gave IAM Directors permission to create SSH credentials and retailer them in Secrets and techniques Supervisor encrypted with a particular key.
- I can not create the IAM coverage as a result of it’s depending on a non-existent KMS key.
- However I can not create the KMS key as a result of I’ve not but created the KMS Admins.
- However I can not but create the KMS admins as a result of I can’t deploy the IAM admin coverage.
Separate KMS IAM Insurance policies and Deployment Order
I received round this downside however separating out all KMS permissions for every position right into a KMS particular coverage. These KMS particular insurance policies rely on the KMS keys to be created first.
- Deploy the IAM admins
- Deploy the non-KMS portion of the coverage
- Deploy the KMS admins with their KMS privileges
- Deploy the remainder of the IAM customers, roles, and teams
- Deploy the KMS keys which have dependencies on IAM sources
- Deploy the KMS insurance policies for every other roles that rely on the keys
- Deploy every other sources that rely on the keys (Secrets and techniques, EC2 occasion, Parameters, and many others.)
Overcoming dangers related to AWS altering ARNs in KSM insurance policies
I ended up including delete permissions for the “root” person in KMS key insurance policies as a result of it’s too straightforward to delete the admin person and then you definitely can not administer the KMS key. The foundation person permissions consists of the foundation person within the account that can not be deleted. (It additionally consists of directors as defined in a a lot earlier submit.) At the least that means, if I’ve an issue I can delete the important thing and re-add it if I can’t administer it.
The issue nonetheless exists, nonetheless, that you can lock your self out of all of your information in a manufacturing setting in case you unintentionally deleted all of the roles utilized in your AWS KMS insurance policies and nobody can encrypt or decrypt anymore with that key and nobody can alter the coverage. To me, the design that permits AWS to alter your KMS and belief insurance policies may be very harmful and I hope they cease doing that. I might need to think about different alternate options long run, however for my instant functions, I’m simply testing and need to be sure I can delete the important thing if I make a mistake.
Refactoring and lowering strains of code
Along with fixing the order of deletion of issues in my account I ended up refactoring my code to cut back strains of code and repetitive code I can utterly cut back repetitive strains calling a deletion script by passing in a listing of stacks I need to delete and looping via these gadgets to delete them.
Create the array and move it right into a operate:
Perform that loops via the array and calls the delete stack operate:
Delete stack operate:
I added the same assemble for the IAM profiles operate I wrote about within the final submit:
And I moved the features to a delete_functions.sh file to scrub issues up a bit. I additionally moved them right into a /Delete folder within the root of the repository.
Particular dealing with for KMS keys
KMS keys are tough as a result of in case you delete the important thing administrator earlier than the keys are deleted you won’t be able to manage the keys. I created a particular operate for deletion of KMS admins to verify if keys exist earlier than permitting the script to proceed. Observe that I’m excluding AWS-managed keys and keys which might be pending deletion. I’m additionally querying the Alias since I’ve been having points with the important thing description when utilizing the command line as outlined in prior posts.
After creating this verify it’s clear that my delete performance for KMS keys just isn’t working accurately:
These KMS keys are actually, actually giving me grief. I discover KMS to be essentially the most tough service to work with on AWS for the time being (if you’re attempting to create zero-trust insurance policies, in case you’re not its “straightforward.”)
Deleting KMS Keys created by CloudFormation
I can see the above keys aren’t getting deleted.
Right here’s my code to all delete_keys, with the record of key aliases handed into the delete_keys operate:
The Alias identify was used within the outputs of my CloudFormation stacks, so I can search for the Key ID utilizing that Alias identify by formulating the output identify produced by our Key.yaml CloudFormation template. The Alias can also be used within the stack identify for the important thing Aliases that we have to delete.
Right here’s the important thing deletion operate. What’s the issue? I’m printing out my command line calls so I can see and replicate them.
As I defined within the final submit I added if-then statements to skip to the portion of the script that’s giving me grief. It’s not fairly but it surely will get the job carried out.
What I discover after I run my script is that I by no means get the output for the command strains which might be imagined to be executed, so they’re by no means being run.
The script solely runs if it could discover the important thing id from the CloudFormation stack related to the alias we move in:
Aha. I known as the stack “aliasstack” however then handed in “stack” — another excuse variable and kind checking in languages is useful however anyway let’s repair that.
Then I get an error that my “delete” person which is an administrator just isn’t allowed to delete the important thing. That is smart as a result of our preliminary key insurance policies solely all of the KMS Admins position to manage the important thing.
Let’s change the profile used to run this code to our KMS profile. We are able to change it again to our delete profile which makes use of our root person after fixing the above KMS insurance policies, however proper now I’m nonetheless attempting to delete the previous keys with the prior key coverage the place the foundation person can not delete them.
Now I run into two issues.
First, the DeveloperSecrets alias just isn’t discovered as a result of it was already deleted.
Once I verify in my account, this key has in truth been deleted. I’ve to take care of this error message in some way.
Secondly, the KMS person can’t record keys. We should always in all probability add that however for now I’m going to alter the profile within the KMS admin operate to my “delete” person.
Now first, I believed including || true would repair my downside and permit the script to proceed but it surely didn’t. Once I swallow the error I can’t see if I’ve permissions points. I assume I ought to chunk the bullet and write a “key_exists” operate as my present method just isn’t working.
That received me previous my CLI features however then the code tries to delete the CloudFormation stacks and one thing with that’s failing. It seems just like the script is attempting to delete the CloudFormation key stack however just isn’t capable of:
Heading over to CloudFormation I see the next error:
Discover KMS within the coverage identify. I don’t need to delete my IAM administrator till the very finish of my delete script. What I did as you’ll be able to see above is put the KMS permissions in a separate coverage that I can delete previous to deletion of the KMS keys.
For the time being I see two KMS coverage stacks I beforehand created whereas testing re-deployment:
I can add the next code to delete them:
Passing arrays to features in bash
After additional testing, I noticed that solely the primary merchandise in my array was making it via as to the operate I’m calling after I print out my record of stacks every time the delete_stacks operate is known as as proven beneath.
Because it seems you’ll be able to solely ship the expanded values (a listing of strings) to a operate in bash. You try this by passing the operate identify adopted by [@].
Then within the known as operate, reconstruct the array utilizing the record:
Now I get each my KMS insurance policies that I have to delete:
Logic for Deleting KMS Keys
KMS Keys are cumbersome to handle if you’d like zero-trust insurance policies. That doesn’t imply you shouldn’t use zero-trust insurance policies, it implies that it’s a must to watch out with them. And maybe AWS will ultimately make this simpler.
For some cause I’m presently not remembering I discovered that I wanted to individually name instructions to schedule key and alias deletion on high of the CloudFormation stack deletion. I’m unsure why however this received my script working. I’ll have to revisit this and see if the separate CLI instructions are required as soon as I add my new insurance policies to the keys that can permit my delete CLI profile to delete keys. I additionally want so as to add KMS list-keys to my KMS Admin IAM profile.
I had numerous different issues with the logic and order of issues, and accurately acquiring the KMS key. As soon as the stack was deleted I couldn’t pull the KMS Key ID and the CloudFormation alias stacks didn’t delete for some cause. I feel I had a bug in there someplace that brought about my script to fail earlier than it received to that time. As soon as the important thing was gone my logic by no means received to the alias stack deletion command, so I needed to rearrange issues.
For the time being my KMS key deletion operate seems like this:
I additionally created a separate operate to loop via and delete any aliases for the important thing:
Lastly, lastly, lastly! My KMS keys are all scheduled to delete and the aliases and CloudFormation stacks are gone.
Deleting Sources is Not Easy
I bear in mind after I was on the GDPR committee for an organization and I discovered concerning the information deletion necessities. I knew the VP now CIO and knew that once they advised him he would wish to delete sure sources for sure timeframes he would suppose that was easy. The corporate needed to meet sure deadlines. I made certain to go speak to him and clarify the complexities and to ensure he gave his staff sufficient time.
Having written deletion routines for a financial institution, I understood the complexities of information deletion scripts. Should you delete the improper information, that may flip right into a nightmare. Deletion scripts can take a very long time to run, or affect system efficiency.
Knowledge deletion scripts are straightforward to put in writing. Like “Drop Desk x” in a database. Blam. All of your information is gone. The issue and complexities come into play when it’s best to solely delete sure information — akin to the shoppers who’ve requested to delete themselves out of your methods.
Let’s say you don’t have correct integrity checks in your system — form of like our beautiful KMS keys that help you delete key directors when the important thing nonetheless exists. You’ll be able to find yourself with orphan information which might be not possible to decipher as a result of they’ve an ID however no mum or dad file indicating the shopper to which that information belongs. Is that OK? Discuss to your lawyer. GDPR is generally a query for legal professionals and the tech groups write code aligning to authorized specs.
In our case, the complexity lies within the dependencies that received’t allow us to delete issues except we delete in exactly the right order. There’s additionally the truth that if we prematurely delete permissions we would get caught with useful resource in our cloud account we can not handle.
That deletion script that I used to be “simply” going to rapidly create now consists of two scripts that appear to be this — and I’m not carried out.
delete.sh
delete_functions.sh
Now I “simply” have to recollect so as to add every new stack I create to this delete script.
I additionally anticipate to maneuver the deletion performance into completely different useful resource stacks the best way I did with my check scripts ultimately.
Finally I’ll add customers and networking.
Now, can I get again to what I used to be doing earlier than I used to be interrupted by a mangled KMS coverage because of the best way AWS handles ARNs for deleted roles in KMS insurance policies??
The dreaded “Entry to KMS just isn’t allowed” error message:
One strategy to doubtlessly repair it — in case you haven’t deleted your key administrator.
In my case, I believed it could be quicker simply to start out every part and begin over as a result of I received right into a round loop that was proving difficult to handle. I feel it might have been the identical challenge — besides in an IAM belief coverage. I don’t even bear in mind.
I can also’t bear in mind precisely what I used to be attempting to do earlier than that for the time being as a result of this took about three days to finish in between consulting calls, home initiatives, and weekend actions. On the time of this writing I’m about two weeks forward on blogs posts so by the point you learn this it might not be per week finish anymore. 🙂 I’ll determine it out tomorrow.
Comply with for updates.
Teri Radichel
Should you favored this story please clap and comply with:
Medium: Teri Radichel or E mail Listing: Teri Radichel
Twitter: @teriradichel or @2ndSightLab
Requests providers through LinkedIn: Teri Radichel or IANS Analysis
© 2nd Sight Lab 2022
All of the posts on this sequence:
____________________________________________
Creator:
Cybersecurity for Executives within the Age of Cloud on Amazon
Want Cloud Safety Coaching? 2nd Sight Lab Cloud Safety Coaching
Is your cloud safe? Rent 2nd Sight Lab for a penetration check or safety evaluation.
Have a Cybersecurity or Cloud Safety Query? Ask Teri Radichel by scheduling a name with IANS Analysis.
Cybersecurity & Cloud Safety Sources by Teri Radichel: Cybersecurity and Cloud safety courses, articles, white papers, shows, and podcasts
