Puppet 6 introduced the power to defer features to runtime on the agent, and now we have launched enhancements that make this simpler to do. Learn on to seek out out extra and to verify your modules are able to be deferred.
What are Puppet features once more?
First, let’s do a really temporary recap. Puppet features are bits of code which can be executed throughout catalog compilation. They will do varied issues, akin to together with different courses within the catalog or marking assets in a given scope as no-op, however the sorts of features we’re concerned with in the present day are people who return a worth. For instance, fqdnrand() will deterministically generate a random quantity and shellquote() will remodel its enter and return a string secure to execute by shells like Bash.
The values returned by these features are compiled instantly into the catalog and develop into as static as the remainder of the doc. In different phrases, in case you examine a catalog generated for a particular node, you possibly can see precisely what random quantity was used to specify the schedule for a cron job, or you possibly can see precisely how a shell command was quoted or escaped earlier than being invoked. And since they’re immutable, that catalog will all the time end in the identical desired state being enforced.
However in some circumstances, it could be extra helpful to defer that to the Puppet agent to execute as a part of the catalog enforcement. For instance, many infrastructures mandate that secrets and techniques be supplied by one thing like a extremely protected secrets and techniques server slightly than funneling them by way of a configuration administration server. In instances like this, we care much less in regards to the particular worth being enforced than we do about what it represents. We do not want the Puppet server to really know the database account password a node is configured with. We simply must instruct the agent to place the password, no matter it resolves to, within the acceptable configuration file.
In instances like this, you possibly can as an alternative compile a reference to the perform itself into the catalog and instruct the agent to invoke it at runtime like so:
class { 'profile::myappstack':
db_adapter => 'postgresql',
db_address => 'pgsql.instance.com',
db_password => Deferred(
'vault_lookup::lookup',
['appstack/dbpass', 'https://vault.example.com']
),
}
Now, as an alternative of the catalog containing the password and the Puppet server getting access to the password, the agent itself will retrieve the password instantly from Vault utilizing its personal registered credentials. This enables the infrastructure admins to have rather more fine-grained management over the entry to delicate info and to rapidly rotate or revoke them as wanted.
Any perform that returns a worth might be deferred on this method, and in lots of instances it is simply this straightforward. There are some challenges although, some which have been simplified by current Puppet updates. We’ll speak about these first.
Puppet language updates
In Puppet’s first implementation, the catalog was successfully pre-processed to resolve deferred features into values. Which means earlier than the catalog was enforced, the agent would scan by way of it and invoke every deferred perform. The worth returned could be inserted into the catalog rather than the perform. Then the catalog could be enforced as typical. The issue with this strategy is that if the perform relied on any tooling put in as a part of the Puppet run, then it could fail on the primary run as a result of it was invoked previous to set up. If the perform did not gracefully deal with lacking tooling, it might even forestall the catalog from being enforced in any respect.
As of Puppet 7.17, features can now be lazily evaluated with the brand new preprocess_deferred setting. This instructs the agent to resolve deferred features throughout enforcement as an alternative of earlier than. In different phrases, in case you use customary Puppet relationships to make sure that tooling is managed previous to courses or assets that use the deferred features utilizing that tooling then it would function as anticipated and the perform will execute correctly.
Puppet 7.17 additionally improves the best way typed class parameters are checked. The information sort of a deferred perform is Deferred, and older variations of Puppet would really use that sort when checking class signatures. For instance, if the profile::myappstack class we referenced within the instance above specified that the db_password parameter ought to be a String, then the instance would have failed as a result of the Deferred sort wouldn’t match the anticipated String sort.
As of Puppet 7.17, that is now not an issue. Deferred features are introspected and the return sort they declare can be used for sort matching. If the perform would not explicitly declare a return sort, Puppet will print a warning, however the compilation will succeed. No code modifications are required to make the most of this enchancment, however in case you’re writing courses that is likely to be used with older Puppet variations, you would possibly think about using a variant datatype akin to Variant[String, Deferred] for parameters which can be anticipated to be deferred.
class profile::myappstack(
String db_adapter,
String db_address,
Variant[String, Deferred] db_password,
) { ...
The third, and possibly most difficult, concern is that relying on how authors write their modules, chances are you’ll or could not be capable of move deferred features in as parameters to many fashionable Forge modules. Let us take a look at some examples and discover ways to anticipate them and future proof our personal modules for deferred features.
There are 4 main causes of incompatibility, and possibly different variations that observe comparable patterns. We’ll begin with the only and work in direction of essentially the most advanced.
Downside #1: Puppet language features can’t be deferred
As of Puppet 4.2, many features might be written instantly within the Puppet language slightly than in Ruby. Capabilities like this are sometimes used to remodel knowledge, akin to this instance from docs that turns an ACL listing right into a useful resource hash for use with the create_resources() perform. As a result of these features aren’t pluginsynced to the agent, they can’t be deferred. Normally, this is not a lot of a priority as a result of operations like connecting to a Vault server can’t be achieved simply within the Puppet language in any case. However in case you do have such a necessity, then this perform must be rewritten within the Ruby language.
Downside #2: strings and useful resource titles can’t be deferred
A price that comes from a deferred perform can’t be utilized in a useful resource title or interpolated right into a string. For instance, for example that you simply added debugging code to the myappstack profile to see what password the Vault server was returning.
class profile::myappstack(
String db_adapter,
String db_address,
String db_password,
) {
notify { "Password: ${db_password}": }
#...
}
As an alternative of the password you count on to see, it is the textual content type of the Deferred perform!
$ puppet agent -t
…
Discover: Password: Deferred({'title' =>'vault_lookup::lookup', 'arguments' => ['appstack/dbpass', 'https://vault.example.com']})
And in case you wrote it with out string interpolation, like notify { $db_password: }, then it could fail compilation fully and offer you a seemingly nonsensical error which may give seasoned C++ programmers template flashbacks.
Error: Analysis Error: Unlawful title sort at index 0. Anticipated
String, bought Object[{name => 'Deferred', attributes => {'name' =>
Pattern[/A[$]?[a-z][a-z0-9_]*(?:::[a-z][a-z0-9_]*)*z/], 'argum
ents' => {sort => Non-obligatory[Array], worth => undef}}}] (file: /Us
ers/ben.ford/tmp/deferred.pp, line: 6, column: 12) on node arach
ne.native
The answer to this downside is that variables you count on to be deferred shouldn’t be used as useful resource titles or in interpolated strings. The notify on this instance ought to be refactored like so:
notify { 'vault server debugging':
# We can not interpolate a string with a deferred worth
# as a result of that interpolation occurs throughout compilation.
message => $db_password,
}
If that you must interpolate a deferred worth right into a string, you are able to do that by deferring the sprintf() perform. For instance, you would write that notify like so:
notify { 'vault server debugging':
# Defer interpolation to runtime after the worth is resolved.
message => Deferred(
'sprintf',
['Password: %s', $db_password]
),
}
Do not forget to take away this message as soon as the Vault downside has been resolved so it would not leak your secrets and techniques!
Downside #3: perform arguments can (often) not be deferred
Carefully associated to the primary downside, perform arguments can’t be deferred, until the perform is designed for it. Capabilities are evaluated throughout compilation, so in case you defer an argument they will function on a Ruby object as an alternative of the resolved worth. That is often seen when making an attempt to render templated information. For instance, if that myappstack profile managed a configuration file with a template, it could embrace the identical textual content type of the deferred vault lookup perform as above:
$ cat /and so on/myappstack/db.conf
dbpassword = Deferred({'title' =>'vault_lookup::lookup', 'arguments' => ['appstack/dbpass', 'appstack/dbpass']})
With a purpose to correctly deal with deferred features, the perform utilizing them should even be deferred. For instance, you would defer the rendering of the database configuration file utilizing the brand new deferrable_epp() perform that defers template rendering when wanted. Utilizing that perform to generate templated information means that you can transparently deal with deferred parameters. This perform is offered beginning in puppetlabs-stdlib model 8.4.0. Word that it requires you to explicitly move within the variables you will be utilizing within the template.
If that you must assist earlier variations of stdlib, you then’ll want to write down the boilerplate logic your self, which could look one thing like this:
$variables = { 'password' => $db_password }
if $db_password.is_a(Deferred) {
$content material = Deferred(
'inline_epp',
[find_template('profiles/myappstack.db.epp').file, $variables],
)
}
else {
$content material = epp('profiles/myappstack.db.epp', $variables)
}
file { '/path/to/configfile':
guarantee => file,
content material => $content material,
}
Downside #4: deferred values can’t be used for logic
A price that is not identified till runtime can’t be used to make conditional selections, since all logic is resolved throughout compilation. An instance of that is the puppetlabs-postgresql module, which dealt with supplied password hashes in a different way primarily based on the algorithm used to create them. If the consumer expects that password hash to be supplied at runtime by a secret server, then it is not identified at runtime and the compiler cannot select the suitable codepath.
The decision for this type of downside is to refactor so these selections do not have to be made throughout compilation, or in order that completely different knowledge is used to make selections. Within the case of our PostgreSQL module, we refactored that code in order that the entire codepaths affected by that conditional have been all deferred. The identical code will run, however it would all be evaluated throughout runtime on the agent.
Relying on the kind of choice to be made, you would additionally refactor into utilizing info, that are evaluated on the agent previous to catalog compilation after which making conditional selections primarily based on the resolved values of the info.
Abstract
I am positive you see a typical thread in every of those downside eventualities. Values calculated by deferred features are merely not identified at compile time. Which means nothing processed throughout compilation can use them. You can not use a deferred worth to interpolate right into a string, or use it as a key for a selector, or make a logical choice with. You can not render it instantly right into a templated file, as an alternative that you must embrace the template supply within the catalog and compile it at runtime.
Successfully, a deferred perform is simply helpful when passing the worth it generates on to a parameter of a declared useful resource, and modules have to be written to take this into consideration. Module authors ought to anticipate that individuals would possibly wish to defer sure parameters, akin to passwords or tokens or different secret values and deal with these instances by refactoring any use of those values out of compile time and into runtime.
Ecosystem updates that simplify Deferred use instances:
- Set
preprocess_deferredwhen your features depend upon tooling put in by the Puppet run. - Deferred features are interpolated in order that their return varieties can be utilized to match knowledge varieties required by class signatures.
- The brand new
deferrable_epp()perform will routinely defer epp template rendering when acceptable.
Good luck! We’re all the time excited to see the cool stuff you construct.
Ben is the Neighborhood and DevRel lead at Puppet.
