Literate programming in Go | InfoWorld

Throughout a latest firm hackathon, my group’s constitution was to enhance the documentation for the Steampipe plugin SDK. Like different elements of the Steampipe system, the plugin SDK is written in Go and printed to pkg.go.dev. The model that existed after we began is right here. As is typical, the documentation was an autogenerated catalog of features and kinds. To clarify how you can use these features and kinds, we supplied steerage on the steampipe.io website.

Our hackathon problem was to weave such steerage into the generated documentation. In the event you’re writing a Steampipe plugin, your record of duties appears to be like like this:

1. Outline the plugin
2. Create the plugin entry level
3. Outline your first desk
4. …

These duties require that you just use features and kinds, however whereas feedback connected to these features and kinds can improve the generated documentation, they’re too granular for the high-level exposition we aimed for. Looking for inspiration, Steampipe lead developer Kai Daguerre seen that our top-level web page lacked the overview part he noticed within the documentation for pgx, a Go driver for Postgres.

That overview comes from https://github.com/jackc/pgx/blob/grasp/doc.go, which is one lengthy remark (that makes use of Go remark syntax) adopted by a package deal declaration.

So we added a doc.go on the prime degree of our repo to supply this overview.

IDG

Writing a wiki in Go feedback

We used the identify doc.go as a result of that appears to be standard, nevertheless it may have been referred to as foo.go. What’s salient is that it is a legitimate Go file that belongs to a package deal. The package deal accommodates no code, solely documentation. We wrote headers to create the sections of the overview, and in every part we described a plugin author’s job utilizing narrative, inline code examples, inner hyperlinks to features and kinds, and exterior hyperlinks to examples elsewhere.

The power to hyperlink throughout the code’s namespace was a revelation. To explain the duty referred to as Add hydrate features, for instance, we wrote this:

# Add hydrate features

A column could also be populated by a Record or Get name. If a column requires knowledge not present by Record 
or Get, it could outline a [plugin.HydrateFunc] that makes a further API name for every row.
Add a hydrate perform for a column by setting [plugin.Column.Hydrate].

The part outlined by the Add hydrate features header is a hyperlink goal: Add hydrate features. And the bracketed objects render as hyperlinks to a sort, plugin.HydrateFunc, and to a property, plugin.Column.Hydrate. This was beginning to really feel like wiki writing!

What we nonetheless lacked, although, was the flexibility to create new wiki pages the place we may clarify higher-level ideas. The overview was one place to do this, however we needed to order that for narration of the plugin author’s journey. The place may we focus on an idea like dynamic tables, a sophisticated characteristic that allows plugins like CSV which don’t have any fastened schema and should outline columns on the fly?

Kai realized that not solely may we create new documentation-only packages for such subjects, we may additionally import them in order that their names had been out there for a similar form of shorthand linking we may do with, e.g., [plugin.HydrateFunc]. Within the top-level doc.go he did this:

package deal steampipe_plugin_sdk

import (
	"github.com/turbot/steampipe-plugin-sdk/v5/docs/dynamic_tables"
)

var forceImportDynamicPlugin dynamic_tables.ForceImport

And in /docs/dynamic_tables/doc.go he did this:

package deal dynamic_tables

sort ForceImport string

Now we may write a sentence in a remark like this:

/*
Use [dynamic_tables] while you can not know a desk's schema prematurely.
*/
package deal steampipe-plugin-sdk

And the bracketed time period autolinks similar to some other identify within the code’s namespace.

If that looks like extra bother than it is price, you may skip the import gymnastics and simply use an exterior hyperlink:

/*
Use [dynamic_tables] while you can not know a desk's schema prematurely.


[dynamic_tables]: https://pkg.go.dev/github.com/turbot/steampipe-plugin-sdk/v5/docs/dynamic_tables
*/
package deal steampipe-plugin-sdk

Both approach, the authoring expertise now feels extra totally wiki-like. You’ll be able to create new pages as wanted, and weave them into the hypertextual documentation that lives throughout the code base.

It was, admittedly, a little bit of a battle to make use of the Go system within the methods described right here. The remark syntax is Markdown-like however frustratingly not Markdown; it is determined by many implicit formatting conventions. In the event you go this route you may want a neighborhood previewing software, and it isn’t super-obvious that the one you need isn’t godoc however relatively pkgsite. Had we found the command go set up golang.org/x/pkgsite/cmd/pkgsite@newest we’d have saved ourselves lots of grief.

How is that this literate programming?

It is not! In his eponymous paper Knuth wrote:

The practitioner of literate programming could be thought to be an essayist, whose important concern is with exposition and excellence of favor. Such an creator, with thesaurus in hand, chooses the names of variables rigorously and explains what every variable means. She or he strives for a program that’s understandable as a result of its ideas have been launched in an order that’s greatest for human understanding, utilizing a mix of formal and casual strategies that reinforce one another.

To help this apply he invented a system referred to as internet whose elements, tangle and weave, enabled the creator of a program to inform its story in a language that blended code (initially, Pascal) and documentation (initially, TeX) in a narrative-first approach. Our fashionable methods of blending code and documentation, and producing docs from embedded feedback, solely superficially resemble Knuth’s apply, as Mark Jason Dominus identified in his essay POD isn’t Literate Programming (2000).

And but, now that our hackathon train has given me a style of what code-as-wiki could be, it does really feel like a step towards the storytelling method that Knuth advocates. Recall that he named his system internet earlier than there was the net we now inhabit, by no means thoughts wikis! I can not declare that we’re now literate programmers within the Knuthian sense, and I’ve at all times questioned if solely a Knuthian thoughts may implement that unique imaginative and prescient. However this fashion of bettering the narrative high quality of autogenerated docs does really feel like a step in the proper route.