14 Inquiries to Ask When Evaluating Information Lineage | by Prukalpa | Nov, 2022

Searching for an information lineage software? These are the important thing “gotchas” and options you ought to be asking about.

Information lineage is usually a mess.

Consider it like knitting a blanket. There are threads coming and going from each course, far too many to depend. All of those have to come back collectively completely in an intricate sample. In case you get it proper, it’s artwork. If even one aspect will get out of line, it’s chaos.

Lineage is difficult to get proper due to the sheer variety of variables at play — information flowing from a wide range of sources (each ever-changing previous ones and the newest new ones), transformations at each stage, complicated language concerned in naming and describing information property, completely different types for writing information logic and code, and far more.

As tough as that is, we will’t hand over. Lineage is indispensable within the information group toolbox, revealing information flows and powering essential use instances like influence evaluation, root trigger evaluation, governance, and compliance.

Listed here are 14 inquiries to ask throughout your seek for the precise information lineage software to completely assess its depth (variety of distinctive sources supported), breadth (variety of fields or objects supported for every supply), and utility (capability to energy insights and actions throughout various information personas).

Many information platforms present a lineage API, so it’s straightforward for any lineage system to ingest and use lineage from these sources. Nevertheless, not each platform does this. Computerized SQL parsing is essential to plug these gaps and make sure that your lineage is full, overlaying all information sources, processes, and property.

In case you solely parse SQL on the warehouse layer, SQL queries from sources with out native question historical past (e.g. relational databases like PostgreSQL and MySQL) will slip via the cracks.

Search for the power to learn a dump of SQL queries from supply methods that don’t possess a “question historical past” function.

To keep away from gaps in your lineage, it’s essential to parse and register lineage from numerous sorts of SQL statements:

• CREATE TABLE
• CREATE TABLE AS SELECT
• CREATE VIEW
• MERGE
• INSERT INTO
• UPDATE

Most SQL parsers assist SQL CREATE and, in some instances, MERGE statements. Nevertheless, many don’t assist INSERT INTO and UPDATE statements. These account for many transformations in information warehouses, so they’re essential for full lineage protection.

Search for lineage that may additionally parse MERGE, INSERT INTO, and UPDATE statements.

The info ecosystem is continually evolving, and new information sources are rising on a regular basis. Programmatically processing lineage from unsupported sources (by way of an open API) is essential to scaling lineage with out worrying about which new platforms you possibly can and may’t undertake.

Search for two key options:

• Potential to retrieve and create lineage programmatically by way of an API.
• Potential to publish and retrieve desk and column-level lineage throughout any object sort.

In lineage, coping with edge instances is the norm, and new edge instances usually require {custom} options or assist out of your lineage vendor.

Lineage is usually packaged as half of a bigger catalog or anomaly detection product. Typically this lineage is natively accessible and supported by the product’s group. Nevertheless, typically it comes by way of an exterior partnership, which may result in slower assist and fixes.

Search for three key options:

• Whether or not the lineage functionality is natively supported or externally offered.
• Whether or not the product’s group has direct management over the lineage growth.
• Clear SLAs for assist and an engineering dependency matrix (if there may be an exterior dependency).

Information transformation instruments and processes are at all times evolving. Whilst clients change from legacy stacks to the newest information instruments, lineage ought to at all times keep dependable.

Pulling lineage farther away from an information supply — e.g. from inside the transformation course of — can result in issues if the supply system adjustments. Pulling lineage from as near the supply as potential is usually safer and extra future-proof.

Search for lineage thats pull from a supply system’s question historical past (e.g. natively from Snowflake) relatively than integrating with a downstream transformation software or course of.

In lineage, it’s straightforward to finish up with large-scale SQL parsing calls for. (We’ve personally seen clients with over a million queries per day.) Parsing these queries takes important computational sources, so it’s essential that your lineage can sustain.

Cloud-native merchandise use the newest design patterns and microservices invented by corporations like Netflix for limitless scalability. Watch out for platforms that weren’t constructed for the cloud or have legacy tech debt — they are going to be laborious to keep up, resulting in efficiency issues as your lineage scales.

Search for fashionable, cloud-native structure that helps SQL parsing at scale.

Desk-level lineage is taken into account “desk stakes”, however column-level lineage must be too. It’s essential for a spread of use instances:

• Tracing delicate information classifications for remodeled PII information
• Affect evaluation from issues like schema adjustments
• Root trigger evaluation — e.g. investigating why a dashboard seems to be off by tracing a BI discipline to upstream columns within the information warehouse

With out the power to dive into granular columns or discipline lineage, information engineers and analysts might miss key depth throughout their investigations.

Search for two key options:

• Native column-level expertise within the UI, together with viewing graph linkages on the column stage.
• Assist for MERGE, INSERT INTO, and UPDATE SQL statements, that are key for column-level transformations.

Usually, the purpose of lineage is to establish why one thing on the final mile doesn’t look proper. Because the managers for firm information, information engineering groups are liable for ensuring the information that feeds end-user property is reliable and dependable. When this fails, lineage is a vital diagnostic software.

Not all lineage will natively hook up with your chosen BI software (e.g. Looker, Tableau, Energy BI, and so on). Some depend on time-consuming handbook scripts and asset pushing, even for main BI instruments.

Search for both native connectors or automated scripts that mechanically hook up with your BI software of alternative.

Anybody doing root trigger evaluation must dive into an incorrect discipline (i.e. dimension, measure, calculated discipline, and so on.) within the dashboard, and work backwards to zero in on the upstream fields or columns which can be damaged. That is solely potential with field-level lineage for the BI software.

Area-level lineage can also be essential for influence evaluation. If an information engineer is attempting to make a schema change, they should perceive the particular downstream columns and fields that shall be affected — not simply which dashboards shall be affected in some unspecified approach.

Some platforms assist lineage for a couple of fields, however don’t go deep with BI fields which can be essential for most of these evaluation.

Search for two key options:

• Protection of each column-level lineage for SQL sources and BI field-level lineage.
• Whether or not your BI software’s objects are supported and uncovered in lineage. (E.g. in Looker, will lineage cowl all of the fields/objects you care about, comparable to Dashboards, Seems, Explores, Tiles, Fields, and Views?)

We regularly hear that Salesforce is the “Wild West” and nobody is aware of what is going on with that information within the ETL pipeline. Nevertheless, opening up Salesforce (and different essential SaaS supply methods) is usually a game-changer for serving to information and enterprise groups to collaborate. Affect evaluation is a serious use case right here, since Salesforce fields get modified on a regular basis and wreak havoc downstream.

If it’s accessible, be sure to analyze the depth of Salesforce lineage. Some lineage begins on the storage layer (i.e. information warehouse, lake, and so on). Can lineage be generated upstream of the storage layer for SaaS instruments like Salesforce?

In that case, how deep does it go? Some methods can’t go all the way down to Salesforce object and discipline ranges, that are essential for making downstream lineage helpful and understanding context for downstream information property.

Search for object and field-level lineage from Salesforce all the way down to the information warehouse layer.

Constructing lineage upstream of an information warehouse is difficult. Doing so at scale, particularly should you draw information from a number of supply methods, is even tougher.

In case you observe an ELT method, it’s essential that your lineage can join with fashionable information integration instruments like Fivetran. This allows you to construct upstream lineage, creating true end-to-end lineage and displaying what occurs with information earlier than it enters the storage layer.

Search for whether or not it natively connects to Fivetran or different fashionable information integration instruments.

Spark lineage is tough to generate. However should you use Databricks, that is key to unlocking visibility into your transformations and creating usable lineage to assist information scientists, engineers, and analysts with ML and analytics workloads in Databricks.

Search for two key options:

• Whether or not it ingests lineage from Databricks’ Unity Catalog API (which incorporates Spark, Scala, and SQL)
• Whether or not it helps field-level lineage in BI instruments downstream of Databricks.

In isolation, lineage solely tells a part of the story and, due to this fact, solely supplies a part of the worth. Lineage turns into actionable when it’s mixed with key metadata and context:

• Operational metadata: How and when had been property orchestrated?
• High quality and anomaly metadata: What state are the property in? Are they dependable?
• Enterprise/semantic metadata: How do the property hyperlink to key enterprise phrases or KPIs?
• Proprietor and knowledgeable metadata: Who must you contact or collaborate with throughout troubleshooting?
• Social metadata: What’s the human context for this asset — e.g. related Slack discussions or Jira tickets in regards to the asset? That is what machines alone will miss.

Typically lineage graphs seem as one more a siloed view. With out the opposite metadata for these property, it may be laborious to place lineage in context.

Search for three key options:

• Openness: An “open by design”, extensible platform the place you possibly can harvest information and metadata from any supply by way of APIs (together with custom-built connectors).
• Flexibility: Assist for a variety of technical, operational, anomaly/high quality, and enterprise/semantic metadata from these sources.
• Personalization: A personalised information expertise, the place every persona sees the metadata that’s proper for them, relatively than drowning in all of the metadata.

Along with enabling information individuals’s work, lineage may allow automated system actions and workflows.

For instance, if an upstream desk has information high quality points, it’s essential to mechanically add bulletins to downstream BI dashboards. This retains enterprise customers from creating “Rubbish In, Rubbish Out” evaluation, and saves information analysts and engineers from manually sending alerts or warnings.

Some platforms don’t have the underlying structure and scalability to carry out automated actions based mostly on lineage.

Search for open APIs, the power to construct or customise automated workflows, and the power to learn metadata-change occasions and set off adjustments in linked property throughout the lineage graph.