Temporal Fusion Transformer: Time Sequence Forecasting with Deep Studying — Full Tutorial | by Nikos Kafritsas | Nov, 2022

Create correct and interpretable predictions

In accordance with [2], Temporal Fusion Transformer outperforms all outstanding Deep Studying fashions for time sequence forecasting.

Together with a featured Gradient Boosting Tree mannequin for tabular time sequence information.

However what’s Temporal Fusion Transformer (TFT)[3] and why is it so fascinating?

On this article, we briefly clarify the novelties of Temporal Fusion Transformer and construct an end-to-end venture on Vitality Demand Forecasting. Particularly, we are going to cowl:

• The right way to put together our information for the TFT format.
• The right way to construct, practice, and consider the TFT mannequin.
• The right way to get predictions on validation information and out-of-sample predictions.
• The right way to calculate function importances, seasonality patterns, and excessive occasions robustness utilizing the built-in mannequin’s interpretable consideration mechanism.

Let’s dive in!

For an in-depth evaluation of the Temporal Fusion Transformer structure, examine my earlier article.

Temporal Fusion Transformer (TFT) is a Transformer-based mannequin that leverages self-attention to seize the advanced temporal dynamics of a number of time sequences.

TFT helps:

• A number of time sequence: We are able to practice a TFT mannequin on hundreds of univariate or multivariate time sequence.
• Multi-Horizon Forecasting: The mannequin outputs multi-step predictions of a number of goal variables — together with prediction intervals.
• Heterogeneous options: TFT helps many kinds of options, together with time-variant and static exogenous variables.
• Interpretable predictions: Predictions will be interpreted when it comes to variable significance and seasonality.

A type of traits is exclusive to Temporal Fusion Transformer. We’ll cowl this within the subsequent part.

Amongst notable DL time-series fashions (e.g., DeepAR[4]), TFT stands out as a result of it helps varied kinds of options. These are:

• Time-varying recognized
• Time-varying unknown
• Time-invariant actual
• Time-invariant categorical

For instance, think about we’ve got a gross sales forecasting case:

Let’s say we’ve got to foretell the gross sales of three merchandise. The num gross sales is the goal variable. The CPI index or the variety of guests are time-varying unknown options as a result of they’re solely recognized as much as prediction time. Nevertheless, holidaysand particular days are time-varying recognized occasions.

The product id is a time-invariant (static) categorical function. Different options that are numerical and never time-dependent resembling yearly_revenue will be categorized as time-invariant actual.

Earlier than shifting to our venture, we are going to first present a mini-tutorial on find out how to convert your information to the prolonged time-series format.

Notice: All photos and figures on this article are created by the writer.

For this tutorial, we use the TemporalFusionTransformer mannequin from the PyTorch Forecasting library and PyTorch Lightning:

pip set up torch pytorch-lightning pytorch_forecasting

The entire course of includes 3 issues:

1. Create a pandas dataframe with our time-series information.
2. Wrap our dataframe right into a TimeSeriesDataset occasion.
3. Cross our TimeSeriesDataset occasion to TemporalFusionTransformer.

The TimeSeriesDataset may be very helpful as a result of it helps us specify whether or not options are time-varying or static. Plus, it’s the one format that TemporalFusionTransformer accepts.

Let’s create a minimal coaching dataset to indicate how TimeSeriesDataset works:

We should always format our information within the following method: Every coloured field represents a special time sequence, represented by its group worth.

Crucial column of our dataframe is the time_idx — it determines the sequence of samples. If there aren’t any lacking observations, the values ought to enhance by +1 for every time-series.

Subsequent, we wrap our dataframe right into a TimeSeriesDataset occasion:

All arguments are self-explanatory: The max_encoder_length defines the lookback interval and max_prediction_length specifies what number of datapoints shall be predicted. In our case, we glance again 3 time steps prior to now to output 2 predictions.

The TimeSeriesDataset occasion now serves as a dataloader. Let’s print a batch and examine how our information shall be handed to TFT:

This batch incorporates the coaching values [0,1] from the primary time-series (group 0) and the testing values[2,3,4]. When you rerun this code, you’re going to get totally different values as a result of the information are shuffled by default.

Our venture will use the ElectricityLoadDiagrams20112014 [5] dataset from UCI. The pocket book for this instance will be downloaded from right here:

This dataset incorporates the facility utilization (in KWs) of 370 shoppers/shoppers with a 15-minute frequency. The info span 4 years (2011–2014).

Some shoppers had been created after 2011, so their energy utilization initially is zero.

We do information preprocessing based on [3]:

• Combination our goal variable power_usage by hour.
• Discover the earliest date for each time-series the place energy is non-zero.
• Create new options : month, day, hour and day_of_week.
• Choose all days between 2014–01–01 and 2014–09–07.

Let’s begin:

Obtain Information

wget https://archive.ics.uci.edu/ml/machine-learning-databases/00321/LD2011_2014.txt.zip
!unzip LD2011_2014.txt.zip

Information Preprocessing

Every column represents a client. Most preliminary power_usage values are 0.

Subsequent, we mixture to hourly information. As a result of mannequin’s measurement and complexity, we practice our mannequin on 5 shoppers solely (for these with non-zero values).

Now, we put together our dataset for the TimeSeriesDataset format. Discover that every column represents a special time-series. Therefore, we ‘soften’ our dataframe, so that every one time-series are stacked vertically as an alternative of horizontally. Within the course of, we create our new options.

The ultimate preprocessed dataframe known as time_df. Let’s print its contents:

The time_df is now within the correct format for the TimeSeriesDataset. As you could have guessed by now, because the granularity is hourly, the hours_from_start variable would be the time index.

Exploratory Information Evaluation

The selection of 5 shoppers/time-series shouldn’t be random. The energy utilization of every time-series has totally different properties, such because the imply worth:

time_df[[‘consumer_id’,’power_usage’]].groupby(‘consumer_id’).imply()

Let’s plot the primary month of each time-series:

There is no such thing as a noticeable pattern, however every time-series has barely totally different seasonality and amplitude. We are able to additional experiment and examine stationarity, sign decompositions, and so forth, however in our case, we concentrate on the model-building facet solely.

Additionally, discover that different time-series forecasting strategies like ARIMA should fulfill just a few necessities (as an example, the time-series should first turn into stationary.) With TFT, we are able to depart our information as-is.

Create DataLoaders

On this step, we go our time_df to the TimeSeriesDataSet format which is immensely helpful as a result of:

• It spares us from writing our personal Dataloader.
• We are able to specify how TFT will deal with the dataset’s options.
• We are able to normalize our dataset with ease. In our case, normalization is necessary as a result of all time sequences differ in magnitude. Thus, we use the GroupNormalizer to normalize every time-series individually.

Our mannequin makes use of a lookback window of 1 week (7*24) to foretell the facility utilization of the subsequent 24 hours.

Additionally, discover that the hours_from_start is each the time index and a time-varying function. The power_usage is our goal variable. For the sake of demonstration, our validation set is the final day:

Baseline Mannequin

Subsequent, the step that nearly everybody forgets: A baseline mannequin. Particularly in time-series forecasting, you’ll be shocked at how usually a naive predictor outperforms even a fancier mannequin!

As a naive baseline, we predict the facility utilization curve of the day before today:

Coaching the Temporal Fusion Transformer Mannequin

We are able to practice our TFT mannequin utilizing the acquainted Coach interface from PyTorch Lightning.

Discover the next issues:

• We use the EarlyStopping callback to watch the validation loss.
• We use Tensorboard to log our coaching and validation metrics.
• Our mannequin makes use of Quantile Loss — a particular kind of loss that helps us output the prediction intervals. For extra on the Quantile Loss perform, examine this text.
• We use 4 consideration heads, like the unique paper.

We at the moment are able to construct and practice our mannequin:

That’s it! After 6 epochs, EarlyStopping kicks in and halts coaching.

Load and Save the Finest Mannequin

Don’t neglect to save lots of your mannequin. Though we are able to pickle it, the most secure choice is to save lots of the most effective epoch straight:

!zip  -r mannequin.zip lightning_logs/lightning_logs/version_1/*

To load the mannequin once more, unzip mannequin.zip and execute the next — simply keep in mind the most effective mannequin path:

Test Tensorboard

Take a more in-depth have a look at coaching and validation curves with Tensorboard:

Mannequin Analysis

Get predictions on the validation set and calculate the common P50 (quantile median) loss:

The final 2 time-series have a bit greater loss as a result of their relative magnitude can be excessive.

Plot Predictions on Validation Information

If we go the mode=uncooked on the predict() methodology, we get extra data, together with predictions for all seven quantiles. We even have entry to the eye values (extra about that later).

Take a more in-depth have a look at the raw_predictions variable:

We use the plot_prediction() to create our plots. After all, you possibly can make your personal customized plot — the plot_prediction() has the additional good thing about including the eye values.

Notice: Our mannequin predicts the subsequent 24 datapoints in a single go. This isn’t a rolling forecasting situation the place a mannequin predicts a single worth every time and ‘stitches’ all predictions collectively.

We create one plot for every client (5 in whole).

The outcomes are fairly spectacular.

Our Temporal Fusion Transformer mannequin was capable of seize the behaviour of all 5 time-series, when it comes to each seasonality and magnitude!

Additionally, discover that:

• We didn’t carry out any hyperparameter tuning.
• We didn’t implement any fancy function engineering approach.

In a subsequent part, we present find out how to enhance our mannequin with hyperparameter optimization.

Plot Predictions For A Particular Time Sequence

Beforehand, we plot predictions on the validation information utilizing the idx argument, which iterates over all time-series in our dataset. We will be extra particular and output predictions on a selected time-series:

In Determine 7, we plot the day-ahead of MT_004 client for time index=26512.

Bear in mind, our time-indexing column hours_from_start begins from 26304 and we are able to get predictions from 26388 onwards (as a result of we set earlier min_encoder_length=max_encoder_length // 2 which equals 26304 + 168//2=26388

Out-of-Pattern Forecasts

Let’s create out-of-sample predictions, past the ultimate datapoint of validation information — which is 2014–09–07 23:00:00

All we’ve got to do is to create a brand new dataframe that incorporates:

• The variety of N=max_encoder_length previous dates, which act because the lookback window — the encoder information in TFT terminology.
• The longer term dates of measurement max_prediction_length for which we need to compute our predictions — the decoder information.

We are able to create predictions for all 5 of our time-series, or only one. Determine 7 reveals the out-of-sample predictions for client MT_002:

Correct forecasting is one factor, however explainability additionally issues rather a lot these days.

And it’s even worse for Deep Studying fashions, that are thought-about black bins. Strategies resembling LIME and SHAP can present explainability (to some extent) however don’t work properly for time-series. Plus, they’re exterior post-hoc strategies and usually are not tied to a specific mannequin.

Temporal Fusion Transformer gives three kinds of interpretability:

• Seasonality-wise: TFT leverages its novel Interpretable Multi-Head Consideration mechanism to calculate the significance of previous time steps.
• Characteristic-wise: TFT leverages its Variable Choice Community module to calculate the significance of each function.
• Excessive occasions robustness: We are able to examine how time sequence behave throughout uncommon occasions

If you wish to be taught in-depth in regards to the interior workings of Interpretable Multi-Head Consideration and Variable Choice Community, examine my earlier article.

Seasonality-wise Interpretability

TFT explores the eye weights to grasp the temporal patterns throughout previous time steps.

The grey strains in all earlier plots signify the eye scores. Take a look at these plots once more — do you discover something? Determine 8 reveals the findings of Determine 7 and likewise accounts for the eye scores:

The eye scores reveal how impactful are these time steps when the mannequin outputs its prediction. The small peaks replicate the day by day seasonality, whereas the upper peak in direction of the top most likely implies the weekly seasonality.

If we common the eye curves throughout all timesteps and time-series (not simply the 5 ones we used on this tutorial), we are going to get the symmetrically-looking form in Determine 9 from the TFT paper:

Query: What good is that this? Can’t we merely estimate seasonality patterns with strategies resembling ACF plots, time sign decomposotion and so forth. ?

Reply: True. Nevertheless, learning the eye weights of TFT has additional benefits:

1. We are able to verify our mannequin captures the obvious seasonal dynamics of our sequences.
2. Our mannequin might also reveal hidden patterns as a result of the eye weights of the present enter home windows think about all previous inputs.
3. The eye weights plot shouldn’t be the identical as an autocorrelation plot: The autocorrelation plot refers to a specific sequence, whereas the eye weights right here concentrate on the influence of every timestep by wanting throughout all covariates and time sequence.

Characteristic-wise Interpretability

The Variable Choice Community element of TFT can simply estimate the function importances:

In Determine 10, we discover the next:

• The hour and day_of_week have robust scores, each as previous observations and future covariates. The benchmark within the authentic paper shares the identical conclusion.
• The power_usage is clearly essentially the most impactful noticed covariate.
• The consumer_id shouldn’t be very vital right here as a result of we use solely 5 shoppers. Within the TFT paper, the place the authors use all 370 shoppers, this variable is extra vital.

Notice: In case your grouping static variable shouldn’t be necessary, it is vitally doubtless your dataset may also be modeled equally properly by a single distribution mannequin (like ARIMA).

Excessive Occasion Detection

Time sequence are infamous for being prone to sudden adjustments of their properties throughout uncommon occasions (additionally known as shocks).

Even worse, these occasions are very elusive. Think about in case your goal variable turns into unstable for a short interval as a result of a covariate silently adjustments conduct:

Is that this some random noise or a hidden persistent sample that escapes our mannequin?

With TFT, we are able to analyze the robustness of every particular person function throughout their vary of values. Sadly, the present dataset doesn’t exhibit volatility or uncommon occasions — these usually tend to be present in monetary, gross sales information and so forth. Nonetheless, we are going to present find out how to calculate them:

Some options do haven’t all their values current within the validation dataset, so we solely present the hour and consumer_id:

In each Figures, the outcomes are encouraging. In Determine 12, we discover that client MT_004 barely underperforms in comparison with different shoppers. We may confirm this if we normalize the P50 lack of each client with their common energy utilization that we calculated beforehand.

The grey bars denote the distribution of every variable. One factor I at all times do is use which values have a low frequency. Then, I examine how the mannequin performs in these areas. Therefore, you may simply detect in case your mannequin captures the conduct of uncommon occasions.

Typically, you should use this TFT function to probe your mannequin for weaknesses and proceed to additional investigation.

We are able to seamlessly use Temporal Fusion Transformer with Optuna to carry out hyperparameter tuning:

The issue is that since TFT is a Transformer-based mannequin, you have to vital {hardware} assets!

Temporal Fusion Transformer is undoubtedly a milestone for the Time-Sequence neighborhood.

Not solely does the mannequin achieves SOTA outcomes, but in addition gives a framework for the interpretability of predictions. The mannequin can be accessible within the Darts python library, which is predicated on the PyTorch Forecasting library.

Lastly, if you’re curious to be taught in regards to the structure of the Temporal Fusion Transformer intimately, examine the companion article on the unique paper.