The skippable tasks

Taipy is an easy-to-use Python library designed for building data-driven web applications with interactive user interfaces. In this article, we focus on Taipy’s backend capabilities: Taipy Core. Taipy Core helps build and manage complex workflows, i.e., data processing pipelines.

One of Taipy Core’s key features is the “skippable” Task: tasks that can be skipped under specific conditions. This article explores how skippable tasks work in Taipy and how to use them effectively.

Setting up Data nodes for Tasks

A Task is a Taipy representation of a Python function to use in the execution graph.

It holds : 

• input and output Data nodes and
• the user-defined Python function it represents.

Before using skippable tasks, it’s crucial to configure tasks correctly with its Data nodes. Refer to our previous article or documentation for more information on Data nodes.

For example, consider a function (e.g. multiply_and_add()) that takes two parameters and returns two values. How can we model this function as a Taipy Task?

def multiply_and_add(nb1, nb2):
    return nb1 + nb2, nb1 * nb2

In the animation below:

• The first tab corresponds to the creation of the configuration graphically using Taipy Studio.
• The second tab corresponds to the creation of the very same configuration programmatically.

The sequence in which Data nodes are provided to the Task is crucial. Taipy invokes the function using the parameters based on the order of the Data nodes, and it is in this specific order results will be returned.

In this example, the multiply_and_add() function takes two parameters (nb1 and nb2). It returns two values: the product and the sum. We create configurations for the input Data nodes (nb_1_cfg and nb_2_cfg in this order) and the output Data nodes (sum_cfg and product_cfg).

Finally, we configure the Task with the appropriate input and output Data nodes.

Leveraging Skippability in Taipy Tasks

Skippability is an optional parameter that can be set when configuring a Task. When skippable is set to True, Taipy skips the Task execution if its input Data nodes have stayed the same since the last execution (i.e., re-executing the task would be useless since it would lead to the same output!)

This feature can greatly enhance pipeline performance by avoiding unnecessary computations and saving time and resources.

Use case

Let’s take the previous execution graph and set skippable=True to our Task.

With the code below, we create and submit an instance of this scenario configuration.

scenario = tp.create_scenario(scenario_cfg)
tp.submit(scenario)
print("Results (sum):", scenario.sum.read())

The job corresponding to my Task is completed. It means that my function has been executed.
The line below resubmits scenario. Note that I have not modified my input Data nodes in any way.

tp.submit(scenario)
print("Results (sum):",scenario.sum.read())

As expected, Taipy skips the Task because the input parameters have not changed. If this scenario contains multiple tasks, Taipy may skip multiple tasks.

The code below demonstrates what happens when we submit the scenario after modifying an input Data node. The value of nb_1 changes from 21 to 42.

scenario.nb_1.write(42)
tp.submit(scenario)
print("Results (sum):", scenario.sum.read())

The input changed, so Taipy will re-execute my Task and give the appropriate results (44).

Using Global Data nodes

Skipping Tasks don’t just occur when we resubmit a given scenario or a pipeline. It can also happen when creating and submitting a brand new scenario with Global Data nodes.

For example, if you want to preprocess a raw data set and make the result global to the entire application, you can change both Data nodes’ scope (the raw data set and the result) to Global so all scenarios “share” them. The Task related to this operation might be skipped across scenarios.

Let’s have a look at our previous code and change Data nodes to a Global Scope.

The first line creates a scenario consisting of Data nodes, tasks, and pipelines. Following this, we submit it.

scenario_1 = tp.create_scenario(scenario_cfg)
tp.submit(scenario_1)
print("Results (sum):", scenario_1.sum.read())

The only job is completed, and the results are computed.

Now, let’s create another scenario. This new scenario does not create new Global Data nodes (input and output). It will reuse the ones created by scenario_1.

scenario_2 = tp.create_scenario(scenario_cfg)
tp.submit(scenario_2)
print("Results (sum):", scenario_2.sum.read())

Taipy reuses the existing Data nodes and skips the Task if the input Data nodes have not changed.

Manual changes

Taipy keeps track of the last modification date of a Data Node to know if an input Data node has changed.

Modifying the date of a Data node can happen under three different conditions:

• When you submit a Task or a scenario, you change the last modification date.
• You can also do this by manually writing it: <Data node>.write(…).
• Taipy also detects a file’s last modification date your Data Node refers to. If you (or a separate process) change a file (CSV, JSON, etc.), Taipy knows about it and adjusts the skippable logic accordingly.

Conclusion

In conclusion, Taipy greatly simplifies the complexity of your workflows and their execution. By employing skippable tasks, developers can significantly improve the performance of their pipelines, avoiding unnecessary computations and saving computation time and resources for the end users. Skippable tasks can be leveraged in various use cases (including resubmitting scenarios, when manual changes are made to Data nodes, etc.).

By understanding and effectively using skippable tasks in Taipy, developers can create efficient and streamlined applications that better serve their users.