Components of TFX – Pipelines using TensorFlow Extended

Pipelines break down the machine learning workflow into a series of components, with each component being responsible for a certain stage in the ML process. Standard components and custom components are both offered by TFX as separate groups. Users can construct pipelines with only a few standard components as well.

ML process may be expanded with the help of custom components. Constructing components are customized to match users’ requirements, such as absorbing data from a closed-source system, applying data augmentation, sampling, integrating tools developed in languages other than Python into your machine learning process, such as data analysis with R, etc.

Components in TFX are made up of a component specification and an executor class, both of which are contained within a component interface class. The input and output contracts for a component are defined by the component’s specification. This contract outlines the input and output artifacts of the component along with the parameters that are applied during the process of executing the component. The executor class of a component is responsible for implementing the tasks that the component is responsible for carrying out. Specification and the executor are combined into an interface class to use the component in the TFX pipelines.

The execution of the component takes place in three steps:

1. Driver: makes decisions about what needs to be done based on the metadata and coordinates the job execution
2. Executor: does the actual work to complete the job code contributed by users to complete the current project at hand
3. Publisher: is responsible for gathering the results of the executor and bringing the metadata database up to date.

If the standard functionality is all that is required, then we do not need to make any changes to the code that makes up the driver or the publisher. Simply extending the executor is all that is required of us to make modifications to the executor while maintaining the same inputs, outputs, and execution attributes. It is possible to write a completely bespoke executor to achieve an entirely distinct set of functionalities.

Types of custom components

There are three kinds of custom components: based on Python functions, based on containers, and the ones that are fully custom:

• Python functions: Building Python components that are based on functions is easier than building components that are based on containers or that are made from scratch. Type annotations show whether an argument is an input artifact, an output artifact, or a parameter. These annotations are in the Python function’s arguments.
• Container based: Container-based components let you add code written in any language to your pipeline, as long as that code can be run in a Docker container.
• Fully custom: You can build fully custom components by defining the classes for the component specification, the component executor, and the component interface.

There are libraries and pipeline components in TFX. The following figure shows how TFX libraries and pipeline components relate to each other:

Figure 8.1: Libraries and components of TFX 1

1 Image source: https://www.tensorflow.org/tfx/guide