What is TensorFlow Extended – Pipelines using TensorFlow Extended
In the previous chapter we worked on the pipelines of GCP using Kubeflow and built the first pipeline for the custom model training. In this chapter, we will start understanding the TFX and its components, and how to use these components for custom model training.
In this chapter, we will discuss the following topics:
- What is TensorFlow Extended (TFX)
- TFX Pipelines
- Components of TFX
- Data for pipeline building
- Pipeline code walk through
- Deletion of resources
By the end of this chapter, users will be able to build TFX pipelines using standard components and execute the pipeline using Kubeflow.
It is undoubtedly challenging to put machine learning and deep learning models into production. Even though it produces additional value for a business, it is more time-consuming and prone to failure than the modeling itself. A model has to be maintained after it has been deployed. We must evaluate its functionality, the caliber of freshly produced data, and the suitability of the supporting infrastructure. The model should be retrained as necessary. To be long-term viable, the entire process must be automated with as little human involvement as feasible. Additionally, we must ensure that the model we are using is dependable, consistent, secure, and maybe scalable. Google officially launched its in-house platform TensorFlow Extended in early 2019 to assist businesses in implementing an industrial-grade end-to-end production system.
TFX is intended to be an end-to-end ML platform that is strong and versatile. It is built on the TensorFlow (TF) packages, which let Python programmers create custom functions. TFX encapsulates the whole TensorFlow ecosystem (from Keras to Data Validation to Serving) to enable users to construct and operate scalable and effective Machine Learning pipelines, as opposed to just allowing them to serve a model in a highly performant manner. It is structured as a series of adaptable components that need users to write less code.
A TFX pipeline is an implementation of a ML process that is portable and can be performed on a variety of orchestrators, such as Kubeflow Pipelines, Apache beam, and Apache Airflow. The building blocks of a pipeline are called component and input parameters.
The outputs of component instances are artifacts, while the inputs of component instances often consist of artifacts that were made by upstream component instances. Directed Acyclic Graphs (DAG) are created using artifacts which determines the execution order of the pipeline.
Leave a Reply