What is machine learning pipeline – Introduction to Pipelines and Kubeflow

Introduction

In the previous chapters, we worked on workbench of Vertex AI to train custom models including hyperparameter tuning using Vizer. In this chapter, we will get started with the pipelines of Vertex AI. We will understand what pipeline is, what is Kubeflow, what are the components of the pipeline, and how to configure and run Vertex AI pipeline using Kubeflow.

Structure

In this chapter, we will discuss the following topics:

• What is machine learning pipeline
• Benefits of machine learning pipelines
• Introduction to Kubeflow
• Components of Kubeflow
• Tasks of Kubeflow
• Data for model training
• API enablement and additional permissions
• Pipeline code walk through
• Pipeline of vertex AI
• Deletion of resources

Objectives

By the end of this chapter, you will have a good understanding of Machine Learning (ML) pipelines, along with Kubeflow and its components. The readers will also learn how to construct pipelines using Kubeflow on Vertex AI.

What is machine learning pipeline

A machine learning pipeline is a comprehensive framework for managing input and output from a ML model (or set of multiple models). There are inputs of raw data, features, and outputs, as well as the machine learning model and its parameters, and predictions.

Another way of understanding ML pipelines is the process of breaking down large machine learning operations into smaller, reusable modules that may be used in a pipeline to produce models is another sort of ML pipeline. By eliminating unnecessary steps, an ML pipeline speeds up and simplifies the model-building process.

Vertex AI pipelines

Automating, monitoring, and governing your ML systems is made easier with Vertex AI pipelines, which orchestrates the ML workflow in a serverless fashion and stores the artifacts of that process in Vertex ML Metadata. By keeping track of your ML workflow’s outputs in Vertex ML Metadata, users can trace back the steps that went into making each artifact, such as an ML model’s training data, hyperparameters, and code.

Pipelines are used for applying methods from MLOps to streamline and keep tabs on your most-often-performed tasks. Try out various configurations of your machine learning process by playing with the hyperparameters, training steps, iterations, and so on. Re-use pipeline component or entire pipeline to train a new model. Pipelines that were built using Kubeflow SDK or TensorFlow extended can be executed using Vertex AI pipelines.