Data for model training – Pipelines using Kubeflow for Custom Models

Introduction

In the previous chapter, we worked on the pipelines of GCP using Kubeflow. We built the first pipeline using AutoML of the platform for model training. In this chapter, we will see how to build pipelines for custom models and compare the results of different pipelines. We will also understand a few differences between Vertex AI pipelines and Kubeflow pipelines.

Structure

In this chapter, we will cover the following topics:

• Data for model training
• Additional permissions
• Creation of workbench using cloud shell
• Pipeline code walk through
• Pipeline
• Pipeline comparison
• Deletion of resources

Objectives

By the end of this chapter, users will be able to build pipelines using Kubeflow for custom model building and compare the comparison of results of the different pipelines.

Data for model training

For this exercise, data is downloaded from Kaggle (link is provided below), and the dataset is listed under CC0:Public domain licenses. In previous chapters we have used cloud storage to upload the data to train the model, in this chapter data is being uploaded to BigQuery of GCP on which machine learning model is trained.

Refer to Biggquery section from Chapter 1, Basics of Google Cloud Platform to upload the data. Figure 1.43 to 1.47 will help you upload the data to BigQuery. Refer to Figure 1.48 to run a small query to check if the data upload is successful.

To upload data to BigQuery, create a dataset by the name employee_data. Under the dataset, create a table called employee_promotion_data. In the query section, use the query select * from ‘vertex-ai-gcp-1. employee_data. employee_promotion_data’ limit 10 to check the data upload. Figure 7.1 shows data that is uploaded to BigQuery:

Figure 7.1: Data loaded into BigQuery table

In this chapter we will create pipeline on Vertex AI component of GCP using Kubeflow SDK. In the pipeline we will use the AutoML component of GCP for model training.