A model trained by hand in a notebook is an experiment. A reproducible pipeline with full lineage is what separates an experiment from a production system.
Vertex AI Pipelines is Google Cloud's serverless ML workflow orchestration service, based on the Kubeflow Pipelines (KFP) SDK or TensorFlow Extended (TFX). It lets you define a complete flow -- preprocessing, training, evaluation, model registration, conditional deployment -- as a reproducible, versioned graph of components, executed without the team having to manage its own Kubernetes cluster.
The difference versus running loose scripts or manual notebooks is traceability: every run is logged in Vertex ML Metadata, with full lineage of which dataset, which code version, and which hyperparameters produced each specific model.
A pipeline is defined as a Python function decorated with `@dsl.pipeline`, where each step is a component -- a containerized compute unit with typed inputs and outputs.
This pipeline compiles to a portable JSON/YAML file and runs on Vertex AI with no persistent infrastructure: you only pay for compute during execution.
Google maintains a `google-cloud-pipeline-components` library with ready-made operators for common tasks: launching an AutoML job, deploying a model to an endpoint, running batch prediction, or registering a model in Model Registry. For business-specific logic (custom data quality validations, particular transformations), you write custom components as Python functions packaged in a container, reusable across pipelines.
Every pipeline run automatically logs full lineage in Vertex ML Metadata: which dataset artifact went in, what parameters were used, what model came out, and what metrics it achieved. This answers the question that almost always comes up in an audit or a production incident: "what data and what code trained the model currently serving right now?", without depending on someone having manually documented the process in a separate spreadsheet.
A common pattern is conditioning production deployment on the model exceeding a metric threshold against the currently deployed model, preventing an automatic retraining from degrading the production system without human oversight.
Vertex AI Pipelines integrates with Cloud Scheduler for periodic retraining (weekly, monthly) and with Cloud Build or GitHub Actions to trigger pipeline compilation and execution as part of a standard CI/CD flow when training code is updated in the repository. This turns retraining from a manual, ad hoc event into a versioned process just like any software deployment.
Vertex AI Pipelines doesn't charge for orchestration itself beyond a minimal per-run charge; the real cost is each component's compute (training, evaluation). The operational recommendation is keeping pipelines granular -- separating preprocessing from training from deployment -- so you can cache steps that didn't change between runs (KFP supports component output caching by default), reducing time and cost on frequent iterations where, for example, only the final hyperparameter changes.
Carlos Montiel is an enterprise AI solutions architect. He implements LLMs, Agents, RAG, and orchestrators for companies across Guatemala and Latin America. Reach out for a consultation.
Contact Carlos Montiel