Vertexai Hybrid Runner | Official Docs

Connect VertexAI to Collate using the Hybrid Runner. Deploy the ingestion agent in your environment for secure, private network metadata extraction.

In this section, we provide guides and references to use the VertexAI connector. Configure and schedule VertexAI metadata workflows from the Collate UI:

• Requirements
• Metadata Ingestion

Requirements

VertexAI API Permissions

• Go to Cloud VertexAI Library enable API
• Select the GCP Project ID.
• Click on Enable API which will enable the data catalog api on the respective project.

GCP Permissions

To execute metadata extraction workflow successfully the user or the service account should have enough access to fetch required data. Following table describes the minimum required permissions

#	GCP Permission	Required For
1	aiplatform.models.get	Metadata Ingestion
2	aiplatform.models.list	Metadata Ingestion

Metadata Ingestion

1. Test the Connection

   Once the credentials have been added, click on Test Connection and Save the changes.

   
2. 7. Configure Metadata Ingestion

   In this step we will configure the metadata ingestion pipeline, Please follow the instructions below

   
3. Schedule the Ingestion and Deploy

   Scheduling can be set up at an hourly, daily, weekly, or manual cadence. The timezone is in UTC. Select a Start Date to schedule for ingestion. It is optional to add an End Date.

   Review your configuration settings. If they match what you intended, click Deploy to create the service and schedule metadata ingestion.

   If something doesn’t look right, click the Back button to return to the appropriate step and change the settings as needed.

   After configuring the workflow, you can click on Deploy to create the pipeline.

   
4. View the Ingestion Pipeline

   Once the workflow has been successfully deployed, you can view the Ingestion Pipeline running from the Service Page.

   

   If AutoPilot is enabled, workflows like usage tracking, data lineage, and similar tasks will be handled automatically. Users don’t need to set up or manage them - AutoPilot takes care of everything in the system.

When using a Hybrid Ingestion Runner, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:

password: secret:/my/database/password

This applies only to fields marked as secrets in the connection form (these typically mask input and show a visibility toggle icon).

For a complete guide on managing secrets in hybrid setups, see the Hybrid Ingestion Runner Secret Management Guide.

Troubleshooting

VertexAI Troubleshooting

Learn more about how to troubleshoot common VertexAI connector issues and resolve configuration or ingestion errors.