Ingest Metadata
Collate supports metadata ingestion from third-party sources through the UI. Admin users can connect to multiple service types — Databases, Dashboards, Pipelines, ML Models, Messaging, Storage, Search, and Metadata services — and schedule metadata ingestion at any frequency.Note: You must have Admin access in Collate to add a connector and ingest metadata.
Step 1: Add a New Service
- In the left navigation, click Connections.
-
On the Connections page, click Add New Service.
Step 2: Select a Service Type and Connector
In the dropdown, select a service type that matches your data source, and then choose a connector. Collate supports the following service types:- Database
- Dashboard
- Pipeline
- Messaging
- ML Model
- Storage
- Search
- Metadata
- API
- Drive
- Security
Step 3: Add a Service Name and Description
- Enter a unique, descriptive Service Name. Collate identifies services by their service name — choose a name that distinguishes this service from others of the same type.
- (Optional) Enter a Description to document the service and help other users understand what it contains.
Note: The service name can’t be changed after you set it.
Step 4: Configure Connection Options
Select Ingestion Runner
Select an Ingestion Runner — the environment where the ingestion pipeline will execute.
Enter Connection Details
Enter the connection credentials for your source. The required fields differ by connector — refer to the connector-specific documentation in the Connectors section for the full list of fields and accepted values. The right-hand panel in the Collate UI displays inline help for each field as you fill in the form.Test Connection
Once credentials are entered, click Test Connection to verify access before saving. The test checks connectivity and confirms which data assets can be ingested using the provided credentials.
Step 5: Configure What to Ingest
Use filter patterns to control which assets Collate ingests from the service. How filter patterns work:- Include — Add one or more comma-separated regular expressions. Collate ingests only assets whose names match at least one expression. Leave blank to include all assets.
- Exclude — Add one or more comma-separated regular expressions. Collate skips any asset whose name matches at least one expression. Leave blank to exclude nothing.
- contains — matches any name containing the value.
- starts with — matches names beginning with the value.
- ends with — matches names ending with the value.
- is exactly — matches the exact name only.
- matches regex — matches names using a full regular expression.
Step 6: Create & Deploy
Click Create & Deploy to deploy the agent and start the first metadata ingestion run. Collate saves the service configuration and immediately begins pulling metadata from the source. To monitor progress or view the service you just added, go to Connections in the left navigation and select your service.Configure and Schedule the Metadata Agent
After the service is created, you can configure the metadata agent and set an ingestion schedule.-
Go to the Agents tab on the service page and click Add Agent > Metadata.
-
Enter a name for the agent. The name is pre-populated with the service name and a suffix to ensure uniqueness.
-
Define when the agent runs. Select a preset schedule or enter a Custom Cron expression.
- Click Add & Deploy to save the schedule and start the agent.
View and Manage Agent
After deployment, go to the Agents tab on the service page.- The Metadata Agents section lists each deployed agent with its status badge (Success, Failed, Running, and so on), asset count, and the time it last ran.
- Click View run history under an agent to see a full log of past runs.
- To manage an agent, use the action buttons on each agent row:
- Logs — View the pipeline logs for the most recent run.
- Run — Trigger the pipeline immediately.
- ⋮ menu — Contains the following options:
- Pause — Pause the scheduled pipeline without deleting it. For a paused agent, the menu shows Resume instead.
- Kill run — Stop the run that is in progress. The menu shows this option only while the agent is running.
- Re-deploy — Re-apply updated credentials or configuration and re-ingest data. Use this when source credentials change after the initial setup. The menu shows this option when the agent is not running.
- Edit configuration — Modify the agent’s filter patterns, schedule, or settings.
- Delete agent — Permanently remove the agent.
Browsing Ingested Data
After ingestion completes, open the service page and go to the relevant asset tab — for example, Databases for a database service. From there, drill down to view schemas, tables, and columns. The service page also includes the following:- Data Model: For dashboard services, lists the dashboard data models.
- Files and Spreadsheets: For drive services, lists the files and spreadsheets next to the directories.
- Metadata Details: Opens expanded and shows the service’s data products. If you have the EditAll permission, you can edit them on a service that isn’t deleted.
- Version history: The version button in the page header opens the service’s version history.
- Follow: For database services that aren’t deleted, the page header has a button to follow the service.
- Announcements: Active announcements appear in a banner at the top of every tab.
Managing Services
Admin users can create, edit, or delete services, and view connection details for existing services from the Connections page.Connectors
Browse all supported connectors and connector-specific setup guides.
Delete a Service Connection
Permanently delete a service connection and its associated agents.