Metadata Exporter Application

The Metadata Exporter application enables organizations to export all metadata from Collate into downstream analytical or data storage systems such as Snowflake, BigQuery, Databricks, and Trino. This includes:

Entity metadata — Asset definitions, descriptions, tags, and ownership
Test case results — Data Quality (DQ) pass/fail flags, rule IDs, asset metadata, and timestamps
Profiler stats data — Column-level statistics and profiling metrics

Overview

This capability supports:

Feeding downstream dashboards (Power BI, Tableau).
Triggering alerting and remediation workflows.
Historical tracking and versioning of data quality scores (DQI).

Key Features

Supports Snowflake, Databricks, BigQuery, and Trino destinations.
Manual or scheduled export runs.
Selective export of entity metadata, data quality events, or profiler stats.

How to Access

Go to Settings > Applications > Metadata Exporter.

Review available tabs: Schedule, Configuration, Recent Runs.

Configuration Options

Ingestion Runner

Defines the agent responsible for executing the ingestion pipeline.

Example: Collate SaaS Agent

Connection Configuration

Establishes connectivity to your export destination (Snowflake, BigQuery, Databricks).

Snowflake Configuration

Field	Description
Service Type	Snowflake
Username	Snowflake user login
Password	User password (optional if using private key)
Account	Snowflake account identifier (e.g., `AAAAA-99999`)
Role	Snowflake role to assume (use a dedicated least-privilege role, see Required Snowflake Privileges below)
Database	Target database (e.g., `OBS_ANALYTICS`)
Warehouse	Target virtual warehouse (e.g., `COMPUTE_WH`)
Query Tag	Optional tagging for traceability
Private Key & Passphrase	For key-pair auth (optional, secure)

Advanced Option

Client Session Keep Alive: Useful for long-running exports.

Required Snowflake Privileges The role used by the exporter needs the following privileges:

USAGE on the warehouse
USAGE on the database
USAGE on the schema
CREATE TABLE on the schema

The exporter creates and owns its target table, so INSERT, MERGE, ALTER and DROP all work through ownership with no further grants.

-- Create a new role and user (skip if you already have them)
CREATE ROLE COLLATE_EXPORTER_ROLE;
CREATE USER COLLATE_EXPORTER_USER DEFAULT_ROLE=COLLATE_EXPORTER_ROLE PASSWORD='PASSWORD';
GRANT ROLE COLLATE_EXPORTER_ROLE TO USER COLLATE_EXPORTER_USER;

-- Grant the required privileges to the role
GRANT USAGE ON WAREHOUSE WAREHOUSE_NAME TO ROLE COLLATE_EXPORTER_ROLE;
GRANT USAGE ON DATABASE DATABASE_NAME TO ROLE COLLATE_EXPORTER_ROLE;
GRANT USAGE ON SCHEMA DATABASE_NAME.SCHEMA_NAME TO ROLE COLLATE_EXPORTER_ROLE;
GRANT CREATE TABLE ON SCHEMA DATABASE_NAME.SCHEMA_NAME TO ROLE COLLATE_EXPORTER_ROLE;

On connect, the exporter runs the standard Snowflake connection test, which also probes SNOWFLAKE.ACCOUNT_USAGE for tags, queries and lineage. With this role those checks are skipped with a warning and the export still succeeds. IMPORTED PRIVILEGES is not required.

BigQuery Configuration

Field	Description
Service Type	Must be `BigQuery`
Project ID	GCP project where the BigQuery dataset resides
Dataset ID	Target dataset where the metadata will be exported
Table Name	Destination table name (BQ table to export metadata to)
Service Account JSON	Contents of the service account key in JSON format with write access
Location	BigQuery region (e.g., `us-central1`)

Ensure the service account has the BigQuery Data Editor and BigQuery Job User roles.

Databricks Configuration

Field	Description
Service Type	Must be `Databricks`
Host URL	Databricks workspace URL (e.g., `https://<region>.azuredatabricks.net`)
Token	Personal Access Token (PAT) for API authentication
Cluster ID	Target cluster where jobs will run
Database Name	Target database within the Databricks environment
Schema Name	Schema (if applicable)
Table Name	Destination table to store metadata
Path (Optional)	DBFS path or external location (if exporting to files instead of a table)

Requirements

The token must have workspace-wide read/write access.
The cluster must have access to the target database or mount location.

Trino Configuration

Available from version 1.12.3

Field	Description
Service Type	Must be `Trino`
Username	Trino user login with sufficient privileges
Password	User password for Basic Authentication (or use JWT)
Host and Port	Fully qualified hostname and port number for your Trino deployment
Catalog	Trino catalog where the target database is stored
Database Name	Target database within the Trino catalog
Schema Name	Target schema (optional, restricts to a single schema)
Table Name	Destination table to store exported metadata

Authentication Options

Basic Auth: Username and password
JWT Auth: Provide JWT token for authentication
Azure: Client ID, Client Secret, Tenant ID, and Scopes for Azure AD authentication

Ensure the Trino user has SELECT and INSERT privileges on the target table and schema.

Export Range

Defines the temporal scope of the data to be exported.

Field	Description
Range Type (`exportRange.rangeType`)	Options: `ALL`, `LATEST`, or `DATE_RANGE`
Interval (`exportRange.interval`)	Used with `DATE_RANGE` (e.g., `7`)
Unit (`exportRange.unit`)	Time unit for the interval (e.g., `days`, `hours`)
Event Types	Select which types of DQ events to export (`All`, or specific types)
Backfill	Enable to process historical data on first run

Table Configuration

Specifies the target table where exported metadata will be written.

Field	Description
Database Name (`tableConfiguration.databaseName`)	e.g., `OBS_ANALYTICS`
Schema Name (`tableConfiguration.schemaName`)	e.g., `OBS_DATA`
Table Name (`tableConfiguration.tableName`)	e.g., `COLLATE_METADATA`

Scheduling

Configure how often the metadata export runs:

Manual: Click Run Now on the Schedule tab.
Scheduled: Set up periodic exports (feature roadmap).

Monitoring Runs

Under the Recent Runs tab:

View status: Success or Failed.
Check run time, duration, logs, and the configuration used.

A successful export shows Status: Success, with details on execution duration and timestamps. Monitoring Runs