OpenLineage Compatibility

Send OpenLineage events to Collate and understand dataset resolution, supported facets, and ingestion limits.

​

OpenLineage Compatibility

Collate accepts OpenLineage RunEvent payloads and converts their input and output datasets into lineage edges. You can send events directly over HTTP or consume events through the OpenLineage pipeline connector.

​

Choose an Ingestion Method

Use the method that matches where your producer sends events.

Method	Configuration	Use Case
HTTP API	Point the producer’s HTTP transport to /api/v1/openlineage/lineage.	Send events directly from a client or integration.
Kafka or AWS Kinesis	Configure the OpenLineage connector to read the broker or stream.	Consume events already published to Kafka or Kinesis.
Native Lineage API	Send entity references to /api/v1/lineage.	Create explicit relationships between assets already in Collate.

The HTTP API runs in the Collate server. It does not require a Kafka broker, Kinesis stream, or scheduled connector workflow. The connector has its own connection and ingestion settings. The HTTP settings below apply to direct event ingestion.

​

Prepare Collate

Before you send events, complete these steps:

1. Create or ingest the database services referenced by the events. Ingest their tables and columns too, unless you rely on table creation for mapped namespaces. Column-level lineage needs the columns.
2. Obtain an Collate JSON Web Token (JWT) for a user or bot with EditLineage permission on the table resource.
3. Configure namespace mappings so each dataset resolves to the intended database service.
4. To associate events with pipelines, create or ingest the pipelines under the pipeline service named by defaultPipelineService. Its default name is openlineage. The server never creates pipelines.

For more information, see API Authentication.

​

Match Datasets to Catalog Assets

An OpenLineage dataset has a namespace and a name. The namespace identifies its source system. The name identifies the dataset within that system.

For example, this dataset can resolve to the Collate table prod-postgres.analytics.public.orders:

{
  "namespace": "postgresql://warehouse.example.com:5432",
  "name": "analytics.public.orders"
}

Map postgresql://warehouse.example.com:5432 to the existing database service prod-postgres through namespaceToServiceMapping. Use the exact namespace emitted by your producer. Include the database and schema in the dataset name when available.

The resolver tries the namespace mapping, datasource facet information, and table-name suffix matches. A namespace is not automatically a Collate service name. Namespace mappings match the exact namespace first, then the longest matching prefix.

A dataset matches only the database services that hold its table. If the table exists in one of several services of the same type, the dataset resolves to that service. If the table exists in more than one of them, the match is ambiguous. The server uses the first match and logs a warning. Add a namespaceToServiceMapping entry to pin the service.

For more information, see Cross Database Lineage.

​

Configure HTTP Ingestion Settings

An administrator can read the current configuration with GET /api/v1/system/settings/openLineageSettings. To update it, send a settings object to PUT /api/v1/system/settings.

Preserve existing mappings and other settings when you update the configuration. The following example processes completed runs against assets already in the catalog:

{
  "configType": "openLineageSettings",
  "configValue": {
    "enabled": true,
    "autoCreateEntities": false,
    "eventTypeFilter": ["COMPLETE"],
    "defaultPipelineService": "openlineage",
    "namespaceToServiceMapping": {
      "postgresql://warehouse.example.com:5432": "prod-postgres"
    }
  }
}

The API is enabled by default. If eventTypeFilter is absent or empty, the server processes COMPLETE events. It also treats an omitted eventType as eligible when COMPLETE is allowed. Send an explicit event type to make the producer’s intent clear.

autoCreateEntities defaults to true. When it is enabled, the server creates a missing table only when the dataset’s namespace maps to a database service through namespaceToServiceMapping. The server creates the table with the columns in the event’s schema facet. It creates the database and schema with the table if they are missing. It never creates a table without columns, and it never creates a database service. If the namespace has no mapping, the dataset is reported as unresolved.

The server never creates pipelines. Ingest the pipeline first. If the pipeline is missing, the server writes the lineage edge without a pipeline and reports the job as unresolved.

​

Configure an HTTP Producer

For the OpenLineage Python client, save this configuration as openlineage.yml:

transport:
  type: http
  url: https://your-company.getcollate.io
  endpoint: /api/v1/openlineage/lineage
  auth:
    type: api_key
    api_key: "<COLLATE_JWT>"

Replace the host and token with your deployment values. Set OPENLINEAGE_CONFIG to the file path if the file is outside the client’s working directory. The api_key authentication provider sends the Collate JWT in the Authorization: Bearer header.

Set the endpoint explicitly. The client’s default /api/v1/lineage path is the native Collate Lineage API, which expects a different request body.

For more information, see the OpenLineage Python Client Configuration. Other clients and integrations have their own configuration syntax. Configure the same URL, endpoint, and bearer token using that client’s HTTP transport options.

​

Understand the Supported Metadata

The HTTP mapper uses the following event fields and facets:

Event Metadata	Collate Behavior
inputs and outputs	Creates a lineage edge for each resolved input and output pair. Both lists must contain a dataset.
Dataset schema facet	Supplies column names and types when the server creates a table for a mapped namespace. It does not replace metadata ingestion for existing tables.
Job sql facet	Stores job.facets.sql.query on the lineage edge. SQL text alone does not supply input and output datasets.
Output columnLineage facet	Maps output fields to input fields. The mapper reads outputFacets.columnLineage first, then facets.columnLineage.
Run parent facet	Uses the parent job to resolve the pipeline reference. Without a usable parent job, it uses the event’s job.

For column-level lineage, each inputFields entry must match an input dataset’s namespace and name. Its field must identify a catalog column. Ingest the input tables and columns before sending column mappings.

The mapper can resolve storage datasets to existing container assets by path. It does not create missing containers. Compatibility depends on the dataset names and facets emitted by your integration. An accepted event does not mean that every custom facet becomes Collate metadata.

​

Send an Event and Verify Lineage

Use the complete request examples in the OpenLineage Events API to send a single event or a batch.

After sending an event, check lineageEdgesCreated. A 200 response with status: success can still contain zero edges for a filtered event or an event without inputs or outputs. Failed edge writes are logged and can also leave the count at zero. A single event with unresolved datasets and no edges returns 400.

If no edges appear, check the event type and dataset names first. Then check namespace mappings, existing catalog assets, and the Collate server logs. Use the Get Lineage API or the asset’s Lineage tab to confirm the resulting graph.