Kubernetes On Premises Deployment | Collate

Deploy Collate Binaries on an on-premises Kubernetes cluster using Argo Workflows for ingestion orchestration.

Guide to Deploy Collate Binaries On-Premises

This guide will help you start using Collate Docker Images to run the OpenMetadata Application in an on-premises Kubernetes cluster, connecting with Argo Workflows for running ingestion from the OpenMetadata Application itself.

This guide presumes you have an on-premises Kubernetes cluster already set up. All code snippets assume the collate namespace unless otherwise noted.

Architecture

Collate OpenMetadata requires 4 components:

1. Collate Server
2. Database — Collate Server stores the metadata in a relational database. We support MySQL or Postgres.
   • MySQL version 8.0.42 or greater
   • Postgres version 17.6 or greater
3. Search Engine — OpenSearch 3.4. ElasticSearch is not supported in Collate BYOC because Collate AI relies on OpenSearch’s vector capabilities for Semantic and Hybrid Search.
4. Workflow Orchestration — We use Argo Workflows as the orchestrator for ingestion pipelines.

Sizing Requirements

Hardware Requirements

A Kubernetes Cluster with at least 1 Master Node and 3 Worker Nodes is the required configuration. Each Worker Node should have at least:

• 4 vCPUs
• 16 GiB Memory
• 128 GiB Storage capacity

If you want Collate workloads scheduled on dedicated nodes, use Kubernetes taints and tolerations. Collate OpenMetadata supports tolerations via custom Helm values.

Software Requirements

• Collate OpenMetadata supports Kubernetes Cluster version 1.24 or greater.
• Collate Docker Images are available via private AWS Elastic Container Registry (ECR). The Collate Team will share credentials and steps to configure Kubernetes to pull Docker Images from AWS ECR.
• For Argo Workflows, Collate OpenMetadata is currently compatible with application version 3.4+.

Database Sizing and Capacity

Our recommendation is to configure PostgreSQL. For 100,000 Data Assets and 1,000 Users:

• 8 vCPUs
• 64 GiB Memory
• 256 GiB Storage Capacity
• 3,500 IOPS storage

Search Client Sizing and Capacity

For 100,000 Data Assets and 1,000 Users:

• 8 vCPUs
• 64 GiB Memory
• 256 GiB Storage Capacity

Argo Workflows Ingestion Runners

The recommended resources are 4 vCPUs and 16 GiB of Memory.

On-Premises Prerequisites

Object Storage for Argo Workflows Artifacts

Argo Workflows requires object storage to archive ingestion logs. On-premises deployments can use MinIO as an S3-compatible object store.

Deploy MinIO (if you don’t have an existing object store)

helm repo add minio https://charts.min.io/
helm repo update

helm upgrade --install minio minio/minio \
  --namespace minio \
  --create-namespace \
  --set rootUser=admin \
  --set rootPassword=<MINIO_ROOT_PASSWORD> \
  --set persistence.size=50Gi \
  --set service.type=ClusterIP

Create the Argo Workflows artifacts bucket:

kubectl run minio-client --image=minio/mc --rm -it --restart=Never -- \
  /bin/sh -c "mc alias set local http://minio.minio:9000 admin <MINIO_ROOT_PASSWORD> && mc mb local/argo-artifacts"

Create Kubernetes Secret for MinIO Credentials

kubectl create secret generic argo-artifact-credentials \
  --from-literal=accessKey=<MINIO_ACCESS_KEY> \
  --from-literal=secretKey=<MINIO_SECRET_KEY> \
  --namespace argo-workflows

Setup AWS ECR

Collate will provide the credentials to pull Docker Images from a private registry located in AWS ECR.

Install AWS CLI

Follow the AWS CLI installation guide to install AWS CLI on your machine.

Configure AWS Credentials

aws configure --profile ecr-collate

The command will prompt for credentials. The Collate team will securely share these via a 1Password link.

Confirm the credentials are correctly set:

aws configure list --profile ecr-collate

Kubernetes Docker Registry Secrets for AWS ECR

kubectl create secret docker-registry ecr-registry-creds \
  --docker-server=118146679784.dkr.ecr.eu-west-1.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password --profile ecr-collate) \
  --namespace <<NAMESPACE_NAME>>

Replace <<NAMESPACE_NAME>> with the namespace where you want to deploy Collate OpenMetadata Server. If the namespace does not exist yet, create it with kubectl create namespace <<NAMESPACE_NAME>>.

AWS ECR Token Refresh

ECR tokens expire after 12 hours. If a pod is rescheduled after 12 hours, you will get an ImagePullBackOff error. Delete the secret and recreate it using the command above.

Install Argo Workflows

Add Helm Repository

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

Create the Argo Namespace

kubectl create namespace argo-workflows

Kubernetes Secret for Argo Workflows DB Credentials

kubectl create secret generic argo-db-credentials \
  --from-literal=username=<DB_USERNAME> \
  --from-literal=password=<DB_PASSWORD> \
  --namespace argo-workflows

Create Custom Helm Values for Argo Workflows

Create a file named argo-workflows.values.yml:

# argo-workflows.values.yml
controller:
  serviceAccount:
    create: true
    name: argo-workflows-controller-sa
  name: workflow-controller
  workflowDefaults:
    spec:
      serviceAccountName: om-role

server:
  serviceAccount:
    create: true
    name: argo-workflows-server-sa
  extraArgs:
    - "--auth-mode=server"
    - "--request-timeout=5m"

persistence:
  archive: true
  postgresql:
    host: <DATABASE_INSTANCE_ENDPOINT>
    database: <DATABASE_NAME>
    tableName: argo_workflows
    userNameSecret:
      name: argo-db-credentials
      key: username
    passwordSecret:
      name: argo-db-credentials
      key: password
    ssl: true
    sslMode: require

useDefaultArtifactRepo: true
useStaticCredentials: true
artifactRepository:
  archiveLogs: true
  s3:
    # MinIO endpoint — replace with your object store endpoint if different
    endpoint: minio.minio:9000
    insecure: true
    bucket: argo-artifacts
    keyFormat: 'workflows/{{workflow.namespace}}/{{workflow.name}}/{{pod.name}}'
    accessKeySecret:
      name: argo-artifact-credentials
      key: accessKey
    secretKeySecret:
      name: argo-artifact-credentials
      key: secretKey

If you are using an existing S3-compatible store (e.g., Ceph, NetApp StorageGRID) instead of MinIO, update endpoint, bucket, and the secret reference to match your environment. Set insecure: false and configure TLS if your store uses HTTPS.

For further customisation, refer to the community Helm chart values.

Deploy Argo Workflows

We target application version 3.7.1 using Helm chart version 0.45.23 (Artifact Hub):

helm upgrade --install argo-workflows argo/argo-workflows \
  --version 0.45.23 \
  --namespace argo-workflows \
  --values argo-workflows.values.yml

[Optional] Enable Prometheus Metrics

If you have a Prometheus Application running on your cluster, enable metrics using:

controller:
  serviceMonitor:
    enabled: true
server:
  serviceMonitor:
    enabled: true

Refer to the official Argo Workflows documentation for further configuration.

Install OpenMetadata/Collate

Create the Collate Namespace

kubectl create namespace collate

Kubernetes Service Account for Ingestion

The OpenMetadata Application communicates with Argo Workflows to dynamically trigger ephemeral pods that run ingestion workloads. Create a dedicated Kubernetes Service Account:

kubectl create serviceaccount om-role -n collate

Create Long-Lived API Token for the ServiceAccount

kubectl apply -n collate -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: om-role.service-account-token
  annotations:
    kubernetes.io/service-account.name: om-role
type: kubernetes.io/service-account-token
EOF

Configure Kubernetes Roles for the Service Account

Create a file om-argo-role.yml:

# om-argo-role.yml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: om-argo-role
  namespace: collate
rules:
  - verbs: [list, watch, create, update, patch, get, delete]
    apiGroups:
      - argoproj.io
    resources:
      - workflows
  - verbs: [list, watch, patch, get]
    apiGroups:
      - ''
    resources:
      - pods/log
      - pods
  - verbs: [list, watch, create, update, patch, get, delete]
    apiGroups:
      - argoproj.io
    resources:
      - cronworkflows
  - verbs: [create, patch]
    apiGroups:
      - argoproj.io
    resources:
      - workflowtaskresults

Apply the role and create the role binding:

kubectl apply -f om-argo-role.yml

kubectl create rolebinding om-argo-role-binding \
  --role=om-argo-role \
  --serviceaccount=collate:om-role \
  --namespace collate

Install OpenMetadata Helm Chart

Create Kubernetes Secrets for the database connection:

kubectl create secret generic db-credentials \
  --from-literal=password=<<DATABASE_PASSWORD>> \
  --namespace collate

Add the Helm chart repository:

helm repo add open-metadata https://helm.open-metadata.org/
helm repo update

If you plan to use the DeltaLake connector, the ARGO_INGESTION_IMAGE value should be: 118146679784.dkr.ecr.eu-west-1.amazonaws.com/collate-customers-ingestion-eu-west-1:om-1.13.0-cl-1.13.0

Create a file openmetadata.values.yml:

# openmetadata.values.yml
replicaCount: 1
openmetadata:
  config:
    elasticsearch:
      host: ${es_host}
      port: ${es_port}
      scheme: ${es_scheme}
      searchType: opensearch
      auth:
        enabled: true
        username: ${es_username}
        password:
          secretRef: es-credentials
          secretKey: password
    database:
      host: ${db_host}
      port: ${db_port}
      driverClass: org.postgresql.Driver
      dbScheme: postgresql
      auth:
        username: ${db_user}
        password:
          secretRef: db-credentials
          secretKey: password
      dbParams: "allowPublicKeyRetrieval=true&useSSL=true&serverTimezone=UTC"
    pipelineServiceClientConfig:
      className: "io.collate.pipeline.argo.ArgoServiceClient"
      apiEndpoint: "http://argo-workflows-server.argo-workflows:2746"
      metadataApiEndpoint: "http://openmetadata:8585/api"
      auth:
        enabled: false
image:
  repository: 118146679784.dkr.ecr.eu-west-1.amazonaws.com/collate-customers-eu-west-1
  tag: om-1.13.0-cl-1.13.0
  imagePullPolicy: IfNotPresent
imagePullSecrets:
  - name: ecr-registry-creds
extraEnvs:
  - name: ARGO_NAMESPACE
    value: collate
  - name: ARGO_TOKEN
    valueFrom:
      secretKeyRef:
        name: "om-role.service-account-token"
        key: "token"
  - name: ARGO_INGESTION_IMAGE
    value: "118146679784.dkr.ecr.eu-west-1.amazonaws.com/collate-customers-ingestion-slim-eu-west-1:om-1.13.0-cl-1.13.0"
  - name: ARGO_WORKFLOW_EXECUTOR_SERVICE_ACCOUNT_NAME
    value: om-role
  - name: ARGO_IMAGE_PULL_SECRETS
    value: ecr-registry-creds
  - name: ASSET_UPLOADER_PROVIDER
    value: "s3"
  - name: ASSET_UPLOADER_MAX_FILE_SIZE
    value: "10485760"
  - name: ASSET_UPLOADER_S3_ENDPOINT
    value: "http://minio.minio:9000"
  - name: ASSET_UPLOADER_S3_BUCKET_NAME
    value: "argo-artifacts"
  - name: ASSET_UPLOADER_S3_PREFIX_PATH
    value: "assets/collate"
serviceAccount:
  name: "openmetadata"

Install the Collate OpenMetadata Application:

helm upgrade --install openmetadata open-metadata/openmetadata \
  --values openmetadata.values.yml \
  --namespace collate

[Optional] Enable Prometheus Metrics

Collate Application exposes Prometheus metrics on port 8586. Enable the integration using:

serviceMonitor:
  enabled: true

Post Installation/Upgrade Steps

Configure ReIndexing

After installation or upgrade, configure ReIndexing from the OpenMetadata UI. For detailed steps, refer to the OpenMetadata upgrade documentation.

Troubleshooting

Pods Stuck in Pending State

Check for resource constraints or missing secrets:

kubectl describe pod -n collate -l app.kubernetes.io/name=openmetadata

Symptom	Cause	Fix
ImagePullBackOff	ECR secret missing or expired	Recreate ecr-registry-creds with a fresh ECR token
Insufficient cpu / memory	Cluster at capacity	Reduce resources.requests in openmetadata.values.yml or add nodes
Pending on PVC	No default StorageClass	Set a default StorageClass or pass explicit storageClass in values

Argo Workflows Cannot Connect to Object Storage

Verify the MinIO service is reachable from the argo-workflows namespace:

kubectl run test --image=curlimages/curl --rm -it --restart=Never -n argo-workflows -- \
  curl -s http://minio.minio:9000/minio/health/live

Expected response: 200 OK. If it fails, check that MinIO is running and the endpoint in argo-workflows.values.yml is correct.

Environment Variables for Collate OpenMetadata Argo

Environment Name	Description	Default Value	Required
ARGO_IMAGE_PULL_SECRETS	Image Pull Secret Name to pull Docker Images for Ingestion from a Private Registry. Multiple secrets can be supplied comma-separated.	Empty String	False
ARGO_INGESTION_IMAGE	Docker Image and Tag for Ingestion Images	openmetadata/ingestion-base:1.13.0	True
ARGO_NAMESPACE	Namespace in which Argo Workflows will be executed. Must match the namespace where OpenMetadata is deployed.	argo	True
ARGO_SERVER_CERTIFICATE_PATH	SSL Certificate Path to connect to Argo Server	Empty String	False
ARGO_TEST_CONNECTION_BACKOFF_TIME	Backoff retry time in seconds to test the connection	5	False
ARGO_TOKEN	JWT Token to authenticate with Argo Workflow API	Empty String	True
ARGO_WORKFLOW_CPU_LIMIT	Kubernetes CPU Limits for Argo Workflows created with Ingestion	1000m	False
ARGO_WORKFLOW_CPU_REQUEST	Kubernetes CPU Requests for Argo Workflows created with Ingestion	200m	False
ARGO_WORKFLOW_CUSTOMER_TOLERATION	Kubernetes Node Toleration to schedule Ingestion Workflow Pods to specific Nodes	argo	False
ARGO_WORKFLOW_EXECUTOR_SERVICE_ACCOUNT_NAME	Service Account Name to be used for Argo Workflows for Ingestion	om-role	True
ARGO_WORKFLOW_MEMORY_LIMIT	Kubernetes Memory Limits for Argo Workflows created with Ingestion	4096Mi	False
ARGO_WORKFLOW_MEMORY_REQUEST	Kubernetes Memory Requests for Argo Workflows created with Ingestion	256Mi	False
ASSET_UPLOADER_ENABLE	Enable Asset Upload Feature	True	False
ASSET_UPLOADER_PROVIDER	Asset Upload Provider Name. Can be s3 or azure.	s3	False
ASSET_UPLOADER_MAX_FILE_SIZE	Max File Size to support for Asset Upload (in bytes)	5242880	False
ASSET_UPLOADER_S3_ENDPOINT	Custom S3-compatible endpoint (e.g. MinIO)	Empty String	False
ASSET_UPLOADER_S3_BUCKET_NAME	Asset Upload S3/MinIO Bucket Name	Empty String	False
ASSET_UPLOADER_S3_PREFIX_PATH	Asset Upload S3/MinIO Prefix Path	assets/default	False