Configuration file format for synq-recon, the Coalesce Quality data reconciliation tool that compares datasets across databases.
No Additional PropertiesShort machine identifier for the suite.
Must match regular expression:^[a-zA-Z0-9][a-zA-Z0-9_-]*$
Must be at least 1 characters long
Must be at most 255 characters long
Human-readable title for the suite.
Human-readable description.
Database connection definitions.
Each additional property must conform to the following schema
Type: objectConnection is a single database connection entry in a connections map.
No Additional PropertiesDisplay name for this connection. Defaults to the connection ID (map key).
When true, the connection is skipped during execution.
Maximum number of parallel queries. Range: 1-256. Defaults to 8.
Value must be greater or equal to 1 and lesser or equal to 256
PostgresConf contains PostgreSQL connection parameters.
No Additional PropertiesValue must be greater or equal to 1 and lesser or equal to 65535
Disable SSL certificate verification.
SnowflakeConf contains Snowflake connection parameters.
No Additional PropertiesSnowflake account identifier.
Virtual warehouse to use for queries.
Role to assume after connecting.
PEM-encoded private key content for key-pair authentication.
Path to a PEM-encoded private key file.
Passphrase to decrypt the private key.
Databases to include. If empty, all accessible databases are scraped.
No Additional ItemsUse GET_DDL() to retrieve DDL for tables and views.
Database containing the ACCOUNT_USAGE schema. Defaults to SNOWFLAKE.
Set to "externalbrowser" to use SSO browser-based authentication.
BigQueryConf contains BigQuery connection parameters.
No Additional PropertiesGCP project ID.
Region for BigQuery resources.
Inline JSON content of the service account key.
Path to the service account key JSON file.
Explicit list of dataset names to scrape. When set, only these datasets are queried
and project-level bigquery.datasets.list permission is not required.
RedshiftConf contains Amazon Redshift connection parameters.
No Additional PropertiesValue must be greater or equal to 1 and lesser or equal to 65535
Estimate table freshness from Redshift query logs instead of metadata.
MySQLConf contains MySQL connection parameters.
No Additional PropertiesValue must be greater or equal to 1 and lesser or equal to 65535
Disable SSL certificate verification.
Additional DSN parameters passed to the driver.
Each additional property must conform to the following schema
Type: stringClickhouseConf contains ClickHouse connection parameters.
No Additional PropertiesValue must be greater or equal to 1 and lesser or equal to 65535
Name this ClickHouse is published under, and the top element of every path
and breadcrumb its tables appear in — "prod", "staging", "eu-analytics".
ClickHouse has no container above a database, so something has to name the
service itself. Left empty, that is the host above, which is correct but
unreadable for a ClickHouse Cloud endpoint; a name given here replaces it.
Two connections to the same service must give the same name, and the name is
the identity of everything scraped through it: changing it republishes those
tables under new paths, and the old ones stop being produced. Pick one per
service and keep it.
"prod"
"staging"
Database the connection opens with, so an unqualified table name in a query
resolves against it. It does not restrict what is scraped: metadata comes
from system tables and covers every database the user can see, whatever this
says. Empty opens on "default".
Disable SSL certificate verification.
ClickHouse server settings applied to every connection, written as they
would be in a DSN query string (e.g. maxexecutiontime: "300"). Values are
typed the way ClickHouse types them in a connection string: "true" and
"false" become 1 and 0, whole numbers become integers, anything else is
passed through as text. A name given here replaces the value the scrape
would otherwise use.
Each additional property must conform to the following schema
Type: stringHow ClickHouse system tables are read. Omit to keep reading across a
cluster named "default", which every ClickHouse Cloud service provides.
How system tables are read. Optional — defaults to "allreplicas".
- "allreplicas": read through clusterAllReplicas(<name>, ...) so metadata
covers every replica. Requires GRANT REMOTE ON ..
- "singlenode": read on the connected node only. The one setting that
works on an install whose remoteservers defines no cluster, and it
needs no REMOTE grant — but on an install that does have replicas it
reports the metadata of a single node, so choose it deliberately.
"all_replicas"
"single_node"
Cluster to read through, as named under remoteservers in the ClickHouse
configuration. SELECT DISTINCT cluster FROM system.clusters lists what a
server has. Empty means "default". Ignored when mode is "singlenode".
TrinoConf contains Trino / Starburst connection parameters.
No Additional PropertiesValue must be greater or equal to 1 and lesser or equal to 65535
Use a plain HTTP connection instead of HTTPS.
Catalogs to include. Required for most Trino deployments.
No Additional ItemsDatabricksConf contains Databricks connection parameters.
No Additional PropertiesDatabricks workspace URL.
Personal access token for authentication.
OAuth client ID (M2M authentication).
OAuth client secret (M2M authentication).
SQL warehouse ID to use for queries.
MSSQLConf contains Microsoft SQL Server / Azure SQL Database connection parameters.
No Additional PropertiesValue must be greater or equal to 1 and lesser or equal to 65535
Trust the server certificate without validation.
Encryption mode (e.g. "true", "false", "strict").
Federated authentication method (e.g. "ActiveDirectoryDefault").
Pre-acquired access token for Azure AD authentication.
Azure AD application client ID for service principal auth.
OracleConf contains Oracle Database connection parameters.
No Additional PropertiesValue must be greater or equal to 1 and lesser or equal to 65535
Oracle service name.
Enable SSL/TLS for the connection.
Verify the server's SSL certificate.
Path to Oracle Wallet directory for authentication.
Enable Oracle Diagnostics Pack features (AWR, ASH).
DuckDBConf contains DuckDB / MotherDuck connection parameters.
No Additional PropertiesFile path, ':memory:' for in-memory, or MotherDuck database name.
MotherDuck organization/account name (for cloud mode).
MotherDuck authentication token (required for cloud MotherDuck).
AthenaConf contains Amazon Athena connection parameters.
No Additional PropertiesAWS region hosting the Athena service and Glue Data Catalog.
Athena workgroup. Defaults to "primary" when empty. Must have a
ResultConfiguration.OutputLocation configured.
Glue Data Catalog name. Defaults to "AwsDataCatalog" when empty.
Static AWS credentials. Pair accesskeyid with secretaccesskey.
Optional STS session token, when accesskeyid+secretaccesskey are
short-lived STS credentials.
Named AWS shared-config profile (from ~/.aws/credentials or
~/.aws/config). Used only when static credentials are absent.
IAM role ARN to assume via STS. Wraps whichever base credentials
resolved above (or the host's default chain when no other auth is set).
External ID required by the role's trust policy. Pair with role_arn.
Optional STS session name. Defaults to "synq-athena".
Scope filter for include/exclude filtering by Glue catalog, Glue
database, and table. Mapping: ScopeRule.database = Glue catalog,
ScopeRule.schema = Glue database, ScopeRule.table = Glue table/view.
Include rules. If non-empty, only matching objects are accepted.
No Additional ItemsScopeRuleConf is a single include/exclude rule.
No Additional PropertiesDatabase-level pattern (catalog for Athena/Trino/Databricks, project for BigQuery).
Schema-level pattern (Glue database for Athena, dataset for BigQuery).
Table or view name pattern.
Exclude rules. Matching objects are rejected, even if they match an include rule.
No Additional ItemsScopeRuleConf is a single include/exclude rule.
Same definition as connections_additionalProperties_athena_scope_include_itemsUse SHOW CREATE TABLE to retrieve full table DDL (CTAS bodies, Iceberg
TBLPROPERTIES, Hive external LOCATION/SerDe). One Athena query per
table — billed at the 10MB scan minimum each.
Use SHOW CREATE VIEW to retrieve full view DDL instead of the
rewritten body from informationschema.views.viewdefinition.
For Iceberg tables, fan out one Athena query per table to read row
count, total file size, snapshot commit timestamp, and partition
columns from the table's $files / $snapshots / $partitions metadata
tables. Hive externals are unaffected.
FabricConf contains the connection settings for a Microsoft Fabric Warehouse or Lakehouse SQL analytics endpoint.
No Additional PropertiesHostname of the workspace's SQL analytics endpoint. Copy it from the
Fabric portal: open your Warehouse or Lakehouse, then Settings → SQL
connection string.
"my-workspace.datawarehouse.fabric.microsoft.com"
Default database for queries that don't name one explicitly. Optional —
defaults to "master". Because metadata and metric queries are always
fully qualified, this only affects ad-hoc SQL that omits the database.
"my_warehouse"
How to authenticate to Fabric. Optional — defaults to a service principal
(clientid + clientsecret). Values are matched case-insensitively, and
the equivalent dbt-fabric and Microsoft ODBC spellings are also accepted:
- "serviceprincipal" (default): Entra ID service principal. Set
clientid, clientsecret and, if needed, tenantid.
- "azurecli": reuse the machine's az login session. On-prem agent only.
- "default": try Azure's default credential chain (environment, managed
identity, CLI, ...). On-prem agent only.
- "managedidentity": use an Azure managed identity; set client_id to
select a user-assigned identity. On-prem agent only.
"service_principal"
"azure_cli"
"default"
"managed_identity"
Application (client) ID of the Entra ID service principal. When authtype
is "managedidentity", this instead selects a user-assigned identity.
"00000000-0000-0000-0000-000000000000"
Client secret for the service principal. Supply it through an environment
variable (e.g. ${FABRICCLIENTSECRET}) rather than committing it in plain
text.
"${FABRIC_CLIENT_SECRET}"
Entra ID tenant (directory) ID. Optional — inferred from the endpoint
hostname when omitted. Set it only when the service principal lives in a
different tenant than the workspace.
"00000000-0000-0000-0000-000000000000"
A pre-acquired Entra ID OAuth access token for the SQL scope
(https://database.windows.net/.default). Optional — when set it overrides
every other authentication method. Mainly for hosted deployments that mint
their own token.
"${FABRIC_ACCESS_TOKEN}"
Optional include/exclude filter that limits which databases, schemas and
tables are scanned. When omitted, the whole workspace is scanned.
Reconciliation scenario definitions.
Each additional property must conform to the following schema
Type: objectReconciliation defines a single reconciliation scenario.
No Additional PropertiesHuman-readable title.
Human-readable description.
Source dataset.
No Additional PropertiesConnection name (must exist in connections map).
Base SQL query (mutually exclusive with 'table'). Supports {{variable_name}} interpolation.
Table reference (mutually exclusive with 'query'). Accepts "db.schema.table" string or structured object.
Table in dotted format: 'table', 'schema.table', or 'database.schema.table'
Structured table reference
Database/catalog name
Schema/dataset name
Table or view name
Must be at least 1 characters long
Explicit column list (only with 'table'). Mutually exclusive with 'exclude_columns'.
No Additional ItemsColumns to exclude (only with 'table', resolved at runtime). Mutually exclusive with 'columns'.
No Additional ItemsSQL WHERE clause filter (only with 'table'). Applied as WHERE (condition). Supports {{variable_name}} interpolation.
Time-travel timestamp. Wraps query with database-specific time-travel syntax.
Single key column for bisection split points and aggregate grouping.
Deprecated in favor of keycolumns; accepts a string or a one-element list.
Set either keycolumn or key_columns, not both.
Ordered key columns for bisection split points and aggregate grouping.
Supports composite (multi-column) keys; the bisection orders and
range-filters on the column tuple. Accepts a string or a list of strings.
Reconciliation mode.
Hash algorithm for row checksums.
Automatically match columns case-insensitively.
Column name mapping between source and target.
Map source column names to target column names (object format)
Each additional property must conform to the following schema
Type: stringMap source column names to target column names (array format)
No Additional ItemsMust be at least 1 characters long
Must be at least 1 characters long
BisectionConfig controls the bisection drill-down behavior.
No Additional PropertiesNumber of segments per bisection level.
Value must be greater or equal to 2 and lesser or equal to 1024
Stop drilling when segment row count falls below this.
Value must be greater or equal to 1
Segmentation strategy.
Column for time-based partitioning (required when strategy is 'time').
Time truncation granularity.
ReportingConfig controls what level of detail is reported.
No Additional PropertiesValue must be greater or equal to 1
AggregateConfig defines aggregate comparison settings.
No Additional PropertiesMust contain a minimum of 1 items
MeasureConfig defines a column + aggregation function pair.
No Additional PropertiesMust be at least 1 characters long
Single aggregation function
Multiple aggregation functions
Must contain a minimum of 1 items
Column(s) defining the drill-down hierarchy.
Single group column
Hierarchical group columns for drill-down
Must contain a minimum of 1 items
ThresholdConfig defines tolerance for aggregate differences.
No Additional PropertiesMaximum absolute difference.
Value must be greater or equal to 0
Maximum percentage difference (0.1 = 10%).
Value must be greater or equal to 0
Percentage formula: source, target, or symmetric.
Per group_column threshold overrides.
Each additional property must conform to the following schema
Type: objectThresholdConfig defines tolerance for aggregate differences.
Same definition as thresholdsPer measure threshold overrides, keyed by FUNC(column).
Each additional property must conform to the following schema
Type: objectThresholdConfig defines tolerance for aggregate differences.
Same definition as thresholdsErrorHandlingConfig controls retry behavior for query failures.
No Additional PropertiesPer-query timeout as a Go duration string (e.g., '30s', '5m').
Maximum retry attempts for transient query failures.
Value must be greater or equal to 0 and lesser or equal to 10
Initial backoff delay before the first retry.
Backoff multiplier applied after each retry.
Value must be greater or equal to 1
WindowConfig defines time-windowed reconciliation settings.
No Additional PropertiesColumn being windowed.
How far back to look from the current time (e.g., '14d', '2h').
Windowing strategy: 'sliding' or 'fixed'.
Dynamic cutoff filter. Derives a watermark value from actual data to exclude rows not yet synced.
No Additional PropertiesSimple: same column for both source and target watermark derivation.
Medium: different columns per side.
Advanced: full per-side watermark derivation config.
No Additional PropertiesColumn to derive the watermark from (e.g., "createdat", "syncedat").
Required when query is not set.
Aggregate function for watermark derivation: MAX (default) or MIN.
Custom SQL query for watermark derivation. Must return a single row with a "watermark" column.
When not set, auto-generated from the dataset at runtime.
How to combine watermarks when both sides are configured: min, max, source, target.
Truncate the combined watermark to a time boundary before applying.
Time offset applied after truncation. Negative durations create a safety buffer (e.g., '-30m').
Advanced: custom apply config for WHERE clause generation.
No Additional PropertiesCutoffApplySide configures WHERE clause generation for one side.
No Additional PropertiesColumn to filter on. When not set, uses the derivation column.
Comparison operator. Default: "<=".
CutoffApplySide configures WHERE clause generation for one side.
Same definition as sourceAnnotations attached to this reconciliation case. Merged with suite-level annotations.
Canonical list form. Each entry has a name and optional values.
No Additional ItemsMust be at least 1 characters long
Must be at most 50 characters long
Must contain a maximum of 20 items
Must be at least 1 characters long
Must be at most 50 characters long
Map shorthand: key is annotation name; value is null (name-only), a single string, or an array of strings.
Each additional property must conform to the following schema
Must be at least 1 characters long
Must be at most 50 characters long
Must contain a maximum of 20 items
Must be at least 1 characters long
Must be at most 50 characters long
Per-reconciliation setup/teardown.
Each additional property must conform to the following schema
Type: objectEach additional property must conform to the following schema
Type: objectEach additional property must conform to the following schema
Type: objectEach additional property must conform to the following schema
Type: objectWhether teardown runs on failure. Inherits from suite level when not set.
When true, logs setup errors as warnings. Inherits from suite level when not set.
Template variables for query interpolation via {{variable_name}} syntax.
Each additional property must conform to the following schema
Type: stringSetup queries run once before all reconciliations begin.
Each additional property must conform to the following schema
Type: objectTeardown queries run once after all reconciliations complete.
Each additional property must conform to the following schema
Type: objectSQL file(s) to load and run as setup queries.
Each additional property must conform to the following schema
Type: objectSQL file(s) to load and run as teardown queries.
Each additional property must conform to the following schema
Type: objectWhether teardown runs even when reconciliations fail with infrastructure errors.
When true, logs setup query errors as warnings and continues instead of aborting.
When true, time reference detection (NOW(), CURRENT_DATE, etc.) returns an error instead of a warning.
Coalesce Quality API credentials for sending audit logs.
No Additional PropertiesOAuth client ID for authenticating with Coalesce Quality.
OAuth client secret for authenticating with Coalesce Quality.
gRPC endpoint for the Coalesce Quality API (e.g. "developer.synq.io:443").
gRPC endpoint for the Coalesce Quality ingest API. Defaults to Endpoint if empty.
OAuth token URL. Derived from Endpoint if empty.