Configuration file format for synq-recon database reconciliation tool
No Additional PropertiesShort machine identifier for the suite.
Must match regular expression:^[a-zA-Z0-9][a-zA-Z0-9_-]*$
Must be at least 1 characters long
Must be at most 255 characters long
Human-readable title for the suite.
Human-readable description.
Database connection definitions.
Each additional property must conform to the following schema
Type: objectConnection is a single database connection entry in a connections map.
No Additional PropertiesDisplay name for this connection. Defaults to the connection ID (map key).
When true, the connection is skipped during execution.
Maximum number of parallel queries. Range: 1-256. Defaults to 8.
Value must be greater or equal to 1 and lesser or equal to 256
PostgresConf contains PostgreSQL connection parameters.
No Additional PropertiesValue must be greater or equal to 1 and lesser or equal to 65535
Disable SSL certificate verification.
SnowflakeConf contains Snowflake connection parameters.
No Additional PropertiesSnowflake account identifier.
Virtual warehouse to use for queries.
Role to assume after connecting.
PEM-encoded private key content for key-pair authentication.
Path to a PEM-encoded private key file.
Passphrase to decrypt the private key.
Databases to include. If empty, all accessible databases are scraped.
No Additional ItemsUse GET_DDL() to retrieve DDL for tables and views.
Database containing the ACCOUNT_USAGE schema. Defaults to SNOWFLAKE.
Set to "externalbrowser" to use SSO browser-based authentication.
BigQueryConf contains BigQuery connection parameters.
No Additional PropertiesGCP project ID.
Region for BigQuery resources.
Inline JSON content of the service account key.
Path to the service account key JSON file.
Explicit list of dataset names to scrape. When set, only these datasets are queried
and project-level bigquery.datasets.list permission is not required.
RedshiftConf contains Amazon Redshift connection parameters.
No Additional PropertiesValue must be greater or equal to 1 and lesser or equal to 65535
Estimate table freshness from Redshift query logs instead of metadata.
MySQLConf contains MySQL connection parameters.
No Additional PropertiesValue must be greater or equal to 1 and lesser or equal to 65535
Disable SSL certificate verification.
Additional DSN parameters passed to the driver.
Each additional property must conform to the following schema
Type: stringClickhouseConf contains ClickHouse connection parameters.
No Additional PropertiesValue must be greater or equal to 1 and lesser or equal to 65535
Database to connect to. If empty, all databases are scraped.
Disable SSL certificate verification.
TrinoConf contains Trino / Starburst connection parameters.
No Additional PropertiesValue must be greater or equal to 1 and lesser or equal to 65535
Use a plain HTTP connection instead of HTTPS.
Catalogs to include. Required for most Trino deployments.
No Additional ItemsDatabricksConf contains Databricks connection parameters.
No Additional PropertiesDatabricks workspace URL.
Personal access token for authentication.
OAuth client ID (M2M authentication).
OAuth client secret (M2M authentication).
SQL warehouse ID to use for queries.
MSSQLConf contains Microsoft SQL Server / Azure SQL Database connection parameters.
No Additional PropertiesValue must be greater or equal to 1 and lesser or equal to 65535
Trust the server certificate without validation.
Encryption mode (e.g. "true", "false", "strict").
Federated authentication method (e.g. "ActiveDirectoryDefault").
Pre-acquired access token for Azure AD authentication.
Azure AD application client ID for service principal auth.
OracleConf contains Oracle Database connection parameters.
No Additional PropertiesValue must be greater or equal to 1 and lesser or equal to 65535
Oracle service name.
Enable SSL/TLS for the connection.
Verify the server's SSL certificate.
Path to Oracle Wallet directory for authentication.
Enable Oracle Diagnostics Pack features (AWR, ASH).
DuckDBConf contains DuckDB / MotherDuck connection parameters.
No Additional PropertiesFile path, ':memory:' for in-memory, or MotherDuck database name.
MotherDuck organization/account name (for cloud mode).
MotherDuck authentication token (required for cloud MotherDuck).
AthenaConf contains Amazon Athena connection parameters.
No Additional PropertiesAWS region hosting the Athena service and Glue Data Catalog.
Athena workgroup. Defaults to "primary" when empty. Must have a
ResultConfiguration.OutputLocation configured.
Glue Data Catalog name. Defaults to "AwsDataCatalog" when empty.
Static AWS credentials. Pair accesskeyid with secretaccesskey.
Optional STS session token, when accesskeyid+secretaccesskey are
short-lived STS credentials.
Named AWS shared-config profile (from ~/.aws/credentials or
~/.aws/config). Used only when static credentials are absent.
IAM role ARN to assume via STS. Wraps whichever base credentials
resolved above (or the host's default chain when no other auth is set).
External ID required by the role's trust policy. Pair with role_arn.
Optional STS session name. Defaults to "synq-athena".
Scope filter for include/exclude filtering by Glue catalog, Glue
database, and table. Mapping: ScopeRule.database = Glue catalog,
ScopeRule.schema = Glue database, ScopeRule.table = Glue table/view.
Include rules. If non-empty, only matching objects are accepted.
No Additional ItemsScopeRuleConf is a single include/exclude rule.
No Additional PropertiesDatabase-level pattern (catalog for Athena/Trino/Databricks, project for BigQuery).
Schema-level pattern (Glue database for Athena, dataset for BigQuery).
Table or view name pattern.
Exclude rules. Matching objects are rejected, even if they match an include rule.
No Additional ItemsScopeRuleConf is a single include/exclude rule.
Same definition as connections_additionalProperties_athena_scope_include_itemsUse SHOW CREATE TABLE to retrieve full table DDL (CTAS bodies, Iceberg
TBLPROPERTIES, Hive external LOCATION/SerDe). One Athena query per
table — billed at the 10MB scan minimum each.
Use SHOW CREATE VIEW to retrieve full view DDL instead of the
rewritten body from informationschema.views.viewdefinition.
For Iceberg tables, fan out one Athena query per table to read row
count, total file size, snapshot commit timestamp, and partition
columns from the table's $files / $snapshots / $partitions metadata
tables. Hive externals are unaffected.
Reconciliation scenario definitions.
Each additional property must conform to the following schema
Type: objectReconciliation defines a single reconciliation scenario.
No Additional PropertiesHuman-readable title.
Human-readable description.
Source dataset.
No Additional PropertiesConnection name (must exist in connections map).
Base SQL query (mutually exclusive with 'table'). Supports {{variable_name}} interpolation.
Table reference (mutually exclusive with 'query'). Accepts "db.schema.table" string or structured object.
Table in dotted format: 'table', 'schema.table', or 'database.schema.table'
Structured table reference
Database/catalog name
Schema/dataset name
Table or view name
Must be at least 1 characters long
Explicit column list (only with 'table'). Mutually exclusive with 'exclude_columns'.
No Additional ItemsColumns to exclude (only with 'table', resolved at runtime). Mutually exclusive with 'columns'.
No Additional ItemsSQL WHERE clause filter (only with 'table'). Applied as WHERE (condition). Supports {{variable_name}} interpolation.
Time-travel timestamp. Wraps query with database-specific time-travel syntax.
Column used for bisection split points and aggregate grouping.
Reconciliation mode.
Hash algorithm for row checksums.
Automatically match columns case-insensitively.
Column name mapping between source and target.
Map source column names to target column names (object format)
Each additional property must conform to the following schema
Type: stringMap source column names to target column names (array format)
No Additional ItemsMust be at least 1 characters long
Must be at least 1 characters long
BisectionConfig controls the bisection drill-down behavior.
No Additional PropertiesNumber of segments per bisection level.
Value must be greater or equal to 2 and lesser or equal to 1024
Stop drilling when segment row count falls below this.
Value must be greater or equal to 1
Segmentation strategy.
Column for time-based partitioning (required when strategy is 'time').
Time truncation granularity.
ReportingConfig controls what level of detail is reported.
No Additional PropertiesValue must be greater or equal to 1
AggregateConfig defines aggregate comparison settings.
No Additional PropertiesMust contain a minimum of 1 items
MeasureConfig defines a column + aggregation function pair.
No Additional PropertiesMust be at least 1 characters long
Single aggregation function
Multiple aggregation functions
Must contain a minimum of 1 items
Column(s) defining the drill-down hierarchy.
Single group column
Hierarchical group columns for drill-down
Must contain a minimum of 1 items
ThresholdConfig defines tolerance for aggregate differences.
No Additional PropertiesMaximum absolute difference.
Value must be greater or equal to 0
Maximum percentage difference (0.1 = 10%).
Value must be greater or equal to 0
Percentage formula: source, target, or symmetric.
Per group_column threshold overrides.
Each additional property must conform to the following schema
Type: objectThresholdConfig defines tolerance for aggregate differences.
Same definition as thresholdsPer measure threshold overrides, keyed by FUNC(column).
Each additional property must conform to the following schema
Type: objectThresholdConfig defines tolerance for aggregate differences.
Same definition as thresholdsErrorHandlingConfig controls retry behavior for query failures.
No Additional PropertiesPer-query timeout as a Go duration string (e.g., '30s', '5m').
Maximum retry attempts for transient query failures.
Value must be greater or equal to 0 and lesser or equal to 10
Initial backoff delay before the first retry.
Backoff multiplier applied after each retry.
Value must be greater or equal to 1
WindowConfig defines time-windowed reconciliation settings.
No Additional PropertiesColumn being windowed.
How far back to look from the current time (e.g., '14d', '2h').
Windowing strategy: 'sliding' or 'fixed'.
Dynamic cutoff filter. Derives a watermark value from actual data to exclude rows not yet synced.
No Additional PropertiesSimple: same column for both source and target watermark derivation.
Medium: different columns per side.
Advanced: full per-side watermark derivation config.
No Additional PropertiesColumn to derive the watermark from (e.g., "createdat", "syncedat").
Required when query is not set.
Aggregate function for watermark derivation: MAX (default) or MIN.
Custom SQL query for watermark derivation. Must return a single row with a "watermark" column.
When not set, auto-generated from the dataset at runtime.
How to combine watermarks when both sides are configured: min, max, source, target.
Truncate the combined watermark to a time boundary before applying.
Time offset applied after truncation. Negative durations create a safety buffer (e.g., '-30m').
Advanced: custom apply config for WHERE clause generation.
No Additional PropertiesCutoffApplySide configures WHERE clause generation for one side.
No Additional PropertiesColumn to filter on. When not set, uses the derivation column.
Comparison operator. Default: "<=".
CutoffApplySide configures WHERE clause generation for one side.
Same definition as sourceAnnotations attached to this reconciliation case. Merged with suite-level annotations.
Canonical list form. Each entry has a name and optional values.
No Additional ItemsMust be at least 1 characters long
Must be at most 50 characters long
Must contain a maximum of 20 items
Must be at least 1 characters long
Must be at most 50 characters long
Map shorthand: key is annotation name; value is null (name-only), a single string, or an array of strings.
Each additional property must conform to the following schema
Must be at least 1 characters long
Must be at most 50 characters long
Must contain a maximum of 20 items
Must be at least 1 characters long
Must be at most 50 characters long
Per-reconciliation setup/teardown.
Each additional property must conform to the following schema
Type: objectEach additional property must conform to the following schema
Type: objectEach additional property must conform to the following schema
Type: objectEach additional property must conform to the following schema
Type: objectWhether teardown runs on failure. Inherits from suite level when not set.
When true, logs setup errors as warnings. Inherits from suite level when not set.
Template variables for query interpolation via {{variable_name}} syntax.
Each additional property must conform to the following schema
Type: stringSetup queries run once before all reconciliations begin.
Each additional property must conform to the following schema
Type: objectTeardown queries run once after all reconciliations complete.
Each additional property must conform to the following schema
Type: objectSQL file(s) to load and run as setup queries.
Each additional property must conform to the following schema
Type: objectSQL file(s) to load and run as teardown queries.
Each additional property must conform to the following schema
Type: objectWhether teardown runs even when reconciliations fail with infrastructure errors.
When true, logs setup query errors as warnings and continues instead of aborting.
When true, time reference detection (NOW(), CURRENT_DATE, etc.) returns an error instead of a warning.
SYNQ API credentials for sending audit logs.
No Additional PropertiesOAuth client ID for authenticating with the SYNQ platform.
OAuth client secret for authenticating with the SYNQ platform.
gRPC endpoint for the SYNQ API (e.g. "developer.synq.io:443").
gRPC endpoint for the SYNQ ingest API. Defaults to Endpoint if empty.
OAuth token URL. Derived from Endpoint if empty.