Warehouse to Warehouse
Yes, Athia connects to your warehouse or data lake directly. It reads from Snowflake, Databricks, BigQuery or Redshift, including lakehouse tables on Databricks Unity Catalog and external data exposed through your warehouse. Your warehouse admin grants access to a defined set of tables or views, and Athia reads from them on an agreed cadence. There is no pipeline to build, no export to schedule and no file format to agree. When your models change, you change the view.
If you already model payments in your warehouse, this is usually the least work for the most complete data. The joins between attempts, settlement, fees and disputes are already done, and history and the recurrent feed come from the same grant, so there is no separate backfill to arrange.
What your warehouse admin sets up
Four things, on every platform:
- A dedicated machine identity. A user, service account or service principal created for Athia, never a person's account, so access survives staff changes and shows up cleanly in your audit log.
- Read access, scoped to named objects. Two kinds, and the second is the one most often forgotten: permission to read the rows, and permission to read the metadata, the catalog entries that let Athia discover which tables and columns exist. Without the metadata grant, schemas do not appear at all when the connection is set up.
- A decision on how changes are detected. See below. this is the one that affects your bill.
- A network path. How Athia reaches the warehouse.
Per platform
| Platform | Identity | Also needs | Network path |
|---|---|---|---|
| Snowflake | A user with a dedicated role, authenticating by key pair or password | Usage on the database and schema, select on the payment objects, and usage on the compute that runs the reads | Your network policy allowlist, or private connectivity on plans that support it |
| BigQuery | A service account | Dataset and table metadata read, table data read, and a role that can run query jobs. A working dataset for scratch objects, in the same region as the source | IAM-mediated, so there is no IP allowlist step |
| Databricks | A token or service principal on Unity Catalog | Schema usage across the catalog so schemas resolve, plus read on the Delta tables | Allowlist Athia's addresses on your workspace |
| Redshift | A database user | Select on the system catalog views as well as on the payment tables | A security group inbound rule, or private connectivity |
Exact grant statements are issued by your Athia team for your platform and scope, they differ enough between these four that a generic script is not useful.
How changes are detected, and what it costs
This is the decision to make before the grant is written, because it determines both the grant and the running cost.
Change tracking. The warehouse itself records what changed, and Athia reads only the delta. This is the cheaper and faster option, and on large tables it is the difference between a sync that costs cents and one that costs real money.
Full scan with comparison. Athia reads the selected objects each cycle and works out what changed. No change tracking is needed. This is the right answer where policy forbids any write on source objects. but you pay to scan the data on every sync, and that cost scales with table size and how often you sync.
The catch worth knowing before you promise your security reviewer a read-only grant: turning change tracking on generally requires a narrow write permission on the source objects.
Decide these before the grant is written
- Scope. Which databases, catalogs, datasets and schemas. Decide too whether the grant enumerates specific tables or covers future ones: enumerate, and tables created after onboarding stay invisible until someone re-runs the grant.
- Change tracking or full scan. As above. Changing it later means going back to your admin.
- Network posture. Public endpoint with an allowlist, or private connectivity. Private options can carry prerequisites: a particular plan tier, an endpoint to provision, a region to pin.
- Compute and scratch location. Which warehouse or cluster runs the reads, and therefore who pays for them. On BigQuery, also where the working dataset lives, since it must be co-regional with the source.
What commonly goes wrong
| What you see | Usually because |
|---|---|
| Schemas do not appear when the connection is set up | Metadata read was not granted, only row read |
| New tables never show up | The grant enumerated tables rather than covering future ones |
| Warehouse costs climb after go-live | Change tracking was never enabled, so every sync is a full scan |
| A table fails at sync rather than at grant time | It is an object type that cannot be read, external tables, and partitioned tables that require a partition filter, are the usual culprits |
| A table syncs slowly or is refused | It has no primary key. Tables without one are read less efficiently and some platforms limit how large they can be |
Limits worth knowing when you scope
- Primary keys matter. Confirm which of your payment tables have one before you finalise the list.
- Object types. External tables and partition-filter-required tables are generally not readable. Views usually are, by a slower path than tables.
- Databricks requires Unity Catalog and Delta-format tables.
- Exotic data types, nested structures, intervals, geography, may be converted or skipped. If a field you need is one of these, raise it during scoping rather than after the first load.
Related
- Offline Data Transfer, this path against files, and how both compare with real-time connectors.
- Integrating Athia, all the integration paths and how much history to send.
- Athia Data Dictionary, the fields Athia reasons over, and how naming is mapped on ingest.
- Files (SFTP or S3), the alternative where a direct grant is not possible.
Updated about 23 hours ago