Databricks data source
This topic describes the capabilities and support for synchronizing Databricks data with DataWorks.
Usage notes
Limits
Resource group: Databricks Reader supports only Serverless resource groups. Configure Internet NAT gateway and an EIP for the VPC bound to the resource group.
Read mode: Data synchronization supports only reading through
JDBC.Catalog type and feature limits: To test data source connectivity or configure a synchronization task in wizard mode, DataWorks uses
databricks-sdkto call the Databricks REST API, which supports only Unity Catalog. If your catalog, such ashive_metastore, is not Unity Catalog, these features are unavailable. Choose either of the following options:Option 1 (recommended): Migrate to Unity Catalog. Databricks recommends migrating your data and metadata to Unity Catalog to use all DataWorks features. For instructions, see Migrate to Unity Catalog.
Option 2: Use script mode directly. After configuring the data source in DataWorks, skip Test Connectivity and use script mode to develop the synchronization task.
Concurrent reads and data consistency
Databricks Reader uses
splitPkto split data and start concurrent tasks for faster synchronization. Note the following:Concurrent tasks are not part of the same database transaction and do not start at exactly the same time.
If the data source is still receiving writes, concurrent reads can produce an incomplete or inconsistent snapshot.
Workarounds: Technical limitations prevent a perfectly consistent snapshot across multiple threads. Choose one of these approaches based on your workload:
Option 1: Use single-threaded synchronization without data splitting (
splitPk). This provides strict consistency but is slower.Option 2: Keep the data source static during synchronization, for example by locking the table, temporarily disabling application writes, or pausing replica synchronization. This is faster but may affect production traffic.
Encoding
Databricks Reader extracts data with JDBC, which automatically detects and converts encodings across databases. You do not need to configure encoding manually.
Incremental synchronization
Databricks Reader extracts data by running a
SELECT ... WHERE ...statement. Incremental synchronization depends on constructing theWHEREcondition.Recommended approach (timestamp field):
Add a timestamp field, such as
modify_time, to the source table.Whenever the application inserts or updates a record, including a logical deletion, update the timestamp field too.
Use the timestamp field in the task's
WHEREcondition to fetch changes since the last synchronization point.
Unsupported case: If the business table has no field that distinguishes inserted or updated records, such as a timestamp or auto-incrementing ID, Databricks Reader cannot synchronize incrementally and must perform full synchronization.
Supported field types
For offline reads, Databricks Reader supports most Databricks types, but some types are unsupported. Check your field types.
The following table lists the type conversions supported by Databricks Reader.
Type category | Databricks data type |
Integer | TINYINT, SMALLINT, INT, BIGINT |
Floating point | FLOAT, DOUBLE, DECIMAL |
String | STRING |
Date and time | DATE, TIMESTAMP, TIMESTAMP_NTZ |
Boolean | BOOLEAN |
Complex type | ARRAY, MAP, STRUCT |
Other types | INTERVAL, BINARY, GEOGRAPHY(srid), GEOMETRY(srid) |
Create a data source
Before developing a data synchronization task, create the corresponding data source in DataWorks. For the procedure, see Data source configuration.Explanations of the configuration parameters are available in the configuration interface..
Develop a data synchronization task
See the following guides for entry points and common steps for configuring a synchronization task.
Single-table offline synchronization configuration guide
For the procedure, see Configure an offline synchronization task in wizard mode and Configure an offline synchronization task in script mode.
For all script-mode parameters and a sample script, see Appendix: Sample script and parameter descriptions.
FAQ
Q: A read fails with
[Databricks][JDBCDriver](500313)Error getting the data value from result set: Column13:[Databricks][JDBCDriver](500312)Error in fetching data rows: Timestamp Conversion has failed.A: Databricks TIMESTAMP values can fall outside the range supported by Java Timestamp, causing a JDBC driver error. Modify the
columnconfiguration, for example"column": ["CAST(col_timestamp AS STRING)"].
Appendix: Sample script and parameter descriptions
Offline task script configuration
If you use script mode for an offline task, write the required parameters in the standard script format. For details, see Configure an offline synchronization task in script mode. The following describes the data source parameters in script mode.
Sample Reader script
{
"type": "job",
"version": "2.0",
"steps": [
{
"stepType": "databricks",
"parameter": {
"datasource": "databricks",
"schema": "schema1",
"table": "table1",
"readMode": "jdbc",
"where": "id>1",
"splitPk": "id",
"column": [
"c1",
"c2"
]
},
"name": "Reader",
"category": "reader"
},
{
"stepType": "stream",
"parameter": {},
"name": "Writer",
"category": "writer"
}
],
"setting": {
"errorLimit": {
"record": "0"
},
"speed": {
"concurrent": 1
}
}
}Reader script parameters
Parameter | Description | Required | Default value |
datasource | Data source name. Script mode supports adding data sources; this value must match the name of the added data source. | Yes | None |
column | List of column names to synchronize from the configured table, expressed as a JSON array.
For example:
You must explicitly specify a nonempty set of columns in column. | Yes | None |
splitPk | If you specify
| No | None |
where | Filter condition. Databricks Reader combines it with the specified | No | None |
schema | Schema containing the data to synchronize. | Yes | None |
table | Selected table to synchronize. Each job supports one table. | Yes | None |
readMode | Read mode. Only JDBC is currently supported. | No | jdbc |