Build a meteorological grid data system

Background

Meteorological grid data is multi-dimensional spatiotemporal data. Daily volumes range from tens to hundreds of terabytes and continue to grow, demanding high storage scalability and flexible query capabilities.

A grid dataset is a five-dimensional structure: variables (temperature, humidity, wind speed, and others), time, altitude, longitude, and latitude. Weather forecasting and analysis require slicing queries across these dimensions in various combinations.

• Query a latitude-longitude plane of grid data. For example, retrieve global surface temperatures for the next three hours.

• Query time-series data at a specific grid point. For example, retrieve temperature changes at a location from the next 3 hours to 72 hours.

• Query data for different variables. For example, retrieve all variables at a specific time, altitude, and point.

• Query data produced by different model systems. For example, compare forecast data from different meteorological agencies.

Traditional solutions store grid data in relational databases with file systems, but scalability and query performance degrade as data volume grows.

Tablestore-Grid is a sample project for storing and querying five-dimensional grid data. With automatic horizontal scaling and search index-based retrieval, it supports petabyte-scale storage and millisecond-level multi-dimensional queries.

Data model

A five-dimensional grid dataset (GridDataSet) consists of the following dimensions.

GridDataSet = F(variable, time, z, x, y). Each dataset also contains the following metadata.

This solution uses two tables: a meta table for dataset metadata and a data table for grid data.

meta table: The primary key is GridDataSetId. Attribute columns store dimension information (size and data type of each dimension) and custom attributes. A search index on the meta table enables combined queries and sorting by source, resolution, status, and creation time.

data table: Four primary key columns locate each row of data.

These four primary key columns locate a single row that stores the two-dimensional plane formed by the remaining dimensions (x and y). To balance full-scan and localized query performance, the plane data is split into blocks and stored across multiple attribute columns. A query reads only the blocks that cover the target area.

Quick start

Download the sample code for the meteorological grid data system. The project covers table creation, data ingestion, slicing queries, and multi-condition retrieval.

• Download the sample code: grid.demo.zip.

• Create a configuration file named tablestoreConf.json in your home directory. Specify the endpoint, AccessKey ID, AccessKey Secret, and instance name.

• Prepare a NetCDF test data file and set the file path in ExampleConfig.java.

• Run the examples in order: CreateStoreExample (create tables and indexes), DataImportExample (ingest data), DataFetchExample (slicing queries), and MetaQueryExample (multi-condition retrieval).

Implementation

To build a meteorological grid data system, use the Tablestore-Grid sample project:

// Initialize TableStoreGrid
TableStoreGridConfig config = new TableStoreGridConfig();
config.setTableStoreEndpoint(endpoint);
config.setAccessId(accessKeyId);
config.setAccessKey(accessKeySecret);
config.setTableStoreInstance(instanceName);
config.setDataTableName("grid_data_table");
config.setMetaTableName("grid_meta_table");

TableStoreGrid tableStoreGrid = new TableStoreGrid(config);

// Create meta table and data table
tableStoreGrid.createStore();

// Create search index on meta table for multi-condition query
IndexSchema indexSchema = new IndexSchema();
indexSchema.setFieldSchemas(Arrays.asList(
        new FieldSchema("status", FieldType.KEYWORD).setIndex(true).setEnableSortAndAgg(true),
        new FieldSchema("source", FieldType.KEYWORD).setIndex(true).setEnableSortAndAgg(true),
        new FieldSchema("accuracy", FieldType.KEYWORD).setIndex(true).setEnableSortAndAgg(true),
        new FieldSchema("create_time", FieldType.LONG).setIndex(true).setEnableSortAndAgg(true)
));
tableStoreGrid.createMetaIndex("grid_meta_index", indexSchema);

// Step 1: Write dataset metadata
GridDataSetMeta meta = new GridDataSetMeta(
        "forecast_20260611",           // GridDataSetId
        DataType.FLOAT,                // Data type
        Arrays.asList("temperature"),  // Variables
        24, 10, 720, 1440,            // Dimensions: time=24, z=10, x=720, y=1440
        new StoreOptions(StoreOptions.StoreType.SLICE));

Map<String, Object> attributes = new HashMap<>();
attributes.put("source", "ECMWF");
attributes.put("accuracy", "0.25deg");
attributes.put("status", "INIT");
attributes.put("create_time", System.currentTimeMillis());
meta.setAttributes(attributes);
tableStoreGrid.putDataSetMeta(meta);

// Step 2: Import data from NetCDF file
GridDataWriter writer = tableStoreGrid.getDataWriter(meta);
NetcdfFile ncFile = NetcdfFile.open("forecast_data.nc");
for (Variable variable : ncFile.getVariables()) {
    if (meta.getVariables().contains(variable.getShortName())) {
        for (int t = 0; t < meta.gettSize(); t++) {
            for (int z = 0; z < meta.getzSize(); z++) {
                Array array = variable.read(
                        new int[]{t, z, 0, 0},
                        new int[]{1, 1, meta.getxSize(), meta.getySize()});
                Grid2D grid2D = new Grid2D(
                        array.getDataAsByteBuffer(),
                        variable.getDataType(),
                        new int[]{0, 0},
                        new int[]{meta.getxSize(), meta.getySize()});
                writer.writeGrid2D(variable.getShortName(), t, z, grid2D);
            }
        }
    }
}

// Step 3: Update metadata status to DONE
attributes.put("status", "DONE");
meta.setAttributes(attributes);
tableStoreGrid.updateDataSetMeta(meta);

GridDataFetcher fetcher = tableStoreGrid.getDataFetcher(
        tableStoreGrid.getDataSetMeta("forecast_20260611"));

// Query 1: Get a latitude-longitude plane (fixed time and height)
fetcher.setVariablesToGet(Arrays.asList("temperature"));
fetcher.setOriginShape(
        new int[]{0, 0, 0, 0},       // origin: time=0, z=0, x=0, y=0
        new int[]{1, 1, 720, 1440});  // shape: one full plane
Grid4D grid4D = fetcher.fetch().getVariable("temperature");
Array planeData = grid4D.toArray();

// Query 2: Get a time series at a specific point and height
fetcher.setOriginShape(
        new int[]{0, 0, 360, 720},    // origin: time=0, z=0, specific point
        new int[]{24, 1, 1, 1});      // shape: all 24 time steps, single point
Grid4D timeSeries = fetcher.fetch().getVariable("temperature");

// Query 3: Get an arbitrary 4D sub-space
fetcher.setOriginShape(
        new int[]{6, 2, 100, 200},    // origin: time=6, z=2, x=100, y=200
        new int[]{12, 5, 50, 80});    // shape: 12 time steps, 5 heights, 50x80 area
Grid4D subSpace = fetcher.fetch().getVariable("temperature");

// Query: (status == DONE) AND (create_time > last 24h)
//        AND (accuracy == "0.25deg")
//        AND (source == "ECMWF" OR source == "NMC")
QueryGridDataSetResult result = tableStoreGrid.queryDataSets(
        "grid_meta_index",
        QueryBuilder.and()
                .equal("status", "DONE")
                .greaterThan("create_time", System.currentTimeMillis() - 86400000)
                .equal("accuracy", "0.25deg")
                .query(QueryBuilder.or()
                        .equal("source", "ECMWF")
                        .equal("source", "NMC")
                        .build())
                .build(),
        new QueryParams(0, 10, new Sort(
                Arrays.<Sort.Sorter>asList(
                        new FieldSort("create_time", SortOrder.DESC)))));

for (GridDataSetMeta meta : result.getGridDataSetMetas()) {
    System.out.println("DataSet: " + meta.getGridDataSetId()
            + ", Attributes: " + meta.getAttributes());
}

Resource cleanup

// Delete search index first, then drop tables
SyncClient syncClient = new SyncClient(endpoint, accessKeyId, accessKeySecret, instanceName);
syncClient.deleteSearchIndex(new DeleteSearchIndexRequest("grid_meta_table", "grid_meta_index"));
syncClient.deleteTable(new DeleteTableRequest("grid_meta_table"));
syncClient.deleteTable(new DeleteTableRequest("grid_data_table"));
syncClient.shutdown();

// Or if using TableStoreGrid instance, close it first
tableStoreGrid.close();