Processing API-DataV(DataV)-阿里云帮助中心

The timeline feature shows how data changes over time. To update your isarithmic map data using the timeline, you must create an API or a database. The API retrieves data from national monitoring sites for specific time periods based on a time parameter. This document describes how to build and publish this API using Node.js. The same principles apply to other programming languages.

Background information

You can wrap the processed data from the previous step into a simple API. Then, in the data panel of the DataV map widget, you can configure an API data source to call this API.

The API information is as follows:

Request URL: /api
Request method: GET
Request parameters:
- Parameter name: date.
- Parameter type: string. For example, 2017012722. The time format is YYYYmmDDHH.

Procedure

Install Node.js (including npm) and use the npm install <module_name> command to install dependency modules.

Process all the downloaded data. You can use the glob module in Node.js to batch process all data files in a folder.

var fs = require('fs');
var csv = require("fast-csv");
var glob = require('glob');
var mapdata = require('./site_list_lat_lng_mapping.json');
glob("./sites_20170101-20170202/*.csv", function (err, files) {
files.forEach(function (file) {
 var filename = file.replace(/^.*[\\\/]/, '').split('.')[0].split('_')[2];
 var datas = {};
 csv
   .fromPath(file, { headers: true, objectMode: true })
   .on("data", function (data) {
     if (data.type === 'AQI') {
       datas[data.hour] = [];
       for (var key in data) {
         if (mapdata[key]) {
           datas[data.hour].push({
             name: mapdata[key].name,
             value: +data[key],
             code: mapdata[key].code,
             city: mapdata[key].city,
             lng: +mapdata[key].lng,
             lat: +mapdata[key].lat
           })
         }
       }
     }
   })
   .on("end", function () {
     fs.writeFile('./data/' + filename + '.json', JSON.stringify(datas));
     console.log("done");
   });
});
});

The following figure shows the result. 运行结果

Use the glob module to consolidate the data.

Save the following sample program as a JS script file and run it in the Node.js environment.

// The following method is not suitable for large amounts of data.
// Use the file name, which is the date, as the key and the content as the value to get a consolidated all.json file.
var fs = require('fs');
var csv = require("fast-csv");
var glob = require('glob');
glob("./data/*.json", function (err, files) {
var datas = {};
files.forEach(function (file) {
 var filename = file.replace(/^.*[\\\/]/, '').split('.')[0];
 datas[filename] = require(file);
});
fs.writeFile('./data/all.json', JSON.stringify(datas));
console.log('done');
});

In the Node.js environment, use the Express framework to initialize an Express project. Then, add a simple API that meets the preceding requirements. The following code provides an example.
Note To prevent issues with cross-origin requests, add the cors module to the app.js file.
After you create the API, use the npm start command in the Node.js environment to test it. The following figure shows a successful test.