This topic describes the phase planning, role responsibilities, and overall flow of data warehouse development specifications.
In the era of big data, standardized data asset management is essential. It drives the integration of the Internet, big data, artificial intelligence, and the real economy. Development specifications that align with business needs and cover all development stages improve efficiency and ensure that data development proceeds smoothly. In contrast, an incomplete development process reduces efficiency and increases costs and risks.
In summary, data asset management is essentially the management of objects, whereas R&D process specification management is the management of human behavior. You can implement the data asset management methodology only after the management of human behavior is established as a foundation.
Data warehouse development specifications provide standardized guidance for data developers and managers. The goal is to simplify daily workflows, improve efficiency, and reduce redundant work. This allows businesses and governments to better control growing data volumes and free up resources for business innovation.
Phase planning
Based on common data warehouse development practices, the development process can be broken down into the following phases:
Requirements phase: Data product managers handle changing business needs.
Design phase: Data product managers and developers balance factors such as performance, cost, efficiency, and quality to determine the best way to organize and store data.
Development phase: Data developers write code efficiently and in accordance with established standards.
Testing phase: Testers identify code issues and project risks to improve the quality of the output.
Release phase: Programs that are ready for production are released smoothly to ensure stable output.
O&M phase: Operations and Maintenance (O&M) engineers ensure the timeliness and stability of data output.
Role responsibilities
Data product manager: Receives and evaluates data requirements from business teams, organizes requirement reviews, creates product requirement documents, and oversees detailed technical reviews.
Designer: Explores data based on the final product requirement document to understand its characteristics, such as data quality and data distribution. They then implement detailed designs for tables, mappings, and scheduling based on these findings.
Developer: Creates a plan and writes code based on the designer's specifications. They also perform unit tests and code reviews.
Tester: Verifies that the results match the requirements. Identifies code issues and project risks.
O&M engineer: Publishes tasks and handles anomalous activity related to data, programs, scheduling, or monitoring and alerting. They ensure timely data output, efficient program operation, and production stability.
Security and compliance personnel: Ensure the security and compliance of the implementation, starting from the early stages of the requirement review.
Overall flow of data warehouse development specifications
The following figure shows the overall flow of the data warehouse development specifications, based on the phases and roles described above.