Challenges of big data security governance

Updated at:

To effectively implement data security governance, you must understand key issues such as which assets need protection, the risks they face, whether your organization is compliant, how to handle internal and external attacks, and how to implement security operations. At the same time, big data systems present many security governance challenges due to their characteristics related to storage, users, entry points, data forwarding, and delivery.

Key questions for data security governance

Your ability to answer the following questions clearly and accurately demonstrates how effectively you can implement a security governance project.image.png

What assets need protection?

What assets do you have? Where are they located and what business services do they support? What data do these assets contain? What are the business properties and importance of the data? Have you performed data classification?

Note

The term asset here is not limited to structured and unstructured data assets. It also includes hardware assets, IaaS/PaaS/SaaS, and even personnel. A breach of any single asset can compromise an organization's critical information.

What risks do these assets face?

In real-world business scenarios, an organization's data assets typically go through collection, transmission, storage, processing, use, exchange, and destruction phases. What risks exist in each of these phases? How can the risks in each phase be assessed using qualitative and quantitative methods, such as Annualized Loss Expectancy (ALE) and Annualized Cost of Security (ACS)? What methods can an organization use to detect and predict risks?

Risk assessment helps all business units in an organization understand the risks associated with their assets. This understanding helps senior management justify and implement security projects. After a risk assessment, the organization can adopt different strategies to address risks based on the results:

  • Risk Mitigation: Deploy security protections to reduce the probability of a data security event.

  • Risk Transfer: Purchase commercial insurance to avoid losses when the cost of protection is too high.

  • Risk Acceptance: Take no protective measures when the data value is low and the cost of protection is high.

  • Risk Rejection: Take no protective measures if the probability of an attack is deemed extremely low.

  • Risk Deterrence: Deter attackers by sending them deterrent messages.

  • Risk Avoidance: Take a system offline to prevent it from being attacked.

Is the organization compliant?

Which regulators will review the organization each year? What laws, regulations, or mandatory standards must be followed? What must the organization review to ensure compliance when facing regulators?

How to handle internal and external attacks?

Based on an understanding of asset distribution, asset importance, and risk response strategies, which assets should the organization protect? What methods, tools, and techniques should be used to protect different assets? How can you prevent or quickly detect misuse by internal personnel? How can you defend against various types of external attacks?

How should security operations be implemented?

How should an organization establish a sustainable, platform-based, systematic, and visualized security operations system? What baselines, security policies, risk rules, and response measures should be configured to manage the organization's overall security posture? How should the security team demonstrate its value to senior management and request resources?

Note

Continuous data security operations are critical. Most organizations tend to address issues through multiple, separate projects. This project-based approach is time-consuming, labor-intensive, and difficult to sustain. This is because data security governance is often treated as a one-time project. After the project ends, if no operational rules, security baselines, or risk rules are established, problems such as permission redundancy and data misuse can reappear unchecked.

Characteristics and security governance challenges of big data systems

Answering the questions above is difficult because of the unique characteristics of big data systems in terms of storage, users, entry points, data forwarding, and delivery.image.png

Storage

Big data systems are known for having diverse data types (structured, unstructured, and semi-structured) and large data volumes, often at the petabyte (PB) scale. Some large organizations can add hundreds of thousands or even millions of new tables in a single day. This scale poses an enormous challenge for data classification. Manual data classification is unrealistic and prone to omissions.

Users

Big data systems have a large user base that includes all data-related roles. Common users of big data systems include developers, operations staff, and analysts. Even sales and HR staff query the data they need. This wide variety of user types complicates authorization and management. What permissions should be granted to which personnel? What happens if they leave the company or change departments? This can easily lead to permission creep, excessive authorization, and incomplete permission revocation upon departure. All of these issues create significant data security risks.

Entry points

Because big data systems must serve different roles with varying technical skills, they need to provide different entry points for different types of personnel. For example, technical staff can use the command line, but data analysts or operations staff need to use a visual interface or a Business Intelligence (BI) tool.

Different entry points may have different authentication methods, logic, and audit capabilities. A common issue is the transitive trust risk, where multiple users access or operate data from an entry point using a single identity. In addition, because the audit capabilities of upper-layer systems vary, compliance issues such as missing audit events, missing audit messages, or the inability to audit individual users can occur.

Data forwarding

A big data system is typically an end-to-end suite of data development and governance services. It not only collects and processes data but also provides data to business units. Therefore, it has complex data forwarding paths. These paths include but are not limited to ad hoc query paths, offline transmission paths, real-time transmission paths, DataService Studio API paths, and other underlying API or software development kit (SDK) data export channels.

These paths are direct channels for data confidentiality breaches. Data security events can occur if data forwarding security policies are unclear or undefined, underlying paths are not disabled, API security protections are missing, or personnel behavior is not subject to risk control. Examples include illegal cross-border (or cross-domain) data transfer, database exfiltration, and data leaks.

Delivery

Big data systems are responsible for producing data daily for business decisions. The timeliness and accuracy of this data can directly affect the decisions of senior management. Failure to produce timely and accurate data compromises the integrity and availability of the information.

Workflows in big data systems often involve data from multiple departments, owners, and systems. Coordinating these business systems to produce timely and high-quality data, while avoiding production delays or dirty data caused by system breakdowns or invalid input data, is crucial for business continuity and can even affect the trust of senior management.image.png