Choosing between HBase SQL (Phoenix) and Spark
Updated at:
This topic describes the use cases for Phoenix and Spark and compares them to help you make a choice.
Scenarios
- ApsaraDB for HBase Phoenix is the SQL layer of ApsaraDB for HBase. It is designed for scenarios that require
high concurrency, low latency, and simple queries. It can also handle some analytics needs. Queries must hit an index and return a small amount of data. For join operations, the data returned from one side of the join must be less than 100,000 rows, and the query on the other side must hit an index. To ensure cluster stability, the platform rejects complex or time-consuming SQL statements. - ApsaraDB for HBase Spark is the analytics engine for ApsaraDB for HBase. It is suitable for scenarios with
low concurrency, high latency, and complex computations. It can execute SQL statements of any complexity. Spark also supports SQL, Scala, Java, and Python. It supports streams, Online Analytical Processing (OLAP), batch analytics, data cleaning, and multiple data sources, such as HBase, MongoDB, Redis, and OSS. Spark Streaming supports near real-time active streams, which is beyond the scope of this topic.
Comparison
| Item | Phoenix | Spark |
| SQL complexity | Supports simple queries. Queries must hit an index and return a small amount of data. For join operations, the data returned from one side must be less than 100,000 rows, and the query on the other side must hit an index. To ensure cluster stability, the platform rejects complex or time-consuming SQL statements. | Supports all queries. You can use Spark with Phoenix to achieve performance comparable to Phoenix for simple SQL queries. However, Spark is designed for analytics and is fundamentally different from Phoenix, which is purely for transactional processing (TP). |
| Cluster | Shares a cluster with HBase. It is the SQL layer provided by HBase. | Spark requires a separately purchased cluster. |
| Concurrency | About 10,000 to 50,000 per machine. | The maximum value for Spark is 100. |
| Latency | Millisecond-level latency. SQL statements that retrieve large amounts of data can take seconds. | Typically over 300 ms. Most SQL statements take seconds, minutes, or even hours. |
| Updates | Phoenix is supported. | Not supported. |
| Supported Services | Online services. | Offline or near-online services. |
simple queries, high concurrency, low latency, and online services.complex computations, low concurrency, high latency, offline services, and near-online services.
Example
The following example shows that Spark is mainly used for stream extract, transform, and load (ETL) and secondary data processing. Online queries are handled by Phoenix.
Is this page helpful?