Access OSS with Trino and Hadoop OSS Connector V2

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        Host machine                             │
│                                                                 │
│  /opt/trino-test/                                               │
│  ├── docker-compose.yml                                         │
│  ├── mysql/                                                     │
│  │   └── init.sql                    ← MySQL initialization script│
│  ├── hms/                                                       │
│  │   └── conf/                                                  │
│  │       └── hive-site.xml           ← HMS configuration file   │
│  └── trino/                                                     │
│      ├── etc/                                                   │
│      │   ├── catalog/                                           │
│      │   │   └── hive.properties     ← Trino Hive Catalog configuration │
│      │   ├── jvm.config              ← JVM configuration        │
│      │   └── node.properties         ← Node configuration       │
│      │   └── hadoop/                                            │
│      │       └── conf/                                          │
│      │           └── core-site.xml   ← Hadoop core configuration│
│      └── plugin/                                                │
│          └── hive/                                              │
│              └── hdfs/                                          │
│                  └── hadoop-oss-*.jar ← OSS V2 JAR file         │
└─────────────────────────────────────────────────────────────────┘
         │                          │
         │ Docker Volume mount      │ Docker Volume mount
         ▼                          ▼
┌──────────────────┐      ┌──────────────────┐      ┌────────────────┐
│  MySQL container │      │  HMS container   │      │ Trino container│
│  mysql:5.7.35    │      │  apache/hive:4.0 │      │ trinodb/trino: │
│                  │      │                  │      │     480        │
│  /var/lib/mysql  │      │ /opt/hive/conf/  │      │ /etc/trino/    │
│  /docker-entry.. │      │                  │      │ /usr/lib/trino │
└──────────────────┘      └──────────────────┘      └────────────────┘
         │                          │                      │
         │                          │                      │
         └──────────────────────────┼──────────────────────┘
                                    │
                          Docker network: trino-net
                                    │
                                    ▼
                          ┌──────────────────┐
                          │  Alibaba Cloud   │
                          │  OSS (s3a://)    │
                          └──────────────────┘

Data flow

1. Query initiation: A user runs a SQL query through the Trino CLI.

2. Metadata retrieval: Trino connects to HMS over the Thrift protocol (default port 9083) to retrieve the table schema and data location.

3. Data reading: Trino calls the OSS V2 Connector based on the location (such as s3a://bucket/path).

4. OSS access: The OSS V2 Connector reads credentials and endpoint from core-site.xml and sends HTTPS requests to OSS.

5. Data return: OSS returns a data stream. Trino parses it and returns the query result.

Hardware requirements

Software requirements

Verify the Docker environment

Run these commands on the host to verify Docker is installed and running:

# Check the Docker version
docker --version
# Expected output: Docker version 20.10.x or later

# Check the Docker Compose version
docker compose version
# Expected output: Docker Compose version v2.x.x

# Check if Docker is running correctly
docker ps
# Expected output: A list of containers (might be empty)

The first two commands return Docker and Docker Compose version numbers. docker ps returns a container list (empty after initial installation).

1.1. Create working directories

Create the working directory

# Create the root directory
mkdir -p /opt/trino-test

# Enter the working directory
cd /opt/trino-test

# Verify that the directory was created
ls -la /opt/trino-test
# Expected output: An empty directory

Create subdirectories

# Create MySQL-related directories
mkdir -p /opt/trino-test/mysql

# Create HMS-related directories
mkdir -p /opt/trino-test/hms/conf

# Create Trino-related directories
mkdir -p /opt/trino-test/trino/etc/catalog
mkdir -p /opt/trino-test/trino/etc/hadoop/conf
mkdir -p /opt/trino-test/trino/plugin/hive/hdfs

# Verify the directory structure
tree /opt/trino-test
# Or use the following command if tree is not installed
find /opt/trino-test -type d

Expected output:

/opt/trino-test/
├── docker-compose.yml (to be created)
├── mysql/
├── hms/
│   └── conf/
└── trino/
    ├── etc/
    │   ├── catalog/
    │   └── hadoop/
    │       └── conf/
    └── plugin/
        └── hive/
            └── hdfs/

1.2. Download the OSS V2 JAR file

Place the OSS V2 Connector JAR in the host plugin directory. On startup, it is mounted to /usr/lib/trino/plugin/hive/hdfs/ in the Trino container.

Download the JAR from the download page. Use the shade version hadoop-oss-3.3.5-2.0.25-alpha-shade.jar, which bundles all core dependencies.

Copy the JAR file to the plugin directory:

# Copy an existing local JAR file
cp /path/to/hadoop-oss-3.3.5-2.0.26-alpha-shade.jar \
  /opt/trino-test/trino/plugin/hive/hdfs/

# Verification:
ls -lh /opt/trino-test/trino/plugin/hive/hdfs/hadoop-oss-3.3.5-2.0.26-alpha-shade.jar
# Expected output: File size is about 12 MB

1.3. Create the MySQL initialization script

This script creates the HMS metastore database and access account on MySQL startup.

File path: /opt/trino-test/mysql/init.sql

# Use the cat command to create the file
cat > /opt/trino-test/mysql/init.sql << 'EOF'
CREATE DATABASE metastore CHARACTER SET 'utf8' COLLATE 'utf8_bin';
CREATE USER 'hive'@'%' IDENTIFIED BY 'hive_password';
GRANT ALL PRIVILEGES ON metastore.* TO 'hive'@'%';
FLUSH PRIVILEGES;
EOF

# Verify that the file was created
cat /opt/trino-test/mysql/init.sql

Expected output:

CREATE DATABASE metastore CHARACTER SET 'utf8' COLLATE 'utf8_bin';
CREATE USER 'hive'@'%' IDENTIFIED BY 'hive_password';
GRANT ALL PRIVILEGES ON metastore.* TO 'hive'@'%';
FLUSH PRIVILEGES;

1.4. Create the HMS configuration file

Configure the HMS connection to MySQL and the Thrift service address.

File path: /opt/trino-test/hms/conf/hive-site.xml

cat > /opt/trino-test/hms/conf/hive-site.xml << 'EOF'
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <!-- MySQL connection configuration -->
    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://mysql:3306/metastore?createDatabaseIfNotExist=true</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>hive</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>hive_password</value>
    </property>

    <!-- HMS Thrift service configuration -->
    <property>
        <name>hive.metastore.uris</name>
        <value>thrift://hms:9083</value>
    </property>
    <property>
        <name>hive.metastore.schema.verification</name>
        <value>false</value>
    </property>
</configuration>
EOF
# Verify that the file was created
cat /opt/trino-test/hms/conf/hive-site.xml

1.5. Create the Trino Hive Catalog configuration

Enable Hadoop file system support so Trino can process s3a:// paths through the OSS V2 Connector.

File path: /opt/trino-test/trino/etc/catalog/hive.properties

cat > /opt/trino-test/trino/etc/catalog/hive.properties << 'EOF'
connector.name=hive
hive.metastore.uri=thrift://hms:9083

# Enable Hadoop file system support (critical configuration)
fs.hadoop.enabled=true

# Use the default Hadoop file system (reads core-site.xml)
hive.s3-file-system-type=HADOOP_DEFAULT

# Specify the Hadoop configuration file path (path within the container)
hive.config.resources=/etc/hadoop/conf/core-site.xml
EOF

# Verify that the file was created
cat /opt/trino-test/trino/etc/catalog/hive.properties

Configuration details:

• fs.hadoop.enabled=true: Enables Hadoop file system support. Trino loads JARs from the hdfs/ directory.

• hive.s3-file-system-type=HADOOP_DEFAULT: Uses the Hadoop FileSystem implementation (the OSS V2 Connector).

• hive.config.resources: Path to core-site.xml inside the container.

1.6. Create the Trino JVM configuration

File path: /opt/trino-test/trino/etc/jvm.config

cat > /opt/trino-test/trino/etc/jvm.config << 'EOF'
-server
-Xmx4G
-XX:+UseG1GC
-XX:G1HeapRegionSize=32M
-XX:+ExitOnOutOfMemoryError
-XX:+HeapDumpOnOutOfMemoryError
EOF

# Verify that the file was created
cat /opt/trino-test/trino/etc/jvm.config

1.7. Create the Trino node configuration

File path: /opt/trino-test/trino/etc/node.properties

cat > /opt/trino-test/trino/etc/node.properties << 'EOF'
node.environment=test
node.data-dir=/data/trino
node.id=trino-test-001
EOF

# Verify that the file was created
cat /opt/trino-test/trino/etc/node.properties

1.8. Create the Hadoop core-site.xml

Specifies the OSS V2 Connector implementation class and access credentials. The path must match the hive.config.resources value.

File path: /opt/trino-test/trino/etc/hadoop/conf/core-site.xml

cat > /opt/trino-test/trino/etc/hadoop/conf/core-site.xml << 'EOF'
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <!-- Use Alibaba Cloud OSS Connector V2 to implement the s3a:// protocol -->
    <property>
        <name>fs.s3a.impl</name>
        <value>org.apache.hadoop.fs.aliyun.oss.v2.AliyunOSSPerformanceFileSystem</value>
    </property>
    <property>
        <name>fs.AbstractFileSystem.s3a.impl</name>
        <value>org.apache.hadoop.fs.aliyun.oss.v2.OSSWithS3A</value>
    </property>

    <!-- OSS authentication configuration -->
    <property>
        <name>fs.oss.accessKeyId</name>
        <value>YOUR_ACCESS_KEY_ID</value>
    </property>
    <property>
        <name>fs.oss.accessKeySecret</name>
        <value>YOUR_ACCESS_KEY_SECRET</value>
    </property>

    <!-- OSS endpoint (internal address recommended for ECS in the same region) -->
    <property>
        <name>fs.oss.endpoint</name>
        <value>oss-cn-hangzhou-internal.aliyuncs.com</value>
    </property>

    <!-- OSS Bucket region (required for V4 signature) -->
    <property>
        <name>fs.oss.region</name>
        <value>cn-hangzhou</value>
    </property>
</configuration>
EOF

# Verify that the file was created
cat /opt/trino-test/trino/etc/hadoop/conf/core-site.xml

Key configuration parameters:

1.9. Create the docker-compose.yml file

File path: /opt/trino-test/docker-compose.yml

cat > /opt/trino-test/docker-compose.yml << 'EOF'
version: '3.8'

services:
  # MySQL service: Stores HMS metadata
  mysql:
    image: mysql:5.7.35
    container_name: trino-test-mysql
    environment:
      MYSQL_ROOT_PASSWORD: root_password
    ports:
      - "3306:3306"
    volumes:
      - ./mysql/init.sql:/docker-entrypoint-initdb.d/init.sql
      - mysql_data:/var/lib/mysql
    networks:
      - trino-net
    restart: unless-stopped

  # HMS service: Hive Metastore
  hms:
    image: apache/hive:4.0.0
    container_name: trino-test-hms
    environment:
      SERVICE_NAME: metastore
    ports:
      - "9083:9083"
    volumes:
      - ./hms/conf/hive-site.xml:/opt/hive/conf/hive-site.xml
    depends_on:
      - mysql
    networks:
      - trino-net
    restart: unless-stopped

  # Trino service: Query engine
  trino:
    image: trinodb/trino:480
    container_name: trino-test-trino
    ports:
      - "8080:8080"
    volumes:
      # Mount Trino configuration files
      - ./trino/etc:/etc/trino
      # Mount plugin directory (contains OSS V2 JAR)
      - ./trino/plugin:/usr/lib/trino/plugin
    depends_on:
      - hms
    networks:
      - trino-net
    restart: unless-stopped

networks:
  trino-net:
    driver: bridge

volumes:
  mysql_data:
EOF

# Verify that the file was created
cat /opt/trino-test/docker-compose.yml

Docker Volume mount relationships:

1.10. Verify the directory structure

# View the complete directory structure
find /opt/trino-test -type f | sort

The command should return the following 8 files:

/opt/trino-test/docker-compose.yml
/opt/trino-test/hms/conf/hive-site.xml
/opt/trino-test/mysql/init.sql
/opt/trino-test/trino/etc/catalog/hive.properties
/opt/trino-test/trino/etc/hadoop/conf/core-site.xml
/opt/trino-test/trino/etc/jvm.config
/opt/trino-test/trino/etc/node.properties
/opt/trino-test/trino/plugin/hive/hdfs/hadoop-oss-3.3.5-2.0.26-alpha-shade.jar

Also, confirm the following:

• The JAR file size is about 12 MB.

• In core-site.xml, the AK/SK, Endpoint, and Region are replaced with actual values.

2.1. Start all containers

cd /opt/trino-test
docker compose up -d

# Expected output:
# ✔ Network trino-test_trino-net  Created
# ✔ Container trino-test-mysql    Started
# ✔ Container trino-test-hms      Started
# ✔ Container trino-test-trino    Started

The output shows Network ... Created followed by Started for each container.

2.2. Check container status

# Check the status of all containers
docker compose ps

# Expected output:
# NAME                 IMAGE              STATUS          PORTS
# trino-test-hms       apache/hive:4.0.0  Up (healthy)    0.0.0.0:9083->9083/tcp
# trino-test-mysql     mysql:5.7.35       Up (healthy)    0.0.0.0:3306->3306/tcp
# trino-test-trino     trinodb/trino:480  Up (healthy)    0.0.0.0:8080->8080/tcp

All three containers (trino-test-mysql, trino-test-hms, trino-test-trino) should show Up status. If any shows Exited, run docker compose logs <container_name> to check the cause (Troubleshooting FAQ).

2.3. Verify Trino service readiness

Trino takes 30-60 seconds to start. Verify readiness through the HTTP API:

# Wait for Trino to start completely (about 30-60 seconds)
sleep 45

# Verify that Trino has started successfully
curl -s http://localhost:8080/v1/info | head -20
# Expected output: A JSON response that contains "running": true

The response should contain "starting": false and the Trino nodes are registered successfully.

2.4. Check component logs (optional)

# Check MySQL logs
docker compose logs mysql

# Check HMS logs
docker compose logs hms

# Check Trino logs
docker compose logs trino

# Follow Trino logs in real-time
docker compose logs -f trino
# Press Ctrl+C to exit

2.5. Verify MySQL, HMS, and the Web UI

Verify MySQL: Check that the metastore database is initialized.

# Enter the MySQL container
docker exec -it trino-test-mysql mysql -uroot -proot_password

# Run SQL to verify
mysql> SHOW DATABASES;
# Expected output: Contains the metastore database

mysql> USE metastore;
mysql> SHOW TABLES;
# Expected output: A list of HMS metadata tables (if HMS is initialized)

mysql> EXIT;

The output should contain metastore.

Verify HMS: Check if port 9083 is listening.

# Check if the HMS port is listening
docker exec trino-test-hms netstat -tlnp | grep 9083
# Expected output: Port 9083 is in a LISTEN state

# Or test from the host machine using telnet
telnet localhost 9083
# Expected output: Connected to localhost
# Press Ctrl+] and then type quit to exit

Connection to localhost 9083 port [tcp/*] succeeded! confirms HMS is ready.

Verify the Trino Web UI: Open http://<host machine IP>:8080 in a browser to view cluster status and query history.

3.1. Enter the Trino CLI

# Method A: Interactive session
docker exec -it trino-test-trino trino

# Method B: Single SQL command (suitable for scripting)
docker exec trino-test-trino trino --execute "SHOW CATALOGS;"

3.2. Verify the catalog

-- View the list of available catalogs
SHOW CATALOGS;

-- Expected output:
--  Catalog
-- ---------
--  hive
--  jmx
--  memory
--  system
--  tpcds
--  tpch

The output should contain hive. If missing, run docker exec trino-test-trino cat /etc/trino/catalog/hive.properties to verify the mount, and docker compose logs trino | grep -i error to check for errors.

3.3. Create a schema

Replace your-bucket-name with the actual OSS Bucket name.

-- Create a schema with the location using the s3a:// protocol
CREATE SCHEMA hive.oss_test
WITH (location = 's3a://your-bucket-name/test/');

-- Verification
SHOW SCHEMAS FROM hive;
-- Expected output: Includes oss_test
SHOW CREATE SCHEMA hive.oss_test;
-- Expected output: Includes location = 's3a://your-bucket-name/test/'

A successful schema creation with a queryable location confirms Trino-to-OSS connectivity through the OSS V2 Connector.

3.4. Create, populate, and query an external table

Create an external table, upload test data to OSS, and query it.

Create a CSV-formatted external table:

CREATE TABLE hive.oss_test.sample_csv (
  col1 varchar,
  col2 varchar,
  col3 varchar
)
WITH (
  format = 'CSV',
  csv_separator = ',',
  external_location = 's3a://your-bucket-name/test/sample_csv/'
);

DESCRIBE hive.oss_test.sample_csv;

On the host machine, create test data and upload it to OSS:

# Create a local test file
cat > /tmp/test_data.csv << 'EOF'
1,alice,100
2,bob,200
3,charlie,300
EOF

# Use ossutil to upload the CSV test data to OSS (replace with your bucket name)
ossutil cp /tmp/test_data.csv \
  oss://your-bucket-name/test/sample_csv/test_data.csv

# Verify that the upload was successful
ossutil ls oss://your-bucket-name/test/sample_csv/

Query the data:

-- Count the number of rows
SELECT COUNT(*) FROM hive.oss_test.sample_csv;
-- Expected output: 3

-- Query all data
SELECT * FROM hive.oss_test.sample_csv;
-- Expected output:
--  col1 | col2    | col3
-- ------+---------+------
--  1    | alice   | 100
--  2    | bob     | 200
--  3    | charlie | 300

The query returns 3 and three rows of data.

3.5. Create a Parquet table (optional)

Parquet is recommended for production. This step uses INSERT to verify Trino can write to OSS through the OSS V2 Connector.

-- Create a Parquet format table (recommended for production)
CREATE TABLE hive.oss_test.sample_parquet (
  id bigint,
  name varchar,
  score double
)
WITH (
  format = 'PARQUET',
  external_location = 's3a://your-bucket-name/test/sample_parquet/'
);

-- Insert test data
INSERT INTO hive.oss_test.sample_parquet VALUES 
  (1, 'alice', 95.5),
  (2, 'bob', 87.3),
  (3, 'charlie', 92.1);

-- Query the data
SELECT * FROM hive.oss_test.sample_parquet;

Verify the Parquet file in OSS:

# Check the files generated in OSS
ossutil ls oss://your-bucket-name/test/

# Expected output:
# oss://your-bucket-name/test/sample_csv/test_data.csv
# oss://your-bucket-name/test/sample_parquet/xxxxx.parquet

A container fails to start

Symptom: After you run docker compose up -d, a container's status changes to Exited.

Troubleshooting: Run docker compose ps to identify the failed container, then docker compose logs <container_name> to view the error. Common causes:

Trino cannot connect to HMS

Symptom: Running SHOW SCHEMAS FROM hive returns the Failed connecting to Hive metastore error.

How to troubleshoot:

# 1. Check if the HMS container is running correctly
docker compose ps hms

# 2. Check if the HMS port is listening
docker exec trino-test-hms netstat -tlnp | grep 9083

# 3. Check network connectivity from Trino to HMS
docker exec trino-test-trino nc -zv hms 9083
# Expected output: Connection to hms 9083 port [tcp] succeeded!

# 4. Check the hive.properties configuration
docker exec trino-test-trino cat /etc/trino/catalog/hive.properties
# Confirm that hive.metastore.uri=thrift://hms:9083

Error: NoClassDefFoundError

Symptom: A query fails with an error, such as java.lang.NoClassDefFoundError: org/apache/commons/collections/CollectionUtils.

Cause: The JAR is missing from the expected directory or not mounted into the container.

How to troubleshoot:

# 1. Check if the JAR file exists on the host machine
ls -lh /opt/trino-test/trino/plugin/hive/hdfs/hadoop-oss-*.jar

# 2. Check if the JAR file is visible inside the container
docker exec trino-test-trino ls -lh /usr/lib/trino/plugin/hive/hdfs/hadoop-oss-*.jar

# 3. If the file is missing in the container, confirm the path is correct and restart Trino
docker compose restart trino

Error: Invalid signing region

Symptom: A query returns the error Invalid signing region in Authorization header.

Cause: The fs.oss.region value does not match the bucket's actual region (enforced by V4 signature verification).

Solution: Check and correct fs.oss.region in core-site.xml, then run docker compose restart trino.

# 1. Check the region configuration in core-site.xml
docker exec trino-test-trino cat /etc/hadoop/conf/core-site.xml | grep region

# 2. Confirm that the region value matches the bucket's actual region
# For example, if the bucket is in Hangzhou, the region should be cn-hangzhou

# 3. After modifying, restart Trino
docker compose restart trino

Error: Access Denied / 403 Forbidden

Symptom: A query returns an Access Denied or 403 Forbidden error.

Possible causes and troubleshooting:

• Incorrect AK/SK: Run docker exec trino-test-trino cat /etc/hadoop/conf/core-site.xml | grep -A1 accessKey to check.

• The AccessKey account lacks permission for the target bucket: Grant permissions in the RAM console or adjust the bucket policy in the OSS console.

• Endpoint-region mismatch: Verify fs.oss.endpoint.

# 1. Verify that the AccessKey ID and Secret are correct
docker exec trino-test-trino cat /etc/hadoop/conf/core-site.xml | grep -A1 accessKey

# 2. Verify that the Bucket Policy allows access
# Log in to the Alibaba Cloud console -> OSS -> Bucket -> Permissions -> Bucket Policy

# 3. Verify that the endpoint is correct
docker exec trino-test-trino cat /etc/hadoop/conf/core-site.xml | grep endpoint

HMS metastore is not initialized

Symptom: The HMS log reports the error Table 'metastore.VERSION' doesn't exist.

Cause: HMS connected before MySQL finished initializing, so metadata tables were not created.

Solution:

# 1. Check if the MySQL initialization script has been executed
docker exec trino-test-mysql mysql -uroot -proot_password -e "SHOW DATABASES;"

# 2. If the metastore database does not exist, re-initialize:
docker compose down -v  # Remove all data and containers
docker compose up -d    # Restart

# 3. Wait for MySQL to fully start before starting HMS
docker compose up -d mysql
sleep 10
docker compose up -d hms trino

JAR file information

Key ports

Common command reference

# Start services
cd /opt/trino-test && docker compose up -d

# Stop services
docker compose down

# View real-time logs
docker compose logs -f trino

# Enter the Trino CLI
docker exec -it trino-test-trino trino

# Execute a single SQL command
docker exec trino-test-trino trino --execute "SHOW CATALOGS;"

# Restart a single service
docker compose restart trino

# Check container status
docker compose ps

References

• Trino official documentation

• Trino Hive Connector documentation

• Hadoop OSS Connector V2