Kafka Kerberos authentication-Realtime Compute for Apache Flink(Flink)-阿里云帮助中心

To connect a Flink job to a Kafka cluster with Kerberos authentication enabled, configure the Kerberos and Kafka settings in the Flink development console before starting the job. This topic covers the GSSAPI/Kerberos mechanism and applies to both SQL jobs and DataStream jobs on Realtime Compute for Apache Flink.

Limitations

ApsaraMQ for Kafka does not support Kerberos authentication.

Prerequisites

Before you begin, ensure that you have:

The krb5.conf file for your Kerberos environment. This file defines the Kerberos cluster and connection settings. For details on configuration items, see the Kerberos documentation.
The keytab file containing the user credentials for identity verification with the Kerberos server.

Step 1: Configure domain name resolution

Kerberos authentication relies on domain name resolution. Configure the domain names and IP addresses for both the Key Distribution Center (KDC) and Kafka services in the Flink development console. For instructions, see Manage domain names.

Component	Source of domain name and IP address
KDC server	`krb5.conf`
Kafka brokers	`server.properties`

Important

The Kafka client must connect using the domain name registered for each broker in Kerberos. If the domain name does not match the broker's registered principal, authentication fails. To verify a broker's registered domain name, check its principal. For details, see Basic Kerberos usage. Configure domain name resolution for every broker in the cluster. For example, if broker1's principal is kafka/broker1.example.com@EXAMPLE.com, configure DNS resolution for broker1.example.com.

Step 2: Configure network access

Allow network connections from Flink to port 88 on the KDC server.

Step 3: Upload Kerberos configuration files

Log on to the Realtime Compute for Apache Flink console.
In the Actions column for the target workspace, click Console.
In the left navigation pane, click File Management.

Click Upload Resource and select your krb5.conf and keytab files. After uploading, if the files are named krb5.conf and keytab, they are available at the following paths:

Note

To use different Kerberos users for different Kafka clients in the same job, upload a keytab file for each user.

File	Path
`krb5.conf`	`/flink/usrlib/krb5.conf`
`keytab`	`/flink/usrlib/keytab`

Click Open to complete the upload.

Step 4: Configure Kerberos for your job

Choose the method that matches your job type.

SQL job

Part 1: Configure the Kafka table

Create or open an SQL job. For details, see Job development map.

In the WITH clause of your Kafka table definition, add the following parameters:

CREATE TEMPORARY TABLE kafka_kerberos (
    ...
) WITH (
    'connector' = 'kafka',

    -- Required: use the domain name registered for Kafka in Kerberos
    'properties.bootstrap.servers' = 'broker1.example.com:9092',

    -- Required: SASL authentication settings
    'properties.security.protocol' = 'SASL_PLAINTEXT',
    'properties.sasl.mechanism' = 'GSSAPI',
    'properties.sasl.kerberos.service.name' = 'kafka',

    -- Required: JAAS config — replace the principal with your username and realm
    'properties.sasl.jaas.config' = 'com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="/flink/usrlib/keytab" principal="user@EXAMPLE.COM";',

    -- Add other connector settings as needed
    ...
)

For the full list of Kafka connector parameters, see CONFIGURATION.

Deploy the job. For details, see Deploy a job.

Part 2: Set the JVM option

Go to Operation Center > Jobs and click the target job name.
On the Deployment Details tab, under Parameter Settings, add the following to Other Configurations:
```
env.java.opts: '-Djava.security.krb5.conf=/flink/usrlib/krb5.conf'
```
Click Save.
Click Start in the upper-right corner.

DataStream job

Part 1: Configure the Kafka source and sink

Develop a DataStream job. For details, see Develop a JAR job.

In your code, set the following properties on both KafkaSource and KafkaSink:

KafkaSource.builder()
    ...
    // Required: use the domain name registered for Kafka in Kerberos
    .setBootstrapServers("broker1.example.com:9092")

    // Required: SASL authentication settings
    .setProperty("security.protocol", "SASL_PLAINTEXT")
    .setProperty("sasl.mechanism", "GSSAPI")
    .setProperty("sasl.kerberos.service.name", "kafka")

    // Required: JAAS config — replace the principal with your username and realm
    .setProperty("sasl.jaas.config", "com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab=\"/flink/usrlib/keytab\" principal=\"user@EXAMPLE.COM\";")

KafkaSink.builder()
    ...
    // Required: use the domain name registered for Kafka in Kerberos
    .setBootstrapServers("broker1.example.com:9092")

    // Required: SASL authentication settings
    .setProperty("security.protocol", "SASL_PLAINTEXT")
    .setProperty("sasl.mechanism", "GSSAPI")
    .setProperty("sasl.kerberos.service.name", "kafka")

    // Required: JAAS config — replace the principal with your username and realm
    .setProperty("sasl.jaas.config", "com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab=\"/flink/usrlib/keytab\" principal=\"user@EXAMPLE.COM\";")

For the full list of Kafka properties, see CONFIGURATION.

Part 2: Deploy and set the JVM option

Package your DataStream project into a JAR package.
Upload and deploy the JAR package to the Flink development console. For details, see Deploy a JAR job.
On the Jobs page, click the target job. On the Deployment Details tab, under Parameter Settings, add the following to Other Configurations:
```
env.java.opts: '-Djava.security.krb5.conf=/flink/usrlib/krb5.conf'
```
Click Save.
Click Start in the upper-right corner.