Learn Apache Flink - Security, Authentication & Authorization

#Introduction

Up to episode 11, all your jobs ran in a trusted environment. Episode 12 changes that assumption: once Flink enters production, the cluster communicates over networks that can be snooped, and the dashboard can be accessed by anyone if it's not protected. Security isn't an add-on feature — it's a prerequisite.

We'll cover four layers: securing communication between components with TLS, authentication with Kerberos and RBAC, locking down access to the web dashboard and REST API, and protecting the credentials used by sources and sinks. After this episode, you can configure a production-worthy Flink cluster.

#Secure Cluster Communication with TLS

#Certificates for Encryption

Flink encrypts two paths: internal (between JobManager and TaskManager) and REST (dashboard and API). Both use JKS keystores and truststores:

keytool -genkeypair -alias flink -keyalg RSA \
  -keystore keystore.jks -storepass changeit -validity 365

keytool is the JDK's standard tool for creating certificates. In production, certificates are usually issued by an internal CA, not self-signed as in the example above.

#TLS Configuration in config.yaml

security.ssl.enabled: true
security.ssl.rest.enabled: true
security.ssl.internal.enabled: true
security.ssl.keystore: /secrets/keystore.jks
security.ssl.keystore-password: ${KEYSTORE_PASSWORD}
security.ssl.truststore: /secrets/truststore.jks
security.ssl.truststore-password: ${TRUSTSTORE_PASSWORD}

security.ssl.enabled turns on encryption, while rest and internal choose which paths are encrypted. Store passwords in a secret manager and inject them via environment variables, not hardcoded in the file.

#Authentication with Kerberos

#Login Contexts and Keytabs

In enterprise environments (especially Hadoop), Flink authenticates via Kerberos. The login component is configured per context:

security.kerberos.login.use-ticket-cache: true
security.kerberos.login.contexts: Client,KafkaClient
security.kerberos.login.keytab: /secrets/flink.keytab
security.kerberos.login.principal: flink@EXAMPLE.COM

security.kerberos.login.keytab points to the keytab file and principal sets the service identity. The Client context is used for Hadoop, KafkaClient for Kafka with SASL Kerberos.

#Authorization and RBAC

#Limiting Who Can Do What

Authentication answers "who are you"; authorization answers "what are you allowed to do". Flink enforces access control at several points: the REST API for submitting or cancelling jobs, and external systems like HDFS or Kafka that validate user permissions.

rest.bind-address: 0.0.0.0
rest.bind-port: 8081
jobmanager.rpc.address: flink-jm.internal

Binding the REST API to an internal host and not exposing it to the internet is the simplest form of network authorization. For fine-grained control, combine it with a proxy that checks user identity before forwarding requests.

#Securing the Web Dashboard and REST API

#A Dashboard for Internal Use Only

The web dashboard shows sensitive information: job graphs, metrics, and configuration. Best practices:

• Only accessible from a VPN or the internal network.
• Encrypted with TLS (already configured above).
• Never expose port 8081 directly to the internet.

curl -sk https://localhost:8081/overview

curl -sk calls the REST API with TLS but skips certificate verification — debugging only. In production, remove the -k flag and use a valid certificate.

#Securing Source and Sink Credentials

#Don't Hardcode Secrets

Database, Kafka, and S3 credentials are a primary target for attackers. Some mandatory rules:

• Use a secret manager (Kubernetes Secrets, Vault) and inject via environment variables.
• In Flink SQL, read passwords from the environment with placeholders.
• Never copy credentials into git or Docker images.

CREATE TABLE orders (
  user_id STRING,
  amount  BIGINT
) WITH (
  'connector' = 'kafka',
  'properties.security.protocol' = 'SASL_SSL',
  'properties.sasl.mechanism' = 'PLAIN',
  'properties.sasl.jaas.config' =
    'org.apache.kafka.common.security.plain.PlainLoginModule required username="flink" password="${KAFKA_PASSWORD}";'
);

${KAFKA_PASSWORD} is an environment placeholder — the secret value stays outside the file. The same pattern applies to JDBC, S3, and all other connectors.

#Conclusion

Episode 12 secured the Flink cluster from four sides: TLS for encrypting internal and REST paths, Kerberos for authentication in enterprise environments, access restrictions for the dashboard and API, and secure credential storage practices for all sources and sinks.

The key takeaways:

• TLS encrypts the internal and REST paths; enable both in production.
• Kerberos uses keytabs and principals; login contexts are tailored per service.
• The dashboard and REST API are only for internal networks, always via TLS.
• Credentials are injected via environment variables, never hardcoded.
• The same pattern applies to Kafka, JDBC, and all connectors.

In the next episode, episode 13, we'll discuss data governance & observability — exposing metrics to Prometheus and Grafana, monitoring logging, job metrics, and backpressure, tracking audit events, lineage, and data quality, and tracing for latency analysis. You'll make your streaming system not just run, but be visible.