In this tutorial I will show you how to use Kerberos/SSL with NiFi. I will use self signed certs for this example. Before you begin ensure you have installed Kerberos Server and NiFi.
This assumes your hostname is “hadoop”
Create Kerberos Principals
cd /etc/security/keytabs/ sudo kadmin.local #You can list principals listprincs #Create the following principals addprinc -randkey nifi/hadoop@REALM.CA addprinc -randkey nifi-spnego/hadoop@REALM.CA #Notice this user does not have -randkey because we are a login user #Also notice that this user does not have a keytab created addprinc admin/hadoop@REALM.CA #Create the keytab files. #You will need these for Hadoop to be able to login xst -k nifi.service.keytab nifi/hadoop@REALM.CA xst -k nifi-spnego.service.keytab nifi-spnego/hadoop@REALM.CA
Set Keytab Permissions/Ownership
sudo chown root:hadoopuser /etc/security/keytabs/* sudo chmod 750 /etc/security/keytabs/*
Stop NiFi
sudo service nifi stop
Hosts Update
sudo nano /etc/hosts #Remove 127.0.1.1 line #Change 127.0.0.1 to the following 127.0.0.1 gaudreault_kdc.ca hadoop localhost
Ubuntu Firewall
sudo ufw disable
sysctl.conf
Disable ipv6 as it causes issues in getting your server up and running.
nano /etc/sysctl.conf
Add the following to the end and save
net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1 #Change eth0 to what ifconfig has net.ipv6.conf.eth0.disable_ipv6 = 1
Close sysctl
sysctl -p cat /proc/sys/net/ipv6/conf/all/disable_ipv6 reboot
TrustStore / KeyStore
#Creating your Certificate Authority sudo mkdir -p /etc/security/serverKeys sudo chown -R root:hduser /etc/security/serverKeys/ sudo chmod 750 /etc/security/serverKeys/ cd /etc/security/serverKeys sudo openssl genrsa -aes128 -out nifi.key 4096 sudo openssl req -x509 -new -key nifi.key -days 1095 -out nifi.pem sudo openssl rsa -check -in nifi.key #check it sudo openssl x509 -outform der -in nifi.pem -out nifi.der sudo keytool -import -keystore truststore.jks -file nifi.der -alias nifi #***You must type 'yes' to trust this certificate. sudo keytool -v -list -keystore truststore.jks #Creating your Server Keystore sudo keytool -genkey -alias nifi -keyalg RSA -keystore keystore.jks -keysize 2048 sudo keytool -certreq -alias nifi -keystore keystore.jks -file nifi.csr sudo openssl x509 -sha256 -req -in nifi.csr -CA nifi.pem -CAkey nifi.key -CAcreateserial -out nifi.crt -days 730 sudo keytool -import -keystore keystore.jks -file nifi.pem sudo keytool -import -trustcacerts -alias nifi -file nifi.crt -keystore keystore.jks sudo chown -R root:hduser /etc/security/serverKeys/* sudo chmod 750 /etc/security/serverKeys/*
nifi.properties
cd /usr/local/nifi/conf/ nano nifi.properties #Find "# Site to Site properties" and change the following properties to what is below nifi.remote.input.host= nifi.remote.input.secure=true nifi.remote.input.socket.port=9096 nifi.remote.input.http.enabled=false #Find "# web properties #" and change the following properties to what is below nifi.web.http.host= nifi.web.http.port= nifi.web.https.host=0.0.0.0 nifi.web.https.port=9095 #Find "# security properties #" and change the following properties to what is below nifi.security.keystore=/etc/security/serverKeys/keystore.jks nifi.security.keystoreType=JKS nifi.security.keystorePasswd=PASSWORD nifi.security.keyPasswd=PASSWORD nifi.security.truststore=/etc/security/serverKeys/truststore.jks nifi.security.truststoreType=JKS nifi.security.truststorePasswd=PASSWORD nifi.security.needClientAuth=true nifi.security.user.authorizer=managed-authorizer nifi.security.user.login.identity.provider=kerberos-provider #Find "# Core Properties #" and change the following properties to what is below nifi.authorizer.configuration.file=./conf/authorizers.xml nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml #Find "# kerberos #" and change the following properties to what is below nifi.kerberos.krb5.file=/etc/krb5.conf #Find "# kerberos service principal #" and change the following properties to what is below nifi.kerberos.service.principal=nifi/hadoop@REALM.CA nifi.kerberos.service.keytab.location=/etc/security/keytabs/nifi.service.keytab #Find "# kerberos spnego principal #" and change the following properties to what is below nifi.kerberos.spnego.principal=nifi-spnego/hadoop@REALM.CA nifi.kerberos.spnego.keytab.location=/etc/security/keytabs/nifi-spnego.service.keytab nifi.kerberos.spnego.authentication.expiration=12 hours #Find "# cluster common properties (all nodes must have same values) #" and change the following properties to what is below nifi.cluster.protocol.is.secure=true
login-identity-providers.xml
nano login-identity-providers.xml #Find "kerberos-provider" <provider> <identifier>kerberos-provider</identifier> <class>org.apache.nifi.kerberos.KerberosProvider</class> <property name="Default Realm">REALM.CA</property> <property name="Kerberos Config File">/etc/krb5.conf</property> <property name="Authentication Expiration">12 hours</property> </provider>
authorizers.xml
nano authorizers.xml #Find "file-provider" <authorizer> <identifier>file-provider</identifier> <class>org.apache.nifi.authorization.FileAuthorizer</class> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Users File">./conf/users.xml</property> <property name="Initial Admin Identity">admin/hadoop@REALM.CA</property> <property name="Legacy Authorized Users File"></property> <property name="Node Identity 1"></property> </authorizer>
Start Nifi
sudo service nifi start
Issues:
- If you get the error “No applicable policies could be found” after logging in and no GUI is shown stop the NiFi service and restart. Then you should be good.
- If you can then login but you don’t have any policies still you will need to update “authorizations.xml” and add the below lines. Making sure to change the resource process group id to the root process group id and the user id to the user id
nano /usr/local/nifi/conf/authorizations.xml <policy identifier="1c897e9d-3dd5-34ca-ae3d-75fb5ee3e1a5" resource="/data/process-groups/##CHANGE TO ROOT ID##" action="R"> <user identifier="##CHANGE TO USER ID##"/> </policy> <policy identifier="91c64c2d-7848-371d-9d5f-db71138b152f" resource="/data/process-groups/##CHANGE TO ROOT ID##" action="W"> <user identifier="##CHANGE TO USER ID##"/> </policy> <policy identifier="7aeb4d67-e2e1-3a3e-a8fa-94576f35539e" resource="/process-groups/##CHANGE TO ROOT ID##" action="R"> <user identifier="##CHANGE TO USER ID##"/> </policy> <policy identifier="f5b620e0-b094-3f70-9542-dd6920ad5bd9" resource="/process-groups/##CHANGE TO ROOT ID##" action="W"> <user identifier="##CHANGE TO USER ID##"/> </policy>