Oliver

Hive: Misc

This tutorial will show you some misc usage for working with Hive. If you have no installed Hive yet please follow this tutorial.

NVL:
Check if value is null then substitute other value.

SELECT NVL(columnA, 'was null')
FROM test;

CAST:
If columnE was a date and you wanted it to be a string.

SELECT CAST(columnE AS STRING)
FROM test;

Concat:
This will concat the strings together giving “Test the code!”

SELECT CONCAT('Test', ' the ', 'code!');

CONCAT_WS:
This will concat the strings together starting at “Test” and use the first position as the seperator giving “Test the code”

SELECT CONCAT_WS(' ', 'Test', 'the', 'code');

MIN:
This will give the minimum value of columnC

SELECT MIN(columnC) AS Max_ColumnC
FROM test;

MAX:
This will give the maximum value of columnC

SELECT MAX(columnC) AS Max_ColumnC
FROM test;

DISTINCT:
This will select the distinct columnA and columnB

SELECT DISTINCT columnA, columnB
FROM test;

CASE:

SELECT CASE WHEN columnA = 'val' THEN 'Is Val' ELSE 'Not Val' END
FROM test;

COLLECT_SET:
This will collect all values in columnC and select the first index.

SELECT COLLECT_SET(columnC)[0]
FROM test;

CURRENT_DATE:
This will give you the current date in the format YYYY-MM-dd.

SELECT CURRENT_DATE();

UNIX_TIMESTAMP:
This will give you the current timestamp from EPOCH. IE: 1549591492

SELECT UNIX_TIMESTAMP();

FROM_UNIXTIME:
This will take a timestamp and display it in the format YYYY-MM-dd HH:MM:SS.

SELECT FROM_UNIXTIME(1549591492);

TO_DATE:
This will convert a date to YYYY-MM-dd.

SELECT TO_DATE('2019-02-01 01:01:01');

YEAR:

SELECT YEAR(columnE)
FROM test;

MONTH:

SELECT MONTH(columnE)
FROM test;

DAY:

SELECT DAY(columnE)
FROM test;

DATE_ADD:

SELECT DATE_ADD('2019-01-01', 4);

UPPER:
This will upper case columnA.

SELECT UPPER(columnA)
FROM test;

LOWER:
This will lower case columnA.

SELECT LOWER(columnA)
FROM test;

TRIM:
This will trim leading and trailing spaces in columnA.

SELECT TRIM(columnA)
FROM test;

LITERALS:
If a column contains a space you will need to use literal in order to use the AS keyword or when you are defining it in the create table command. Although I don’t recommend this it is possible.

SELECT columnA AS `test column`
FROM test;

Hive: Variables

This tutorial will show you some common usage for working with Hive variables. If you have no installed Hive yet please follow this tutorial.

Create HiveVar:

SET hivevar:MY_DATE='2019-02-01';

Show HiveVar:
This will show you the value of the variable.

SET MY_DATE;

Use HiveVar:

SELECT ${MY_DATE};

Hive: Views

This tutorial will show you some common usage for creating views. If you have no installed Hive yet please follow this tutorial.

Create View:

CREATE VIEW IF NOT EXISTS test_view AS
SELECT * FROM test;

Drop View:

DROP VIEW IF EXISTS test_view;

Hive: Tables

This tutorial will show you some common usage for working with tables. If you have no installed Hive yet please follow this tutorial.

Show Tables:

SHOW TABLES;
SHOW TABLES LIKE '*test*';

Table Creation:

CREATE TABLE test (
    columnA STRING,
    columnB VARCHAR(15),
    columnC INT,
    columnD TIMESTAMP,
    columnE DATE
)
STORED AS ORC;

Table Creation with Partitioning:

CREATE TABLE test_partition (
    columnA STRING,
    columnB VARCHAR(15),
    columnC INT,
    columnD TIMESTAMP,
    columnE DATE
)
PARTITIONED BY (columnF INT)
STORED AS ORC;

Inline Table Creation:

CREATE TABLE test_inline STORED AS ORC AS
SELECT *
FROM test;

Temporary Table Creation:

CREATE TEMPORARY TABLE temp (
    columnA STRING,
    columnB VARCHAR(15),
    columnC INT,
    columnD TIMESTAMP,
    columnE DATE
)
STORED AS ORC;

DESC Table:
This will show you the basic definition of a table.

DESC test;

DESC EXTENDED Table:
This will show you the extended definition of a table.

DESC EXTENDED test;

Drop Table:

DROP TABLE IF EXISTS temp;

Shell: AWK Command

In this tutorial I will show you some useful awk uses.

Determine Line Number

This will find the line number of the text_to_find.

lineNumber=`awk '/TEXT_TO_FIND/{print NR}' /file/path/name.extension`

Split Text By Space

If you want to split a word by a space and then select the second one

val='test test2'
echo $val | awk '{ print $2 }'

Split Text By Comma

If you want to split a word by a space and then select the second one

val='test test2'
echo $val | awk -F, '{ print $2 }'

Shell: SED Command

In this tutorial I will show a few useful examples of using “sed”.

-i: inline replace
-e: command to run

Add Line At End

Notice ($) and (a after $). Match last line (‘$’) and append (‘a’)

sudo sed -i -e "\$amy new line" /file/path/name.extension

Match Replace

This will replace the text on the left with the text on the right. Notice the “/” at the beginning and “/c\” in the middle.

sudo sed -i '/myCustomText=/c\myCustomText=newtext' /file/path/name.extension

Remove Matched Line

This will remove the line that matches the text. Notice the “/d” at the end and the “/” at the beginning.

sudo sed -i '/RemoveMyCustomText/d' /file/path/name.extension

Remove All Occurrences

This will remove all occurrences of “NIFI.APACHE.ORG” and replace with “REALM.CA”. Notice “/g” at end for global replace. Notice “s/” at the beginning. If means substitute. The “/” in the middle is used to denote the separation of the text to find and the text to replace.

sudo sed -i 's/NIFI.APACHE.ORG/REALM.CA/g' /file/path/name.extension

Handle ” Quotes / Handle XML / Add Tab

So the below example will go a few things. It will show you how to escape double quotes. It shows you how to add a tab and shows you how to handle xml. It also will add a line under text.

sudo sed -i -e "/<property name=\"Default Realm\">REALM.CA<\/property>/a \ \t<property name=\"Kerberos Config File\">/etc/krb5.conf<\/property>" /file/path/name.extension

Delete Lines By Number

sudo sed -i -e "$lineNumber,$maxLineNumber d" /file/path/name.extension

Kerberos: Commands

In this tutorial I will give you a few useful commands when using Kerberos. If you haven’t installed Kerberos yet go here. I will keep this updated as time goes on. Also note that the commands below have a variety of options. Please go check.

Admin

This will open Kerberos V5 administration system.

kadmin.local

Add Principal

This will add a new principal. -randkey is optional. When specified the encrypted key will be chosen at random instead of derived from a password. Be sure to change USER to whatever your user is.

addprinc -randkey USER/_HOST@REALM.CA

Create KeyTab

This will create a keytab in the directory where you generated it. You should put it in /etc/security/keytabs/ folder. You can also specify the full path (IE: /etc/security/keytabs/USER.keytab). Be sure to change USER to whatever your user is.

xst -k USER.keytab USER/_HOST@REALM.CA

Kinit

When using the -kt uses the keytab to grant a ticket

kinit -kt /etc/security/keytabs/USER.keytab USER/_HOST@REALM.CA

Klist

If you want to see what tickets have been granted. You can issue the below command.

klist

Inline Commands

You can do inline Kerberos commands without first opening kadmin.local. To do so you must specify the “-q” option then in quotes the command to issue. See below.

kadmin.local -q "addprinc -randkey USER/_HOST@REALM.CA"

NiFi: ExecuteSQL Processor

In this tutorial I will guide you through how to add a processor for querying a SQL table to NiFi.

For this tutorial you will need an AVRO schema called “dttest” and it’s contents are as follows.

{
     "type": "record",
     "namespace": "com.example",
     "name": "FullName",
     "fields": [
       { "name": "id", "type": "int" },
       { "name": "name", "type": "string" }
     ]
}

First we need to drag the processor onto the grid.

Next we need select the processor ExecuteSQLRecord.

Next we must configure the processor.

Now we must create the JsonRecordWriter service.

Now we name the JsonRecordWriter

Configure the JsonWriter

Next we create the DB Connection Service

Next we name the DB Connection Service

Configure the DB Service

Now validate all the settings are as below

Now you are all done. It will now query your table.

NiFi: Kerberized Kafka Consumer Processor

In this tutorial I will guide you through how to add a Kafka consumer to NiFi which is Kerberized.

For this tutorial you will need an AVRO schema called “person” and it’s contents are as follows.

{
     "type": "record",
     "namespace": "com.example",
     "name": "FullName",
     "fields": [
       { "name": "first_name", "type": "string" },
       { "name": "last_name", "type": "string" }
     ]
}

When ready you can publish this record to Kafka using the Kafka Producer.

{ "first_name": "John", "last_name": "Smith" }

First we need to drag the processor onto the grid.

Next we need select the Kafka Consumer.

Next we configure the processor

We will need to create 5 controller services.
First is the Kerberos Service

Next is the SSL Service

Next is the Json Record Reader

Next is the Avro Registry

Next is the Json Record Writer

Now you have finished configuring the services. Ensure your final Kafka Consumer configuration looks like this and you are ready.

Next we need to enable all the controller services

We need to start the processor to start receiving data

Now the record i gave you earlier you can now put to the queue. As you can see the data starts flowing in.

You can now view the queue to see the data.

We are done now and you can start using the consumer.

Phoenix & Java: Connecting Secure

In this tutorial I will show you how to connect to an Secure Phoenix using Java. It’s rather straight forward.

POM.xml

<dependency>
	<groupId>org.apache.phoenix</groupId>
	<artifactId>phoenix-queryserver</artifactId>
	<version>5.0.0-HBase-2.0</version>
</dependency>

Imports:

import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.sql.ResultSet;
import java.sql.Statement;

Initiate Kerberos Authentication

System.setProperty("java.security.krb5.conf", "C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\krb5.conf");
System.setProperty("java.security.krb5.realm", "REALM.CA");
System.setProperty("java.security.krb5.kdc", "REALM.CA");
System.setProperty("sun.security.krb5.debug", "true");
System.setProperty("javax.net.debug", "all");

Connect:

Now we create the connection.

Class.forName("org.apache.phoenix.jdbc.PhoenixDriver");
String url = "jdbc:phoenix:hadoop:2181:/hbase-secure:hbase/hadoop@REALM.CA:\\data\\hbase.service.keytab";
Connection connection = DriverManager.getConnection(url);
 
System.out.println("Connected");
 
Statement statement = connection.createStatement();
 
//Drop table
String deleteTableSql = "DROP TABLE IF EXISTS employee";		 
System.out.println("Deleting Table: " + deleteTableSql);
statement.executeUpdate(deleteTableSql);
System.out.println("Created Table");
 
//Create a table
String createTableSql = "CREATE TABLE employee ( eid bigint primary key, name varchar)";		 
System.out.println("Creating Table: " + createTableSql);
statement.executeUpdate(createTableSql);
System.out.println("Created Table");
 
//Insert Data
String insertTableSql = "UPSERT INTO employee VALUES(1, 'Oliver')";
System.out.println("Inserting Data: " + insertTableSql);
statement.executeUpdate(insertTableSql);
System.out.println("Inserted Data");
 
connection.commit();
 
//Select Data
String selectTablesSql = "select * from employee";
System.out.println("Show records: " + selectTablesSql);
ResultSet res = statement.executeQuery(selectTablesSql);
 
while (res.next()) {
	System.out.println(String.format("id: %s name: %s", res.getInt("eid"), res.getString("name")));
}

Phoenix: Kerberize Installation

In this tutorial I will show you how to use Kerberos with Phoenix. Before you begin ensure you have installed Kerberos Server, Hadoop, HBase and Zookeeper.

This assumes your hostname is “hadoop”

Install Phoenix

wget http://apache.forsale.plus/phoenix/apache-phoenix-5.0.0-HBase-2.0/bin/apache-phoenix-5.0.0-HBase-2.0-bin.tar.gz
tar -zxvf apache-phoenix-5.0.0-HBase-2.0-bin.tar.gz
sudo mv apache-phoenix-5.0.0-HBase-2.0-bin /usr/local/phoenix/
cd /usr/local/phoenix/

Setup .bashrc:

 sudo nano ~/.bashrc

Add the following to the end of the file.

#PHOENIX VARIABLES START
export PHOENIX_HOME=/usr/local/phoenix
export PHOENIX_CLASSPATH=$PHOENIX_HOME/*
export PATH=$PATH:$PHOENIX_HOME/bin
#PHOENIX VARIABLES END

 source ~/.bashrc

Link Files

ln -sf $HBASE_CONF_DIR/hbase-site.xml $PHOENIX_HOME/bin/hbase-site.xml
ln -sf $HADOOP_CONF_DIR/core-site.xml $PHOENIX_HOME/bin/core-site.xml
ln -sf $PHOENIX_HOME/phoenix-5.0.0-HBase-2.0-server.jar $HBASE_HOME/lib/phoenix-5.0.0-HBase-2.0-server.jar

hbase-env.sh

nano /usr/local/hbase/conf/hbase-env.sh
 
#Ensure the following env variables are set
 
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/usr/local/hadoop/etc/hadoop}
export PHOENIX_CLASSPATH=${PHOENIX_CLASSPATH:-/usr/local/phoenix}
export HBASE_CLASSPATH="$HBASE_CLASSPATH:$CLASSPATH:$HADOOP_CONF_DIR:$PHOENIX_CLASSPATH/phoenix-5.0.0-HBase-2.0-server.jar:$PHOENIX_CLASSPATH/phoenix-core-5.0.0-HBase-2.0.jar:$PHOENIX_CLASSPATH/phoenix-5.0.0-HBase-2.0-client.jar"

hbase-site.xml

nano /usr/local/hbase/conf/hbase-site.xml
 
#Add the following properties
 
<property>
	<name>phoenix.functions.allowUserDefinedFunctions</name>
	<value>true</value>
	<description>enable UDF functions</description>
</property>
<property>
	<name>hbase.regionserver.wal.codec</name>
	<value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value>
</property>
<property>
	<name>hbase.region.server.rpc.scheduler.factory.class</name>
	<value>org.apache.hadoop.hbase.ipc.PhoenixRpcSchedulerFactory</value>
	<description>Factory to create the Phoenix RPC Scheduler that uses separate queues for index and metadata updates</description>
</property>
<property>
	<name>hbase.rpc.controllerfactory.class</name>
	<value>org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory</value>
	<description>Factory to create the Phoenix RPC Scheduler that uses separate queues for index and metadata updates</description>
</property>
<property>
	<name>hbase.defaults.for.version.skip</name>
	<value>true</value>
</property>
<property>
	<name>phoenix.queryserver.http.port</name>
	<value>8765</value>
</property>
<property>
	<name>phoenix.queryserver.serialization</name>
	<value>PROTOBUF</value>
</property>
<property>
	<name>phoenix.queryserver.keytab.file</name>
	<value>/etc/security/keytabs/hbase.service.keytab</value>
</property>
<property>
	<name>phoenix.queryserver.kerberos.principal</name>
	<value>hbase/hadoop@REALM.CA</value>
</property>
<property>
	<name>hoenix.queryserver.http.keytab.file</name>
	<value>/etc/security/keytabs/hbaseHTTP.service.keytab</value>
</property>
<property>
	<name>phoenix.queryserver.http.kerberos.principal</name>
	<value>hbaseHTTP/hadoop@REALM.CA</value>
</property>
<property>
	<name>phoenix.queryserver.dns.nameserver</name>
	<value>hadoop</value>
</property>
<property>
	<name>phoenix.queryserver.dns.interface</name>
	<value>enp0s3</value>
</property>
<property>
		<name>phoenix.schema.mapSystemTablesToNamespace</name>
		<value>true</value>
</property>
<property>
		<name>phoenix.schema.isNamespaceMappingEnabled</name>
		<value>true</value>
</property>

sqlline.py

sqlline.py hadoop:2181:/hbase-secure:hbase/hadoop@GAUDREAULT_KDC.CA:/etc/security/keytabs/hbase.service.keytab

HBASE & Java: Connecting Secure

In this tutorial I will show you how to connect to an Secure HBASE using Java. It’s rather straight forward.

Import SSL Cert to Java:

Follow this tutorial to “Installing unlimited strength encryption Java libraries”

If on Windows do the following

#Import it
"C:\Program Files\Java\jdk1.8.0_171\bin\keytool" -import -file hadoop.csr -keystore "C:\Program Files\Java\jdk1.8.0_171\jre\lib\security\cacerts" -alias "hadoop"
 
#Check it
"C:\Program Files\Java\jdk1.8.0_171\bin\keytool" -list -v -keystore "C:\Program Files\Java\jdk1.8.0_171\jre\lib\security\cacerts"
 
#If you want to delete it
"C:\Program Files\Java\jdk1.8.0_171\bin\keytool" -delete -alias hadoop -keystore "C:\Program Files\Java\jdk1.8.0_171\jre\lib\security\cacerts"

POM.xml

<dependency>
	<groupId>org.apache.hbase</groupId>
	<artifactId>hbase-client</artifactId>
	<version>2.1.0</version>
</dependency>
<dependency>
	<groupId>org.apache.hbase</groupId>
	<artifactId>hbase</artifactId>
	<version>2.1.0</version>
	<type>pom</type>
</dependency>

Imports:

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.security.UserGroupInformation;

Initiate Kerberos Authentication

System.setProperty("java.security.auth.login.config", "C:\\data\\kafkaconnect\\kafka\\src\\main\\resources\\client_jaas.conf");
System.setProperty("https.protocols", "TLSv1,TLSv1.1,TLSv1.2");
System.setProperty("java.security.krb5.conf", "C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\krb5.conf");
System.setProperty("java.security.krb5.realm", "REALM.CA");
System.setProperty("java.security.krb5.kdc", "REALM.CA");
System.setProperty("sun.security.krb5.debug", "false");
System.setProperty("javax.net.debug", "false");
System.setProperty("javax.net.ssl.keyStorePassword", "changeit");
System.setProperty("javax.net.ssl.keyStore", "C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\cacerts");
System.setProperty("javax.net.ssl.trustStore", "C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\cacerts");
System.setProperty("javax.net.ssl.trustStorePassword", "changeit");
System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");

Config:

We will use the basic configuration here. You should secure the cluster and use appropriate settings for that.

// Setup the configuration object.
final Configuration config = HBaseConfiguration.create();
config.set("hbase.zookeeper.quorum", "hadoop");
config.set("hbase.zookeeper.property.clientPort", "2181");
config.set("hadoop.security.authentication", "kerberos");
config.set("hbase.security.authentication", "kerberos");
config.set("hbase.cluster.distributed", "true");
config.set("hbase.rpc.protection", "integrity");
config.set("zookeeper.znode.parent", "/hbase-secure");
config.set("hbase.master.kerberos.principal", "hbase/hadoop@REALM.CA");
config.set("hbase.regionserver.kerberos.principal", "hbase/hadoop@REALM.CA");

Connect:

Now we create the connection.

UserGroupInformation.setConfiguration(config);
UserGroupInformation.setLoginUser(UserGroupInformation.loginUserFromKeytabAndReturnUGI("hbase/hadoop@REALM.CA", "c:\\data\\hbase.service.keytab"));
 
System.out.println(UserGroupInformation.getLoginUser());
System.out.println(UserGroupInformation.getCurrentUser());
 
Connection conn = ConnectionFactory.createConnection(config);
 
//Later when we are done we will want to close the connection.
conn.close();

Hbase Admin:

Retrieve an Admin implementation to administer an HBase cluster. If you need it.

Admin admin = conn.getAdmin();
//Later when we are done we will want to close the connection.
admin.close();

HBase: Kerberize/SSL Installation

In this tutorial I will show you how to use Kerberos/SSL with HBase. I will use self signed certs for this example. Before you begin ensure you have installed Kerberos Server, Hadoop and Zookeeper.

This assumes your hostname is “hadoop”

We will install a Master, RegionServer and Rest Client

Create Kerberos Principals

cd /etc/security/keytabs/
 
sudo kadmin.local
 
#You can list princepals
listprincs
 
#Create the following principals
addprinc -randkey hbase/hadoop@REALM.CA
addprinc -randkey hbaseHTTP/hadoop@REALM.CA
 
#Create the keytab files.
#You will need these for Hadoop to be able to login
xst -k hbase.service.keytab hbase/hadoop@REALM.CA
xst -k hbaseHTTP.service.keytab hbaseHTTP/hadoop@REALM.CA

Set Keytab Permissions/Ownership

sudo chown root:hadoopuser /etc/security/keytabs/*
sudo chmod 750 /etc/security/keytabs/*

Install HBase

wget http://apache.forsale.plus/hbase/2.1.0/hbase-2.1.0-bin.tar.gz
tar -zxvf hbase-2.1.0-bin.tar.gz
sudo mv hbase-2.1.0 /usr/local/hbase/
cd /usr/local/hbase/conf/

Setup .bashrc:

 sudo nano ~/.bashrc

Add the following to the end of the file.

#HBASE VARIABLES START
export HBASE_HOME=/usr/local/hbase
export PATH=$PATH:$HBASE_HOME/bin
export HBASE_CONF_DIR=$HBASE_HOME/conf
#HBASE VARIABLES END

 source ~/.bashrc

hbase_client_jaas.conf

Client {
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=false
        useTicketCache=true;
};

hbase_server_jaas.conf

Client {
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=true
        useTicketCache=false
        keyTab="/etc/security/keytabs/hbase.service.keytab"
        principal="hbase/hadoop@REALM.CA";
};

regionservers

hadoop

hbase-env.sh

Add or modify the following settings.

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
export HBASE_CONF_DIR=${HBASE_CONF_DIR:-/usr/local/hbase/conf}
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/usr/local/hadoop/etc/hadoop}
export HBASE_CLASSPATH="$CLASSPATH:$HADOOP_CONF_DIR"
export HBASE_REGIONSERVERS=${HBASE_CONF_DIR}/regionservers
export HBASE_LOG_DIR=${HBASE_HOME}/logs
export HBASE_PID_DIR=/home/hadoopuser
export HBASE_MANAGES_ZK=false
export HBASE_OPTS="-Djava.security.auth.login.config=$HBASE_CONF_DIR/hbase_client_jaas.conf"
export HBASE_MASTER_OPTS="-Djava.security.auth.login.config=$HBASE_CONF_DIR/hbase_server_jaas.conf"
export HBASE_REGIONSERVER_OPTS="-Djava.security.auth.login.config=$HBASE_CONF_DIR/hbase_server_jaas.conf"

hbase-site.xml

<configuration>
	<property>
		<name>hbase.rootdir</name>
		<value>hdfs://hadoop:54310/hbase</value>
	</property>
	<property>
		<name>hbase.zookeeper.property.dataDir</name>
		<value>/usr/local/zookeeper/data</value>
	</property>
	<property>
		<name>hbase.cluster.distributed</name>
		<value>true</value>
	</property>
	<property>
		<name>hbase.regionserver.kerberos.principal</name>
		<value>hbase/_HOST@REALM.CA</value>
	</property>
	<property>
		<name>hbase.regionserver.keytab.file</name>
		<value>/etc/security/keytabs/hbase.service.keytab</value>
	</property>
	<property>
		<name>hbase.master.kerberos.principal</name>
		<value>hbase/_HOST@REALM.CA</value>
	</property>
	<property>
		<name>hbase.master.keytab.file</name>
		<value>/etc/security/keytabs/hbase.service.keytab</value>
	</property>
	<property>
		<name>hbase.security.authentication.spnego.kerberos.principal</name>
		<value>hbaseHTTP/_HOST@REALM.CA</value>
	</property>
	<property>
		<name>hbase.security.authentication.spnego.kerberos.keytab</name>
		<value>/etc/security/keytabs/hbaseHTTP.service.keytab</value>
	</property>
	<property>
		<name>hbase.security.authentication</name>
		<value>kerberos</value>
	</property>
	<property>
		<name>hbase.security.authorization</name>
		<value>true</value>
	</property>
	<property>
		<name>hbase.coprocessor.region.classes</name>
		<value>org.apache.hadoop.hbase.security.token.TokenProvider</value>
	</property>
	<property>
		<name>hbase.rpc.protection</name>
		<value>integrity</value>
	</property>
	<property>
		<name>hbase.rpc.engine</name>
		<value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
	</property>
	<property>
		<name>hbase.coprocessor.master.classes</name>
		<value>org.apache.hadoop.hbase.security.access.AccessController</value>
	</property>
	<property>
		<name>hbase.coprocessor.region.classes</name>
		<value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController</value>
	</property>
	<property>
		<name>hbase.security.authentication.ui</name>
		<value>kerberos</value>
		<description>Controls what kind of authentication should be used for the HBase web UIs.</description>
	</property>
	<property>
		<name>hbase.master.port</name>
		<value>16000</value>
	</property>
	<property>
		<name>hbase.master.info.bindAddress</name>
		<value>0.0.0.0</value>
	</property>
	<property>
		<name>hbase.master.info.port</name>
		<value>16010</value>
	</property>
	<property>
		<name>hbase.regionserver.hostname</name>
		<value>hadoop</value>
	</property>
	<property>
		<name>hbase.regionserver.port</name>
		<value>16020</value>
	</property>
	<property>
		<name>hbase.regionserver.info.port</name>
		<value>16030</value>
	</property>
	<property>
		<name>hbase.regionserver.info.bindAddress</name>
		<value>0.0.0.0</value>
	</property>
	<property>
		<name>hbase.master.ipc.address</name>
		<value>0.0.0.0</value>
	</property>
	<property>
		<name>hbase.regionserver.ipc.address</name>
		<value>0.0.0.0</value>
	</property>
	<property>
		<name>hbase.ssl.enabled</name>
		<value>true</value>
	</property>
	<property>
		<name>hadoop.ssl.enabled</name>
		<value>true</value>
	</property>
	<property>
		<name>ssl.server.keystore.keypassword</name>
		<value>startrek</value>
	</property>
	<property>
		<name>ssl.server.keystore.password</name>
		<value>startrek</value>
	</property>
	<property>
		<name>ssl.server.keystore.location</name>
		<value>/etc/security/serverKeys/keystore.jks</value>
	</property>
	<property>
		<name>hbase.rest.ssl.enabled</name>
		<value>true</value>
	</property>
	<property>
		<name>hbase.rest.ssl.keystore.store</name>
		<value>/etc/security/serverKeys/keystore.jks</value>
	</property>
	<property>
		<name>hbase.rest.ssl.keystore.password</name>
		<value>startrek</value>
	</property>
	<property>
		<name>hbase.rest.ssl.keystore.keypassword</name>
		<value>startrek</value>
	</property>
	<property>
		<name>hbase.superuser</name>
		<value>hduser</value>
	</property>
	<property>
		<name>hbase.tmp.dir</name>
		<value>/tmp/hbase-${user.name}</value>
	</property>
	<property>
		<name>hbase.local.dir</name>
		<value>${hbase.tmp.dir}/local</value>
	</property>
	<property>
		<name>hbase.zookeeper.property.clientPort</name>
		<value>2181</value>
	</property>
	<property>
		<name>hbase.unsafe.stream.capability.enforce</name>
		<value>false</value>
	</property>
	<property>
		<name>hbase.zookeeper.quorum</name>
		<value>hadoop</value>
	</property>
	<property>
		<name>zookeeper.znode.parent</name>
		<value>/hbase-secure</value>
	</property>
	<property>
		<name>hbase.regionserver.dns.interface</name>
		<value>enp0s3</value>
	</property>
        <property>
                <name>hbase.rest.authentication.type</name>
                <value>kerberos</value>
        </property>
        <property>
                <name>hadoop.proxyuser.HTTP.groups</name>
                <value>*</value>
        </property>
        <property>
                <name>hadoop.proxyuser.HTTP.hosts</name>
                <value>*</value>
        </property>
        <property>
                <name>hbase.rest.authentication.kerberos.keytab</name>
                <value>/etc/security/keytabs/hbaseHTTP.service.keytab</value>
        </property>
        <property>
                <name>hbase.rest.authentication.kerberos.principal</name>
                <value>hbaseHTTP/_HOST@REALM.CA</value>
        </property>
        <property>
                <name>hbase.rest.kerberos.principal</name>
                <value>hbase/_HOST@REALM.CA</value>
        </property>
        <property>
                <name>hbase.rest.keytab.file</name>
                <value>/etc/security/keytabs/hbase.service.keytab</value>
        </property>
</configuration>

Change Ownership of HBase files

sudo chown hadoopuser:hadoopuser -R /usr/local/hbase/*

Hadoop HDFS Config Changes

You will need to add two properties into the core-site.xml file of Hadoop.

nano /usr/local/hadoop/etc/hadoop/core-site.xml
 
<property>
	<name>hadoop.proxyuser.hbase.hosts</name>
	<value>*</value>
</property>
<property>
	<name>hadoop.proxyuser.hbase.groups</name>
	<value>*</value>
</property>
<property>
	<name>hadoop.proxyuser.HTTP.hosts</name>
	<value>*</value>
</property>
<property>
	<name>hadoop.proxyuser.HTTP.groups</name>
	<value>*</value>
</property>

AutoStart

crontab -e
 
@reboot /usr/local/hbase/bin/hbase-daemon.sh --config /usr/local/hbase/conf/ start master
@reboot /usr/local/hbase/bin/hbase-daemon.sh --config /usr/local/hbase/conf/ start regionserver
@reboot /usr/local/hbase/bin/hbase-daemon.sh --config /usr/local/hbase/conf/ start rest --infoport 17001 -p 17000

Validation

kinit -kt /etc/security/keytabs/hbase.service.keytab hbase/hadoop@REALM.ca
hbase shell
status 'detailed'
whoami
kdestroy

References

https://hbase.apache.org/0.94/book/security.html
https://pivotalhd-210.docs.pivotal.io/doc/2100/webhelp/topics/ConfiguringSecureHBase.html
https://ambari.apache.org/1.2.5/installing-hadoop-using-ambari/content/ambari-kerb-2-3-2-1.html
https://hbase.apache.org/book.html#_using_secure_http_https_for_the_web_ui

Zookeeper Kerberos Installation

We are going to install Zookeeper. Ensure you install Kerberos.

This assumes your hostname is “hadoop”

Install Java JDK

apt-get update
apt-get upgrade
apt-get install default-jdk

Download Zookeeper:

wget http://apache.forsale.plus/zookeeper/zookeeper-3.4.13/zookeeper-3.4.13.tar.gz
tar -zxvf zookeeper-3.4.13.tar.gz
sudo mv zookeeper-3.4.13 /usr/local/zookeeper/
sudo chown -R root:hadoopuser /usr/local/zookeeper/

Setup .bashrc:

 sudo nano ~/.bashrc

Add the following to the end of the file.

#ZOOKEEPER VARIABLES START
export ZOOKEEPER_HOME=/usr/local/zookeeper
export PATH=$PATH:$ZOOKEEPER_HOME/bin
#ZOOKEEPER VARIABLES STOP

 source ~/.bashrc

Create Kerberos Principals

cd /etc/security/keytabs
sudo kadmin.local
addprinc -randkey zookeeper/hadoop@REALM.CA
xst -kt zookeeper.service.keytab zookeeper/hadoop@REALM.CA
q

Set Keytab Permissions/Ownership

sudo chown root:hadoopuser /etc/security/keytabs/*
sudo chmod 750 /etc/security/keytabs/*

zoo.cfg

cd /usr/local/zookeeper/conf/
cp zoo_sample.cfg zoo.cfg
nano zoo.cfg

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/usr/local/zookeeper/data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to “0” to disable auto purge feature
#autopurge.purgeInterval=1

server.1=hadoop:2888:3888

authProvider.1 = org.apache.zookeeper.server.auth.SASLAuthenticationProvider
kerberos.removeHostFromPrincipal = true
kerberos.removeRealmFromPrincipal = true
jaasLoginRenew=3600000

java.env

cd /usr/local/zookeeper/conf/
touch java.env
nano java.env

ZOO_LOG4J_PROP=”INFO,ROLLINGFILE”
ZOO_LOG_DIR=”/usr/local/zookeeper/logs”

zookeeper_client_jaas.conf

cd /usr/local/zookeeper/conf/
touch zookeeper_client_jaas.conf
nano zookeeper_client_jaas.conf

Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=false
useTicketCache=true;
};

zookeeper_jaas.conf

cd /usr/local/zookeeper/conf/
touch zookeeper_jaas.conf
nano zookeeper_jaas.conf

Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
useTicketCache=false
keyTab=”/etc/security/keytabs/zookeeper.service.keytab”
principal=”zookeeper/hadoop@REALM.CA”;
};

zkServer.sh

cd /usr/local/zookeeper/bin/
nano zkServer.sh
 
#Add the following at the top
 
export CLIENT_JVMFLAGS="-Djava.security.auth.login.config=/usr/local/zookeeper/conf/zookeeper_client_jaas.conf"
export SERVER_JVMFLAGS="-Xmx1024m -Djava.security.auth.login.config=/usr/local/zookeeper/conf/zookeeper_jaas.conf"

zkCli.sh

cd /usr/local/zookeeper/bin/
nano zkCli.sh
 
#Add the following at the top
 
export CLIENT_JVMFLAGS="-Djava.security.auth.login.config=/usr/local/zookeeper/conf/zookeeper_client_jaas.conf"
export SERVER_JVMFLAGS="-Xmx1024m -Djava.security.auth.login.config=/usr/local/zookeeper/conf/zookeeper_jaas.conf"

MkDir

mkdir /usr/local/zookeeper/data/
mkdir /usr/local/zookeeper/logs/
 
echo "1" > /usr/local/zookeeper/data/myid
 
sudo chown -R hduser:hduser /usr/local/zookeeper

Auto Start

crontab -e
 
#Add the following
@reboot /usr/local/zookeeper/bin/zkServer.sh start

Run Client

kinit -kt /etc/security/keytabs/zookeeper.service.keytab zookeeper/hadoop@REALM.CA
./zkCli.sh -server 127.0.0.1:2181
 
#Now you can list all directories
ls /
 
#Or delete directories
 
rmr /folder

References

https://my-bigdata-blog.blogspot.com/2017/07/apache-Zookeeper-install-Ubuntu.html
https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_command-line-installation/content/zookeeper_configuration.html
https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_command-line-installation/content/securing_zookeeper_with_kerberos.html

Kafka & Java: Secured Consumer Read Record

In this tutorial I will show you how to read a record to Kafka. Before you begin you will need Maven/Eclipse all setup and a project ready to go. If you haven’t installed Kafka Kerberos yet please do so.

Import SSL Cert to Java:

Follow this tutorial to “Installing unlimited strength encryption Java libraries”

If on Windows do the following

#Import it
"C:\Program Files\Java\jdk1.8.0_171\bin\keytool" -import -file hadoop.csr -keystore "C:\Program Files\Java\jdk1.8.0_171\jre\lib\security\cacerts" -alias "hadoop"
 
#Check it
"C:\Program Files\Java\jdk1.8.0_171\bin\keytool" -list -v -keystore "C:\Program Files\Java\jdk1.8.0_171\jre\lib\security\cacerts"
 
#If you want to delete it
"C:\Program Files\Java\jdk1.8.0_171\bin\keytool" -delete -alias hadoop -keystore "C:\Program Files\Java\jdk1.8.0_171\jre\lib\security\cacerts"

POM.xml

<dependency>
	<groupId>org.apache.kafka</groupId>
	<artifactId>kafka-clients</artifactId>
	<version>1.1.0</version>
</dependency>

Imports

import org.apache.kafka.clients.consumer.*;
import java.util.Properties;
import java.io.InputStream;
import java.util.Arrays;

Consumer JAAS Conf (client_jaas.conf)

KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useTicketCache=false
    refreshKrb5Config=true
    debug=true
    useKeyTab=true
    storeKey=true
    keyTab="c:\\data\\kafka.service.keytab"
    principal="kafka/hadoop@REALM.CA";
};

Consumer Props File

You can go here to view all the options for consumer properties.

bootstrap.servers=hadoop:9094
group.id=test
 
security.protocol=SASL_SSL
sasl.kerberos.service.name=kafka
 
#offset will be periodically committed in the background
enable.auto.commit=true
 
# The serializer for the key
key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
 
# The serializer for the value
value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
 
# heartbeat to detect worker failures
session.timeout.ms=10000
 
#Automatically reset offset to earliest offset
auto.offset.reset=earliest

Initiate Kerberos Authentication

System.setProperty("java.security.auth.login.config", "C:\\data\\kafkaconnect\\kafka\\src\\main\\resources\\client_jaas.conf");
System.setProperty("https.protocols", "TLSv1,TLSv1.1,TLSv1.2");
System.setProperty("java.security.krb5.conf", "C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\krb5.conf");
System.setProperty("java.security.krb5.realm", "REALM.CA");
System.setProperty("java.security.krb5.kdc", "REALM.CA");
System.setProperty("sun.security.krb5.debug", "false");
System.setProperty("javax.net.debug", "false");
System.setProperty("javax.net.ssl.keyStorePassword", "changeit");
System.setProperty("javax.net.ssl.keyStore", "C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\cacerts");
System.setProperty("javax.net.ssl.trustStore", "C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\cacerts");
System.setProperty("javax.net.ssl.trustStorePassword", "changeit");
System.setProperty("javax.security.auth.useSubjectCredsOnly", "true");

Consumer Connection/Send

The record we will read will just be a string for both key and value.

Consumer<String, String> consumer = null;
 
try {
	ClassLoader classLoader = getClass().getClassLoader();
 
	try (InputStream props = classLoader.getResourceAsStream("consumer.props")) {
		Properties properties = new Properties();
		properties.load(props);
		consumer = new KafkaConsumer<>(properties);
	}
	
	System.out.println("Consumer Created");
 
	// Subscribe to the topic.
	consumer.subscribe(Arrays.asList("testTopic"));
 
	while (true) {
		final ConsumerRecords<String, String> consumerRecords = consumer.poll(1000);
		
		if (consumerRecords.count() == 0) {
			//Keep reading till no records
			break;
		}
 
		consumerRecords.forEach(record -> {
			System.out.printf("Consumer Record:(%s, %s, %d, %d)\n", record.key(), record.value(), record.partition(), record.offset());
		});
 
		//Commit offsets returned on the last poll() for all the subscribed list of topics and partition
		consumer.commitAsync();
	}
} finally {
	consumer.close();
}
System.out.println("Consumer Closed");

References

I used kafka-sample-programs as a guide for setting up props.

Kafka & Java: Secured Producer Send Record

In this tutorial I will show you how to put a record to a secured Kafka. Before you begin you will need Maven/Eclipse all setup and a project ready to go. If you haven’t installed Kafka Kerberos yet please do so.

Import SSL Cert to Java:

Follow this tutorial to “Installing unlimited strength encryption Java libraries”

If on Windows do the following

#Import it
"C:\Program Files\Java\jdk1.8.0_171\bin\keytool" -import -file hadoop.csr -keystore "C:\Program Files\Java\jdk1.8.0_171\jre\lib\security\cacerts" -alias "hadoop"
 
#Check it
"C:\Program Files\Java\jdk1.8.0_171\bin\keytool" -list -v -keystore "C:\Program Files\Java\jdk1.8.0_171\jre\lib\security\cacerts"
 
#If you want to delete it
"C:\Program Files\Java\jdk1.8.0_171\bin\keytool" -delete -alias hadoop -keystore "C:\Program Files\Java\jdk1.8.0_171\jre\lib\security\cacerts"

POM.xml

<dependency>
	<groupId>org.apache.kafka</groupId>
	<artifactId>kafka-clients</artifactId>
	<version>1.1.0</version>
</dependency>

Imports

import org.apache.kafka.clients.producer.*;
import java.util.Properties;
import java.io.InputStream;
import java.util.Arrays;

Producer JAAS Conf (client_jaas.conf)

KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useTicketCache=false
    refreshKrb5Config=true
    debug=true
    useKeyTab=true
    storeKey=true
    keyTab="c:\\data\\kafka.service.keytab"
    principal="kafka/hadoop@REALM.CA";
};

Producer Props File

You can go here to view all the options for producer properties.

bootstrap.servers=hadoop:9094
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
security.protocol=SASL_SSL
sasl.kerberos.service.name=kafka

Initiate Kerberos Authentication

System.setProperty("java.security.auth.login.config", "C:\\data\\kafkaconnect\\kafka\\src\\main\\resources\\client_jaas.conf");
System.setProperty("https.protocols", "TLSv1,TLSv1.1,TLSv1.2");
System.setProperty("java.security.krb5.conf", "C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\krb5.conf");
System.setProperty("java.security.krb5.realm", "REALM.CA");
System.setProperty("java.security.krb5.kdc", "REALM.CA");
System.setProperty("sun.security.krb5.debug", "false");
System.setProperty("javax.net.debug", "false");
System.setProperty("javax.net.ssl.keyStorePassword", "changeit");
System.setProperty("javax.net.ssl.keyStore", "C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\cacerts");
System.setProperty("javax.net.ssl.trustStore", "C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\cacerts");
System.setProperty("javax.net.ssl.trustStorePassword", "changeit");
System.setProperty("javax.security.auth.useSubjectCredsOnly", "true");

Producer Connection/Send

The record we will send will just be a string for both key and value.

Producer<String, String> producer = null;
 
try {
	ClassLoader classLoader = getClass().getClassLoader();
	//Get the props file and load to the producer.
	try (InputStream props = classLoader.getResourceAsStream("producer.props")) {
		Properties properties = new Properties();
		properties.load(props);
		producer = new KafkaProducer<>(properties);
	}
	//Setting up the record to send
	ProducerRecord<String, String> rec = new ProducerRecord<String, String>("testTopic", "Key", "Value");
	//Send the record and get the response
	RecordMetadata recMetaData = producer.send(rec).get();
 
	//You can now print out any relavent information you want about the RecordMetaData
 
	System.out.println("Producer Record Sent");
} finally {
	producer.flush();
	producer.close();
}

References

I used kafka-sample-programs as a guide for setting up props.

Kafka: Kerberize/SSL

In this tutorial I will show you how to use Kerberos/SSL with NiFi. I will use self signed certs for this example. Before you begin ensure you have installed Kerberos Server and Kafka.

If you don’t want to use the built in Zookeeper you can setup your own. To do that following this tutorial.

This assumes your hostname is “hadoop”

Create Kerberos Principals

cd /etc/security/keytabs/
 
sudo kadmin.local
 
#You can list princepals
listprincs
 
#Create the following principals
addprinc -randkey kafka/hadoop@REALM.CA
addprinc -randkey zookeeper/hadoop@REALM.CA
 
#Create the keytab files.
#You will need these for Hadoop to be able to login
xst -k kafka.service.keytab kafka/hadoop@REALM.CA
xst -k zookeeper.service.keytab zookeeper/hadoop@REALM.CA

Set Keytab Permissions/Ownership

sudo chown root:hadoopuser /etc/security/keytabs/*
sudo chmod 750 /etc/security/keytabs/*

Hosts Update

sudo nano /etc/hosts
 
#Remove 127.0.1.1 line
 
#Change 127.0.0.1 to the following
127.0.0.1 realm.ca hadoop localhost

Ubuntu Firewall

sudo ufw disable

SSL

Setup SSL Directories if you have not previously done so.

sudo mkdir -p /etc/security/serverKeys
sudo chown -R root:hadoopuser /etc/security/serverKeys/
sudo chmod 755 /etc/security/serverKeys/
 
cd /etc/security/serverKeys

Setup Keystore

sudo keytool -genkey -alias NAMENODE -keyalg RSA -keysize 1024 -dname "CN=NAMENODE,OU=ORGANIZATION_UNIT,C=canada" -keypass PASSWORD -keystore /etc/security/serverKeys/keystore.jks -storepass PASSWORD
sudo keytool -export -alias NAMENODE -keystore /etc/security/serverKeys/keystore.jks -rfc -file /etc/security/serverKeys/NAMENODE.csr -storepass PASSWORD

Setup Truststore

sudo keytool -import -noprompt -alias NAMENODE -file /etc/security/serverKeys/NAMENODE.csr -keystore /etc/security/serverKeys/truststore.jks -storepass PASSWORD

Generate Self Signed Certifcate

sudo openssl genrsa -out /etc/security/serverKeys/NAMENODE.key 2048
 
sudo openssl req -x509 -new -key /etc/security/serverKeys/NAMENODE.key -days 300 -out /etc/security/serverKeys/NAMENODE.pem
 
sudo keytool -keystore /etc/security/serverKeys/keystore.jks -alias NAMENODE -certreq -file /etc/security/serverKeys/NAMENODE.cert -storepass PASSWORD -keypass PASSWORD
 
sudo openssl x509 -req -CA /etc/security/serverKeys/NAMENODE.pem -CAkey /etc/security/serverKeys/NAMENODE.key -in /etc/security/serverKeys/NAMENODE.cert -out /etc/security/serverKeys/NAMENODE.signed -days 300 -CAcreateserial

Setup File Permissions

sudo chmod 440 /etc/security/serverKeys/*
sudo chown root:hadoopuser /etc/security/serverKeys/*

Edit server.properties Config

cd /usr/local/kafka/config
 
sudo nano server.properties
 
#Edit or Add the following properties.
ssl.endpoint.identification.algorithm=HTTPS
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.key.password=PASSWORD
ssl.keystore.location=/etc/security/serverKeys/keystore.jks
ssl.keystore.password=PASSWORD
ssl.truststore.location=/etc/security/serverKeys/truststore.jks
ssl.truststore.password=PASSWORD
listeners=SASL_SSL://:9094
security.inter.broker.protocol=SASL_SSL
ssl.client.auth=required
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
ssl.keystore.type=JKS
ssl.truststore.type=JKS
sasl.kerberos.service.name=kafka
zookeeper.connect=hadoop:2181
sasl.mechanism.inter.broker.protocol=GSSAPI
sasl.enabled.mechanisms=GSSAPI

Edit zookeeper.properties Config

sudo nano zookeeper.properties
 
#Edit or Add the following properties.
 
server.1=hadoop:2888:3888
clientPort=2181
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=SASL
jaasLoginRenew=3600000

Edit producer.properties Config

sudo nano producer.properties
 
bootstrap.servers=hadoop:9094
security.protocol=SASL_SSL
sasl.kerberos.service.name=kafka
ssl.truststore.location=/etc/security/serverKeys/truststore.jks
ssl.truststore.password=PASSWORD
ssl.keystore.location=/etc/security/serverKeys/keystore.jks
ssl.keystore.password=PASSWORD
ssl.key.password=PASSWORD
sasl.mechanism=GSSAPI

Edit consumer.properties Config

sudo nano consumer.properties
 
zookeeper.connect=hadoop:2181
bootstrap.servers=hadoop:9094
group.id=securing-kafka-group
security.protocol=SASL_SSL
sasl.kerberos.service.name=kafka
ssl.truststore.location=/etc/security/serverKeys/truststore.jks
ssl.truststore.password=PASSWORD
sasl.mechanism=GSSAPI

Add zookeeper_jass.conf Config

sudo nano zookeeper_jass.conf
 
Server {
  com.sun.security.auth.module.Krb5LoginModule required
  debug=true
  useKeyTab=true
  keyTab="/etc/security/keytabs/zookeeper.service.keytab"
  storeKey=true
  useTicketCache=true
  refreshKrb5Config=true
  principal="zookeeper/hadoop@REALM.CA";
};

Add kafkaserver_jass.conf Config

sudo nano kafkaserver_jass.conf
 
KafkaServer {
    com.sun.security.auth.module.Krb5LoginModule required
    debug=true
    useKeyTab=true
    storeKey=true
    refreshKrb5Config=true
    keyTab="/etc/security/keytabs/kafka.service.keytab"
    principal="kafka/hadoop@REALM.CA";
};
 
kafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useTicketCache=true
    refreshKrb5Config=true
    debug=true
    useKeyTab=true
    storeKey=true
    keyTab="/etc/security/keytabs/kafka.service.keytab"
    principal="kafka/hadoop@REALM.CA";
};

Edit kafka-server-start.sh

cd /usr/local/kafka/bin/
 
sudo nano kafka-server-start.sh
 
jaas="$base_dir/../config/kafkaserver_jaas.conf"
 
export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=$jaas"

Edit zookeeper-server-start.sh

sudo nano zookeeper-server-start.sh
 
jaas="$base_dir/../config/zookeeper_jaas.conf"
 
export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=$jaas"

Kafka-ACL

cd /usr/local/kafka/bin/
 
#Grant topic access and cluster access
./kafka-acls.sh  --operation All --allow-principal User:kafka --authorizer-properties zookeeper.connect=hadoop:2181 --add --cluster
./kafka-acls.sh  --operation All --allow-principal User:kafka --authorizer-properties zookeeper.connect=hadoop:2181 --add --topic TOPIC
 
#Grant all groups for a specific topic
./kafka-acls.sh --operation All --allow-principal User:kafka --authorizer-properties zookeeper.connect=hadoop:2181 --add --topic TOPIC --group *
 
#If you want to remove cluster access
./kafka-acls.sh --authorizer-properties zookeeper.connect=hadoop:2181 --remove --cluster
 
#If you want to remove topic access
./kafka-acls.sh --authorizer-properties zookeeper.connect=hadoop:2181 --remove --topic TOPIC
 
#List access for cluster
./kafka-acls.sh --list --authorizer-properties zookeeper.connect=hadoop:2181 --cluster
 
#List access for topic
./kafka-acls.sh --list --authorizer-properties zookeeper.connect=hadoop:2181 --topic TOPIC

kafka-console-producer.sh

If you want to test using the console producer you need to make these changes.

cd /usr/local/kafka/bin/
nano kafka-console-producer.sh
 
#Add the below before the last line
 
base_dir=$(dirname $0)
jaas="$base_dir/../config/kafkaserver_jaas.conf"
export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=$jaas"
 
 
#Now you can run the console producer
./kafka-console-producer.sh --broker-list hadoop:9094 --topic TOPIC -producer.config ../config/producer.properties

kafka-console-consumer.sh

If you want to test using the console consumer you need to make these changes.

cd /usr/local/kafka/bin/
nano kafka-console-consumer.sh
 
#Add the below before the last line
 
base_dir=$(dirname $0)
jaas="$base_dir/../config/kafkaserver_jaas.conf"
export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=$jaas"
 
 
#Now you can run the console consumer
./kafka-console-consumer.sh --bootstrap-server hadoop:9094 --topic TOPIC --consumer.config ../config/consumer.properties --from-beginning

References

https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/
https://github.com/confluentinc/securing-kafka-blog/blob/master/manifests/default.pp

Kafka & Java: Consumer Seek To Beginning

This is a quick tutorial on how to seek to beginning using a Kafka consumer. If you haven’t setup the consumer yet follow this tutorial.

This is all that is required once you have setup the consumer. This will put the kafka offset for the topic of your choice to the beginning so once you start reading you will get all records.

consumer.seekToBeginning(consumer.assignment());

Hive & Java: Connect to Remote Kerberos Hive using KeyTab

In this tutorial I will show you how to connect to remote Kerberos Hive cluster using Java. If you haven’t install Hive yet follow the tutorial.

Import SSL Cert to Java:

Follow this tutorial to “Installing unlimited strength encryption Java libraries”

If on Windows do the following

#Import it
"C:\Program Files\Java\jdk1.8.0_171\bin\keytool" -import -file hadoop.csr -keystore "C:\Program Files\Java\jdk1.8.0_171\jre\lib\security\cacerts" -alias "hadoop"
 
#Check it
"C:\Program Files\Java\jdk1.8.0_171\bin\keytool" -list -v -keystore "C:\Program Files\Java\jdk1.8.0_171\jre\lib\security\cacerts"
 
#If you want to delete it
"C:\Program Files\Java\jdk1.8.0_171\bin\keytool" -delete -alias hadoop -keystore "C:\Program Files\Java\jdk1.8.0_171\jre\lib\security\cacerts"

POM.xml:

<dependency>
	<groupId>org.apache.hive</groupId>
	<artifactId>hive-jdbc</artifactId>
	<version>2.3.3</version>
	<exclusions>
		<exclusion>
			<groupId>jdk.tools</groupId>
			<artifactId>jdk.tools</artifactId>
		</exclusion>
	</exclusions>
</dependency>

Imports:

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.security.UserGroupInformation;
import java.sql.SQLException;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.Statement;
import java.sql.DriverManager;

Connect:

// Setup the configuration object.
final Configuration config = new Configuration();
 
config.set("fs.defaultFS", "swebhdfs://hadoop:50470");
config.set("hadoop.security.authentication", "kerberos");
config.set("hadoop.rpc.protection", "integrity");
 
System.setProperty("https.protocols", "TLSv1,TLSv1.1,TLSv1.2");
System.setProperty("java.security.krb5.conf", "C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\krb5.conf");
System.setProperty("java.security.krb5.realm", "REALM.CA");
System.setProperty("java.security.krb5.kdc", "REALM.CA");
System.setProperty("sun.security.krb5.debug", "true");
System.setProperty("javax.net.debug", "all");
System.setProperty("javax.net.ssl.keyStorePassword","changeit");
System.setProperty("javax.net.ssl.keyStore","C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\cacerts");
System.setProperty("javax.net.ssl.trustStore", "C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\cacerts");
System.setProperty("javax.net.ssl.trustStorePassword","changeit");
System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");
 
UserGroupInformation.setConfiguration(config);
UserGroupInformation.setLoginUser(UserGroupInformation.loginUserFromKeytabAndReturnUGI("hive/hadoop@REALM.CA", "c:\\data\\hive.service.keytab"));
 
System.out.println(UserGroupInformation.getLoginUser());
System.out.println(UserGroupInformation.getCurrentUser());
 
//Add the hive driver
Class.forName("org.apache.hive.jdbc.HiveDriver");
 
//Connect to hive jdbc
Connection connection = DriverManager.getConnection("jdbc:hive2://hadoop:10000/default;principal=hive/hadoop@REALM.CA");
Statement statement = connection.createStatement();
 
//Create a table
String createTableSql = "CREATE TABLE IF NOT EXISTS "
		+" employee ( eid int, name String, "
		+" salary String, designation String)"
		+" COMMENT 'Employee details'"
		+" ROW FORMAT DELIMITED"
		+" FIELDS TERMINATED BY '\t'"
		+" LINES TERMINATED BY '\n'"
		+" STORED AS TEXTFILE";
 
System.out.println("Creating Table: " + createTableSql);
statement.executeUpdate(createTableSql);
 
//Show all the tables to ensure we successfully added the table
String showTablesSql = "show tables";
System.out.println("Show All Tables: " + showTablesSql);
ResultSet res = statement.executeQuery(showTablesSql);
 
while (res.next()) {
	System.out.println(res.getString(1));
}
 
//Drop the table
String dropTablesSql = "DROP TABLE IF EXISTS employee";
 
System.out.println("Dropping Table: " + dropTablesSql);
statement.executeUpdate(dropTablesSql);
 
System.out.println("Finish!");

NiFi: Kerberize/SSL

In this tutorial I will show you how to use Kerberos/SSL with NiFi. I will use self signed certs for this example. Before you begin ensure you have installed Kerberos Server and NiFi.

This assumes your hostname is “hadoop”

Create Kerberos Principals

cd /etc/security/keytabs/
 
sudo kadmin.local
 
#You can list principals
listprincs
 
#Create the following principals
addprinc -randkey nifi/hadoop@REALM.CA
addprinc -randkey nifi-spnego/hadoop@REALM.CA
#Notice this user does not have -randkey because we are a login user
#Also notice that this user does not have a keytab created
addprinc admin/hadoop@REALM.CA
 
 
#Create the keytab files.
#You will need these for Hadoop to be able to login
xst -k nifi.service.keytab nifi/hadoop@REALM.CA
xst -k nifi-spnego.service.keytab nifi-spnego/hadoop@REALM.CA

Set Keytab Permissions/Ownership

sudo chown root:hadoopuser /etc/security/keytabs/*
sudo chmod 750 /etc/security/keytabs/*

Stop NiFi

sudo service nifi stop

Hosts Update

sudo nano /etc/hosts
 
#Remove 127.0.1.1 line
 
#Change 127.0.0.1 to the following
127.0.0.1 gaudreault_kdc.ca hadoop localhost

Ubuntu Firewall

sudo ufw disable

sysctl.conf

Disable ipv6 as it causes issues in getting your server up and running.

nano /etc/sysctl.conf

Add the following to the end and save

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
#Change eth0 to what ifconfig has
net.ipv6.conf.eth0.disable_ipv6 = 1

Close sysctl

sysctl -p
cat /proc/sys/net/ipv6/conf/all/disable_ipv6
reboot

TrustStore / KeyStore

#Creating your Certificate Authority
sudo mkdir -p /etc/security/serverKeys
sudo chown -R root:hduser /etc/security/serverKeys/
sudo chmod 750 /etc/security/serverKeys/
 
cd /etc/security/serverKeys
 
sudo openssl genrsa -aes128 -out nifi.key 4096
sudo openssl req -x509 -new -key nifi.key -days 1095 -out nifi.pem
sudo openssl rsa -check -in nifi.key #check it
sudo openssl x509 -outform der -in nifi.pem -out nifi.der
sudo keytool -import -keystore truststore.jks -file nifi.der -alias nifi
#***You must type 'yes' to trust this certificate.
sudo keytool -v -list -keystore truststore.jks
 
#Creating your Server Keystore
sudo keytool -genkey -alias nifi -keyalg RSA -keystore keystore.jks -keysize 2048
sudo keytool -certreq -alias nifi -keystore keystore.jks -file nifi.csr
sudo openssl x509 -sha256 -req -in nifi.csr -CA nifi.pem -CAkey nifi.key -CAcreateserial -out nifi.crt -days 730
sudo keytool -import -keystore keystore.jks -file nifi.pem
sudo keytool -import -trustcacerts -alias nifi -file nifi.crt -keystore keystore.jks
 
sudo chown -R root:hduser /etc/security/serverKeys/*
sudo chmod 750 /etc/security/serverKeys/*

nifi.properties

cd /usr/local/nifi/conf/
nano nifi.properties
 
#Find "# Site to Site properties" and change the following properties to what is below
 
nifi.remote.input.host=
nifi.remote.input.secure=true
nifi.remote.input.socket.port=9096
nifi.remote.input.http.enabled=false
 
#Find "# web properties #" and change the following properties to what is below
 
nifi.web.http.host=
nifi.web.http.port=
nifi.web.https.host=0.0.0.0
nifi.web.https.port=9095
 
#Find "# security properties #" and change the following properties to what is below
 
nifi.security.keystore=/etc/security/serverKeys/keystore.jks
nifi.security.keystoreType=JKS
nifi.security.keystorePasswd=PASSWORD
nifi.security.keyPasswd=PASSWORD
nifi.security.truststore=/etc/security/serverKeys/truststore.jks
nifi.security.truststoreType=JKS
nifi.security.truststorePasswd=PASSWORD
nifi.security.needClientAuth=true
nifi.security.user.authorizer=managed-authorizer
nifi.security.user.login.identity.provider=kerberos-provider
 
#Find "# Core Properties #" and change the following properties to what is below
 
nifi.authorizer.configuration.file=./conf/authorizers.xml
nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
 
#Find "# kerberos #" and change the following properties to what is below
 
nifi.kerberos.krb5.file=/etc/krb5.conf
 
#Find "# kerberos service principal #" and change the following properties to what is below
 
nifi.kerberos.service.principal=nifi/hadoop@REALM.CA
nifi.kerberos.service.keytab.location=/etc/security/keytabs/nifi.service.keytab
 
#Find "# kerberos spnego principal #" and change the following properties to what is below
 
nifi.kerberos.spnego.principal=nifi-spnego/hadoop@REALM.CA
nifi.kerberos.spnego.keytab.location=/etc/security/keytabs/nifi-spnego.service.keytab
nifi.kerberos.spnego.authentication.expiration=12 hours
 
#Find "# cluster common properties (all nodes must have same values) #" and change the following properties to what is below
 
nifi.cluster.protocol.is.secure=true

login-identity-providers.xml

nano login-identity-providers.xml
 
#Find "kerberos-provider"
<provider>
	<identifier>kerberos-provider</identifier>
	<class>org.apache.nifi.kerberos.KerberosProvider</class>
	<property name="Default Realm">REALM.CA</property>
	<property name="Kerberos Config File">/etc/krb5.conf</property>
	<property name="Authentication Expiration">12 hours</property>
</provider>

authorizers.xml

nano authorizers.xml
 
#Find "file-provider"
<authorizer>
	<identifier>file-provider</identifier>
	<class>org.apache.nifi.authorization.FileAuthorizer</class>
	<property name="Authorizations File">./conf/authorizations.xml</property>
	<property name="Users File">./conf/users.xml</property>
	<property name="Initial Admin Identity">admin/hadoop@REALM.CA</property>
	<property name="Legacy Authorized Users File"></property>
 
	<property name="Node Identity 1"></property>
</authorizer>

Start Nifi

sudo service nifi start

NiFi Web Login

Issues:

If you get the error “No applicable policies could be found” after logging in and no GUI is shown stop the NiFi service and restart. Then you should be good.
If you can then login but you don’t have any policies still you will need to update “authorizations.xml” and add the below lines. Making sure to change the resource process group id to the root process group id and the user id to the user id

nano /usr/local/nifi/conf/authorizations.xml
 
<policy identifier="1c897e9d-3dd5-34ca-ae3d-75fb5ee3e1a5" resource="/data/process-groups/##CHANGE TO ROOT ID##" action="R">
	<user identifier="##CHANGE TO USER ID##"/>
</policy>
<policy identifier="91c64c2d-7848-371d-9d5f-db71138b152f" resource="/data/process-groups/##CHANGE TO ROOT ID##" action="W">
	<user identifier="##CHANGE TO USER ID##"/>
</policy>
<policy identifier="7aeb4d67-e2e1-3a3e-a8fa-94576f35539e" resource="/process-groups/##CHANGE TO ROOT ID##" action="R">
	<user identifier="##CHANGE TO USER ID##"/>
</policy>
<policy identifier="f5b620e0-b094-3f70-9542-dd6920ad5bd9" resource="/process-groups/##CHANGE TO ROOT ID##" action="W">
	<user identifier="##CHANGE TO USER ID##"/>
</policy>

References

https://community.hortonworks.com/articles/34147/nifi-security-user-authentication-with-kerberos.html

https://community.hortonworks.com/content/supportkb/151106/nifi-how-to-create-your-own-certs-for-securing-nif.html