Oliver

Scala: Basic Class Creation

In this tutorial I will show you how to create your first Scala class and then use it. I am just beginning with Scala during the time of this writing. Review the Scala style guide.

So the first thing we want to do is determine what we want to create a class to represent. In this tutorial I am just going to play around and use Person. We will want to create a constructor, getters, setters, toString and then finally a method to combine some properties.

Create your class.

class Person {
}

We could have added variables to the Person declaration. But I thought I’d leave that out for now.

Create our private first and last name.

private var _firstName: String = null

When we set variables as private in the class they are not accessible from outside the class. Notice how the variable starts with “_” this is just one of the Scala naming conventions.

Create our constructor

/**
* @constructor Creates a person with first/last name
* @param firstName the persons first name
* @param lastName the persons last name
*/
def this(firstName: String, lastName: String) {
  this()
  _firstName = firstName
  _lastName = lastName
}

This is where we can set the first and last name when we instantiate our object.

Create a getter

def firstName = _firstName

Create a setter

def firstName_=(firstName: String) {
  _firstName = firstName
}

Override toString

override def toString = s"firstName = $firstName"

Notice how their is “s” before the string and we have $firstname there. That will reference the variable itself.

Create a Method

def fullName: String = {
  return s"$firstName $lastName"
}

This will just give you the full name of the person.

Putting it all together

package models
 
class Person {
  private var _firstName: String = null
  private var _lastName: String = null
  
  /**
   * @constructor Creates a person with first/last name
   * @param firstName the persons first name
   * @param lastName the persons last name
   */
  def this(firstName: String, lastName: String) {
    this()
    _firstName = firstName
    _lastName = lastName
  }
    
  //Getter
  def firstName = _firstName
  def lastName = _lastName
  
  //Setter
  def firstName_=(firstName: String) {
    _firstName = firstName
  }
  
  def lastName_=(lastName: String) {
    _lastName = lastName
  }
  
  def fullName: String = {
    return s"$firstName $lastName"
  }
  
  override def toString = s"firstName = $firstName, lastName = $lastName"
}

So what I have shown you above will get you started on creating your first class but you could make it alot cleaner with less code. it’s entirely up to you how you want to proceed and what you feel comfortable with.

package models
 
class PersonCondensed {
  var firstName:String = null
  var lastName:String = null
  
  /**
   * @constructor Creates a person with first/last name
   * @param firstName the persons first name
   * @param lastName the persons last name
   */
  def this(firstName: String, lastName: String) {
    this()
    this.firstName = firstName
    this.lastName = lastName
  }
  
  def fullName: String = {
    return s"$firstName $lastName"
  }
  
  override def toString = s"firstName = $firstName, lastName = $lastName"
}

Using our class

Here are the three different ways of calling our classes we did above.

import models.Person
import models.PersonCondensed
 
object Test {
  def main(args: Array[String]) {
    val person = new Person() 
    person.firstName_=("John")
    person.lastName_=("Smith")
    person.value=(234)
    
    println(person.fullName)
    println(person.toString())    
    
    val person2 = new Person("John", "Smith") 
    
    println(person2.fullName)
    println(person2.toString())
    
    val person3 = new PersonCondensed()
    person3.firstName=("John")
    person3.lastName=("Smith")
    
    println(person3.firstName)
    println(person3.lastName)
    println(person3.fullName)
    println(person3.toString())
  }
}

Eclipse Installation

In this tutorial I will show you how to install Eclipse using Ubuntu 16.04.

Install JDK 8

sudo apt-get install openjdk-8-jdk

Download Oxygen.

tar -xzvf eclipse-inst-linux64.tar.gz
~/eclipse-installer/eclipse-inst

Install Eclipse

Eclipse Desktop Shortcut

cd ~/Desktop
touch eclipse.desktop
chmod u+x eclipse.desktop
nano eclipse.desktop
 
#Add the below to the file
 
[Desktop Entry]
Type=Application
Name=Eclipse
Icon=~/eclipse/java-oxygen/eclipse/icon.xpm
Exec=~/eclipse/java-oxygen/eclipse/eclipse
Terminal=false
Categories=Development;IDE;Java;
StartupWMClass=Eclipse

Scala IDE Installation for Eclipse

In this tutorial I will show you how to use Scala with Eclipse. This tutorial assumes you have already pre-installed Eclipse with JDK 8.

Step 1: Click install new software.

Step 2: Enter site: http://download.scala-ide.org/sdk/lithium/e47/scala212/stable/site

Step 3: Once installation is finished restart Eclipse.

Step 4: Switch to perspective “Scala”

Hadoop & Java: Connect to Remote Kerberos HDFS using KeyTab

In this tutorial I will show you how to connect to remote Kerberos HDFS cluster using Java. If you haven’t install hdfs with kerberos yet follow the tutorial.

Import SSL Cert to Java:

Follow this tutorial to “Installing unlimited strength encryption Java libraries”

If on Windows do the following

#Import it
"C:\Program Files\Java\jdk1.8.0_171\bin\keytool" -import -file hadoop.csr -keystore "C:\Program Files\Java\jdk1.8.0_171\jre\lib\security\cacerts" -alias "hadoop"
 
#Check it
"C:\Program Files\Java\jdk1.8.0_171\bin\keytool" -list -v -keystore "C:\Program Files\Java\jdk1.8.0_171\jre\lib\security\cacerts"
 
#If you want to delete it
"C:\Program Files\Java\jdk1.8.0_171\bin\keytool" -delete -alias hadoop -keystore "C:\Program Files\Java\jdk1.8.0_171\jre\lib\security\cacerts"

POM.xml:

<dependency>
	<groupId>org.apache.hadoop</groupId>
	<artifactId>hadoop-client</artifactId>
	<version>2.9.1</version>
</dependency>

Imports:

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.security.UserGroupInformation;

Connect:

// Setup the configuration object.
final Configuration config = new Configuration();
 
config.set("fs.defaultFS", "swebhdfs://hadoop:50470");
config.set("hadoop.security.authentication", "kerberos");
config.set("hadoop.rpc.protection", "integrity");
 
System.setProperty("https.protocols", "TLSv1,TLSv1.1,TLSv1.2");
System.setProperty("java.security.krb5.conf", "C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\krb5.conf");
System.setProperty("java.security.krb5.realm", "REALM.CA");
System.setProperty("java.security.krb5.kdc", "REALM.CA");
System.setProperty("sun.security.krb5.debug", "true");
System.setProperty("javax.net.debug", "all");
System.setProperty("javax.net.ssl.keyStorePassword","YOURPASSWORD");
System.setProperty("javax.net.ssl.keyStore","C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\cacerts");
System.setProperty("javax.net.ssl.trustStore", "C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\cacerts");
System.setProperty("javax.net.ssl.trustStorePassword","YOURPASSWORD");
System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");
 
UserGroupInformation.setConfiguration(config);
UserGroupInformation.setLoginUser(UserGroupInformation.loginUserFromKeytabAndReturnUGI("myuser/hadoop@REALM.CA", "c:\\data\\myuser.keytab"));
 
System.out.println(UserGroupInformation.getLoginUser());
System.out.println(UserGroupInformation.getCurrentUser());

VirtualBox

In this post I will just walk through a couple settings that you may need to configure. For these examples I am using Ubuntu 16.04.

Networking

If you want your virtualbox os to have it’s own IP on the network and accessable outside the vm then use “Bridged Adapter”.

Select “Enable Network Adapter”
Attached to: Bridged Adapter
Expand Advanced
Promiscuous Mode: Allow All
Select Cable Connected
Adapter Type: Desktop adapter.

If you want to just forward a port to localhost then use “NAT”.

Select “Enable Network Adapter”
Attached to: NAT
Expand Advanced
Adapter Type: Desktop adapter.
Click “Port Forwarding”
Add new Rule
- Host IP: 127.0.0.1
- Host Port: 22
- Guest IP: Leave empty
- Guest Port: 22

Shared Folders

If you want to share a folder from Host to Guest.

Click Add Share
Folder Path: Select folder from Host machine
- I will us C:\data
Folder Name: What the share will be on the Guest
- data
Turn on VM
Download https://download.virtualbox.org/virtualbox/5.2.12/VBoxGuestAdditions_5.2.12.iso
Mount VBoxGuestAdditions_5.2.12.iso
On the Guest Ubuntu server run
- sudo mount /dev/cdrom /media/cdrom
Install build-essential
- sudo apt-get install build-essential linux-headers-`uname –r`
Run VBoxGuestAdditions
- sudo /media/cdrom/VBoxLinuxAdditions.run
sudo mkdir -p /mnt/data
sudo reboot
Mount Share
- Ubuntu Server
  - sudo mount –t vboxsf data /mnt/data
- Ubuntu Desktop
  - sudo mount -t vboxsf -o uid=$UID,gid=$(id -g) data /mnt/data/
cd /mnt/data
- Now we can share files between host and guest.

HDFS/Yarn/MapRed: Kerberize/SSL

In this tutorial I will show you how to use Kerberos/SSL with HDFS/Yarn/MapRed. I will use self signed certs for this example. Before you begin ensure you have installed Kerberos Server and Hadoop.

This assumes your hostname is “hadoop”

Create Kerberos Principals

cd /etc/security/keytabs/
 
sudo kadmin.local
 
#You can list princepals
listprincs
 
#Create the following principals
addprinc -randkey nn/hadoop@REALM.CA
addprinc -randkey jn/hadoop@REALM.CA
addprinc -randkey dn/hadoop@REALM.CA
addprinc -randkey sn/hadoop@REALM.CA
addprinc -randkey nm/hadoop@REALM.CA
addprinc -randkey rm/hadoop@REALM.CA
addprinc -randkey jhs/hadoop@REALM.CA
addprinc -randkey HTTP/hadoop@REALM.CA
 
#We are going to create a user to access with later
addprinc -pw hadoop myuser/hadoop@REALM.CA
xst -k myuser.keytab myuser/hadoop@REALM.CA
 
#Create the keytab files.
#You will need these for Hadoop to be able to login
xst -k nn.service.keytab nn/hadoop@REALM.CA
xst -k jn.service.keytab jn/hadoop@REALM.CA
xst -k dn.service.keytab dn/hadoop@REALM.CA
xst -k sn.service.keytab sn/hadoop@REALM.CA
xst -k nm.service.keytab nm/hadoop@REALM.CA
xst -k rm.service.keytab rm/hadoop@REALM.CA
xst -k jhs.service.keytab jhs/hadoop@REALM.CA
xst -k spnego.service.keytab HTTP/hadoop@REALM.CA

Set Keytab Permissions/Ownership

sudo chown root:hadoopuser /etc/security/keytabs/*
sudo chmod 750 /etc/security/keytabs/*

Stop the Cluster

stop-dfs.sh
stop-yarn.sh
mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR stop historyserver

Hosts Update

sudo nano /etc/hosts
 
#Remove 127.0.1.1 line
 
#Change 127.0.0.1 to the following
#Notice how realm.ca is there its because we need to tell where that host resides
127.0.0.1 realm.ca hadoop localhost

hadoop-env.sh

We don’t set the HADOOP_SECURE_DN_USER because we are going to use Kerberos

sudo nano /usr/local/hadoop/etc/hadoop/hadoop-env.sh
 
#Locate "export ${HADOOP_SECURE_DN_USER}=${HADOOP_SECURE_DN_USER}"
#and change to
 
export HADOOP_SECURE_DN_USER=

core-site.xml

nano /usr/local/hadoop/etc/hadoop/core-site.xml
 
<configuration>
	<property>
		<name>fs.defaultFS</name>
		<value>hdfs://NAMENODE:54310</value>
		<description>The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming
		the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem.</description>
	</property>
	<property>
		<name>hadoop.tmp.dir</name>
		<value>/app/hadoop/tmp</value>
	</property>
	<property>
		<name>hadoop.proxyuser.hadoopuser.hosts</name>
		<value>*</value>
	</property>
	<property>
		<name>hadoop.proxyuser.hadoopuser.groups</name>
		<value>*</value>
	</property>
	<property>
		<name>hadoop.security.authentication</name>
		<value>kerberos</value> <!-- A value of "simple" would disable security. -->
	</property>
	<property>
		<name>hadoop.security.authorization</name>
		<value>true</value>
	</property>
	<property>
		<name>hadoop.security.auth_to_local</name>
		<value>
		RULE:[2:$1@$0](nn/.*@.*REALM.TLD)s/.*/hdfs/
		RULE:[2:$1@$0](jn/.*@.*REALM.TLD)s/.*/hdfs/
		RULE:[2:$1@$0](dn/.*@.*REALM.TLD)s/.*/hdfs/
		RULE:[2:$1@$0](sn/.*@.*REALM.TLD)s/.*/hdfs/
		RULE:[2:$1@$0](nm/.*@.*REALM.TLD)s/.*/yarn/
		RULE:[2:$1@$0](rm/.*@.*REALM.TLD)s/.*/yarn/
		RULE:[2:$1@$0](jhs/.*@.*REALM.TLD)s/.*/mapred/
		DEFAULT
		</value>
	</property>
	<property>
		<name>hadoop.rpc.protection</name>
		<value>integrity</value>
	</property>
	<property>
		<name>hadoop.ssl.require.client.cert</name>
		<value>false</value>
	</property>
	<property>
		<name>hadoop.ssl.hostname.verifier</name>
		<value>DEFAULT</value>
	</property>
	<property>
		<name>hadoop.ssl.keystores.factory.class</name>
		<value>org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory</value>
	</property>
	<property>
		<name>hadoop.ssl.server.conf</name>
		<value>ssl-server.xml</value>
	</property>
	<property>
		<name>hadoop.ssl.client.conf</name>
		<value>ssl-client.xml</value>
	</property>
	<property>
		<name>hadoop.rpc.protection</name>
		<value>integrity</value>
	</property>
</configuration>

ssl-server.xml

Change ssl-server.xml.example to ssl-server.xml

cp /usr/local/hadoop/etc/hadoop/ssl-server.xml.example /usr/local/hadoop/etc/hadoop/ssl-server.xml
 
nano /usr/local/hadoop/etc/hadoop/ssl-server.xml

Update properties

<configuration>
	<property>
		<name>ssl.server.truststore.location</name>
		<value>/etc/security/serverKeys/truststore.jks</value>
		<description>Truststore to be used by NN and DN. Must be specified.</description>
	</property>
	<property>
		<name>ssl.server.truststore.password</name>
		<value>PASSWORD</value>
		<description>Optional. Default value is "".</description>
	</property>
	<property>
		<name>ssl.server.truststore.type</name>
		<value>jks</value>
		<description>Optional. The keystore file format, default value is "jks".</description>
	</property>
	<property>
		<name>ssl.server.truststore.reload.interval</name>
		<value>10000</value>
		<description>Truststore reload check interval, in milliseconds. Default value is 10000 (10 seconds).</description>
	</property>
	<property>
		<name>ssl.server.keystore.location</name>
		<value>/etc/security/serverKeys/keystore.jks</value>
		<description>Keystore to be used by NN and DN. Must be specified.</description>
	</property>
	<property>
		<name>ssl.server.keystore.password</name>
		<value>PASSWORD</value>
		<description>Must be specified.</description>
	</property>
	<property>
		<name>ssl.server.keystore.keypassword</name>
		<value>PASSWORD</value>
		<description>Must be specified.</description>
	</property>
	<property>
		<name>ssl.server.keystore.type</name>
		<value>jks</value>
		<description>Optional. The keystore file format, default value is "jks".</description>
	</property>
	<property>
		<name>ssl.server.exclude.cipher.list</name>
		<value>TLS_ECDHE_RSA_WITH_RC4_128_SHA,SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA,
		SSL_RSA_WITH_DES_CBC_SHA,SSL_DHE_RSA_WITH_DES_CBC_SHA,
		SSL_RSA_EXPORT_WITH_RC4_40_MD5,SSL_RSA_EXPORT_WITH_DES40_CBC_SHA,
		SSL_RSA_WITH_RC4_128_MD5</value>
		<description>Optional. The weak security cipher suites that you want excluded from SSL communication.</description>
	</property>
</configuration>

ssl-client.xml

Change ssl-client.xml.example to ssl-client.xml

cp /usr/local/hadoop/etc/hadoop/ssl-client.xml.example /usr/local/hadoop/etc/hadoop/ssl-client.xml
 
nano /usr/local/hadoop/etc/hadoop/ssl-client.xml

Update properties

<configuration>
	<property>
		<name>ssl.client.truststore.location</name>
		<value>/etc/security/serverKeys/truststore.jks</value>
		<description>Truststore to be used by clients like distcp. Must be specified.</description>
	</property>
	<property>
		<name>ssl.client.truststore.password</name>
		<value>PASSWORD</value>
		<description>Optional. Default value is "".</description>
	</property>
	<property>
		<name>ssl.client.truststore.type</name>
		<value>jks</value>
		<description>Optional. The keystore file format, default value is "jks".</description>
	</property>
	<property>
		<name>ssl.client.truststore.reload.interval</name>
		<value>10000</value>
		<description>Truststore reload check interval, in milliseconds. Default value is 10000 (10 seconds).</description>
	</property>
	<property>
		<name>ssl.client.keystore.location</name>
		<value></value>
		<description>Keystore to be used by clients like distcp. Must be specified.</description>
	</property>
	<property>
		<name>ssl.client.keystore.password</name>
		<value></value>
		<description>Optional. Default value is "".</description>
	</property>
	<property>
		<name>ssl.client.keystore.keypassword</name>
		<value></value>
		<description>Optional. Default value is "".</description>
	</property>
	<property>
		<name>ssl.client.keystore.type</name>
		<value>jks</value>
		<description>Optional. The keystore file format, default value is "jks".</description>
	</property>
</configuration>

mapred-site.xml

Just add the following to the config to let it know the Kerberos keytabs to use.

nano /usr/local/hadoop/etc/hadoop/mapred-site.xml
 
<property>
	<name>mapreduce.jobhistory.keytab</name>
	<value>/etc/security/keytabs/jhs.service.keytab</value>
</property>
<property>
	<name>mapreduce.jobhistory.principal</name>
	<value>jhs/_HOST@REALM.CA</value>
</property>
<property>
	<name>mapreduce.jobhistory.http.policy</name>
	<value>HTTPS_ONLY</value>
</property>

hdfs-site.xml

Add the following properties

nano /usr/local/hadoop/etc/hadoop/hdfs-site.xml
 
<property>
	<name>dfs.http.policy</name>
	<value>HTTPS_ONLY</value>
</property>
<property>
	<name>hadoop.ssl.enabled</name>
	<value>true</value>
</property>
<property>
	<name>dfs.datanode.https.address</name>
	<value>NAMENODE:50475</value>
</property>
<property>
	<name>dfs.namenode.https-address</name>
	<value>NAMENODE:50470</value>
	<description>Your NameNode hostname for http access.</description>
</property>
<property>
	<name>dfs.namenode.secondary.https-address</name>
	<value>NAMENODE:50091</value>
	<description>Your Secondary NameNode hostname for http access.</description>
</property>
<property>
	<name>dfs.namenode.https-bind-host</name>
	<value>0.0.0.0</value>
</property>
<property>
	<name>dfs.block.access.token.enable</name>
	<value>true</value>
	<description> If "true", access tokens are used as capabilities for accessing datanodes. If "false", no access tokens are checked on accessing datanod</description>
</property>
<property>
	<name>dfs.namenode.kerberos.principal</name>
	<value>nn/_HOST@REALM.CA</value>
	<description> Kerberos principal name for the NameNode</description>
</property>
<property>
	<name>dfs.secondary.namenode.kerberos.principal</name>
	<value>sn/_HOST@REALM.CA</value>
	<description>Kerberos principal name for the secondary NameNode.</description>
</property>
<property>
	<name>dfs.web.authentication.kerberos.keytab</name>
	<value>/etc/security/keytabs/spnego.service.keytab</value>
	<description>The Kerberos keytab file with the credentials for the HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint.</description>
</property>
<property>
	<name>dfs.namenode.keytab.file</name>
	<value>/etc/security/keytabs/nn.service.keytab</value>
	<description>Combined keytab file containing the namenode service and host principals.</description>
</property>
<property>
	<name>dfs.datanode.keytab.file</name>
	<value>/etc/security/keytabs/dn.service.keytab</value>
	<description>The filename of the keytab file for the DataNode.</description>
</property>
<property>
	<name>dfs.datanode.kerberos.principal</name>
	<value>dn/_HOST@REALM.CA</value>
	<description>The Kerberos principal that the DataNode runs as. "_HOST" is replaced by the real host name.</description>
</property>
<property>
	<name>dfs.namenode.kerberos.internal.spnego.principal</name>
	<value>${dfs.web.authentication.kerberos.principal}</value>
</property>
<property>
	<name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
	<value>>${dfs.web.authentication.kerberos.principal}</value>
</property>
<property>
	<name>dfs.web.authentication.kerberos.principal</name>
	<value>HTTP/_HOST@REALM.CA</value>
	<description>The HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint.</description>          
</property>
<property>
	<name>dfs.data.transfer.protection</name>
	<value>integrity</value>
</property>
<property>
	<name>dfs.datanode.address</name>
	<value>NAMENODE:50010</value>
</property>
<property>
	<name>dfs.secondary.namenode.keytab.file</name>
	<value>/etc/security/keytabs/sn.service.keytab</value>
</property>
<property>
	<name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
	<value>HTTP/_HOST@REALM.CA</value>
</property>
<property>
	<name>dfs.webhdfs.enabled</name>
	<value>true</value>
</property>

Remove the following properties

dfs.namenode.http-address
dfs.namenode.secondary.http-address
dfs.namenode.http-bind-host

yarn-site.xml

Add the following properties

nano /usr/local/hadoop/etc/hadoop/yarn-site.xml
 
<property>
	<name>yarn.http.policy</name>
	<value>HTTPS_ONLY</value>
</property>
<property>
	<name>yarn.resourcemanager.webapp.https.address</name>
	<value>${yarn.resourcemanager.hostname}:8090</value>
</property>
<property>
	<name>yarn.resourcemanager.hostname</name>
	<value>NAMENODE</value>
</property>
<property>
	<name>yarn.nodemanager.bind-host</name>
	<value>0.0.0.0</value>
</property>
<property>
	<name>yarn.nodemanager.webapp.address</name>
	<value>${yarn.nodemanager.hostname}:8042</value>
</property>
<property>
	<name>yarn.resourcemanager.principal</name>
	<value>rm/_HOST@REALM.CA</value>
</property>
<property>
	<name>yarn.resourcemanager.keytab</name>
	<value>/etc/security/keytabs/rm.service.keytab</value>
</property>
<property>
	<name>yarn.nodemanager.principal</name>
	<value>nm/_HOST@REALM.CA</value>
</property>
<property>
	<name>yarn.nodemanager.keytab</name>
	<value>/etc/security/keytabs/nm.service.keytab</value>
</property>
<property>
	<name>yarn.nodemanager.hostname</name>
	<value>NAMENODE</value>
</property>
<property>
	<name>yarn.resourcemanager.bind-host</name>
	<value>0.0.0.0</value>
</property>
<property>
	<name>yarn.timeline-service.bind-host</name>
	<value>0.0.0.0</value>
</property>

Remove the following properties

yarn.resourcemanager.webapp.address

SSL

Setup SSL Directories

sudo mkdir -p /etc/security/serverKeys
sudo chown -R root:hadoopuser /etc/security/serverKeys/
sudo chmod 755 /etc/security/serverKeys/
 
cd /etc/security/serverKeys

Setup Keystore

sudo keytool -genkey -alias NAMENODE -keyalg RSA -keysize 1024 -dname "CN=NAMENODE,OU=ORGANIZATION_UNIT,C=canada" -keypass PASSWORD -keystore /etc/security/serverKeys/keystore.jks -storepass PASSWORD
sudo keytool -export -alias NAMENODE -keystore /etc/security/serverKeys/keystore.jks -rfc -file /etc/security/serverKeys/NAMENODE.csr -storepass PASSWORD

Setup Truststore

sudo keytool -import -noprompt -alias NAMENODE -file /etc/security/serverKeys/NAMENODE.csr -keystore /etc/security/serverKeys/truststore.jks -storepass PASSWORD

Generate Self Signed Certifcate

sudo openssl genrsa -out /etc/security/serverKeys/NAMENODE.key 2048
 
sudo openssl req -x509 -new -key /etc/security/serverKeys/NAMENODE.key -days 300 -out /etc/security/serverKeys/NAMENODE.pem
 
sudo keytool -keystore /etc/security/serverKeys/keystore.jks -alias NAMENODE -certreq -file /etc/security/serverKeys/NAMENODE.cert -storepass PASSWORD -keypass PASSWORD
 
sudo openssl x509 -req -CA /etc/security/serverKeys/NAMENODE.pem -CAkey /etc/security/serverKeys/NAMENODE.key -in /etc/security/serverKeys/NAMENODE.cert -out /etc/security/serverKeys/NAMENODE.signed -days 300 -CAcreateserial

Setup File Permissions

sudo chmod 440 /etc/security/serverKeys/*
sudo chown root:hadoopuser /etc/security/serverKeys/*

Start the Cluster

start-dfs.sh
start-yarn.sh
mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver

Create User Directory

kinit -kt /etc/security/keytabs/myuser.keytab myuser/hadoop@REALM.CA
#ensure the login worked
klist
 
#Create hdfs directory now
hdfs dfs -mkdir /user
hdfs dfs -mkdir /user/myuser
 
#remove kerberos ticket
kdestroy

URL

https://NAMENODE:50470
https://NAMENODE:50475
https://NAMENODE:8090

References

https://www.ibm.com/support/knowledgecenter/en/SSPT3X_4.2.0/com.ibm.swg.im.infosphere.biginsights.admin.doc/doc/admin_ssl_hbase_mr_yarn_hdfs_web.html

Kerberos Server Installation

In this tutorial I will show you how to install Kerberos server on Ubuntu 16.04.

sudo apt install krb5-kdc krb5-admin-server krb5-config -y

Enter your realm. I will use REALM.CA

Enter your servers. I will use localhost

Enter your administrative server. I will use localhost

Now you can click Ok and installation will continue.

Next we can create our new realm

sudo krb5_newrealm

Enter your password then confirm it.

Now we can edit our kadm5.acl to have admin. Uncomment “*/admin *”

sudo nano /etc/krb5kdc/kadm5.acl

Now we make our keytabs directory and grant the necessary permissions.

sudo mkdir -p /etc/security/keytabs/
sudo chown root:hduser /etc/security/keytabs
sudo chmod 750 /etc/security/keytabs

Now we edit our krb5.conf file

sudo nano /etc/krb5.conf

Ensure it looks like the below

[libdefaults]
        default_realm = REALM.CA
 
 
[realms]
        REALM.CA = {
                kdc = localhost
                admin_server = localhost
        }
 
 
[domain_realm]
        .realm.ca = REALM.CA
        realm.ca = REALM.CA

Now we can restart the kerberos services

sudo service krb5-kdc restart; service krb5-admin-server restart

Once you create a principal if when you attempt to use kadmin you get the error “GSS-API (or Kerberos) error while initializing kadmin interface”. Then do the following.

sudo RUNLEVEL=1 apt-get install rng-tools
cat /dev/random | rngtest -c 1000
sudo apt-get install haveged
cat /proc/sys/kernel/random/entropy_avail
cat /dev/random | rngtest -c 1000
haveged -n 2g -f - | dd of=/dev/null

Uninstallation

sudo apt remove --purge krb5-kdc krb5-admin-server krb5-config -y
sudo rm -rf /var/lib/krb5kdc

References
I used the following references as a guide.

http://blog.ruanbekker.com/blog/2017/10/18/setup-kerberos-server-and-client-on-ubuntu/
http://csetutorials.com/setup-kerberos-ubuntu.html

Dropwizard: Swagger Integration

This entry is part 5 of 5 in the series Dropwizard

In this tutorial I will show you how to use Swagger in your Maven application. I will also show you how to configure it with Swagger UI so when you start your application you can see the Swagger UI from your generated JSON.

POM.xml

Dependencies

<dependency>
	<groupId>io.dropwizard</groupId>
	<artifactId>dropwizard-assets</artifactId>
	<version>1.3.2</version>
</dependency>
 
<dependency>
	<groupId>io.swagger</groupId>
	<artifactId>swagger-jaxrs</artifactId>
	<version>1.5.19</version>
</dependency>

Plugins

maven-jar-plugin

If you followed creating a basic Dropwizard app then you should have this already installed. If so then just add the following two configs under “manifest” section.

<addDefaultImplementationEntries>true</addDefaultImplementationEntries>
<addDefaultSpecificationEntries>true</addDefaultSpecificationEntries>

maven-clean-plugin

Because we are pulling the latest Swagger-UI code on each build we must clean the old build.

<plugin>
	<artifactId>maven-clean-plugin</artifactId>
	<version>3.1.0</version>
	<configuration>
		<filesets>
			<fileset>
				<directory>${basedir}/src/main/resources/swagger-ui</directory>
				<followSymlinks>false</followSymlinks>
			</fileset>
		</filesets>
	</configuration>
</plugin>

download-maven-plugin

We are downloading the latest Swagger-UI code from github. Notice how lifecycle phase “generate-resources” is used. This is important due to build getting the proper code before beginning build.

<plugin>
	<groupId>com.googlecode.maven-download-plugin</groupId>
	<artifactId>download-maven-plugin</artifactId>
	<version>1.4.0</version>
	<executions>
		<execution>
			<id>swagger-ui</id>
			<phase>generate-resources</phase>
			<goals>
				<goal>wget</goal>
			</goals>
			<configuration>
				<url>
					https://github.com/swagger-api/swagger-ui/archive/master.tar.gz
				</url>
				<unpack>true</unpack>
				<outputDirectory>
					${project.build.directory}
				</outputDirectory>
			</configuration>
		</execution>
	</executions>
</plugin>

replacer

This updates the code downloaded from github to have your swagger.json content instead of the petstore swagger content. Notice how lifecycle phase “generate-resources” is used. This is important due to build getting the proper code before beginning build.

<plugin>
	<groupId>com.google.code.maven-replacer-plugin</groupId>
	<artifactId>replacer</artifactId>
	<version>1.5.3</version>
	<executions>
		<execution>
			<phase>generate-resources</phase>
			<goals>
				<goal>replace</goal>
			</goals>
		</execution>
	</executions>
	<configuration>
		<includes>
			<include>${project.build.directory}/swagger-ui-master/dist/index.html</include>
			<include>${project.build.directory}/swagger-ui-master/dist/swagger-ui-bundle.js</include>
			<include>${project.build.directory}/swagger-ui-master/dist/swagger-ui-bundle.js.map</include>
			<include>${project.build.directory}/swagger-ui-master/dist/swagger-ui-standalone-preset.js</include>
			<include>${project.build.directory}/swagger-ui-master/dist/swagger-ui-standalone-preset.js.map</include>
			<include>${project.build.directory}/swagger-ui-master/dist/swagger-ui.js</include>
			<include>${project.build.directory}/swagger-ui-master/dist/swagger-ui.js.map</include>
		</includes>
		<replacements>
			<replacement>
				<token>http://petstore.swagger.io/v2/swagger.json</token>
				<value>/swagger.json</value>
			</replacement>
		</replacements>
	</configuration>
</plugin>

maven-resources-plugin

This will copy the content that you just downloaded and modified into your resources folder. Notice how lifecycle phase “generate-resources” is used. This is important due to build getting the proper code before beginning build.

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-resources-plugin</artifactId>
	<version>3.1.0</version>
	<executions>
		<execution>
			<id>copy-resources</id>
			<phase>generate-resources</phase>
			<goals>
				<goal>copy-resources</goal>
			</goals>
			<configuration>
				<outputDirectory>
					${basedir}/src/main/resources/swagger-ui
				</outputDirectory>
				<resources>
					<resource>
						<directory>
							${project.build.directory}/swagger-ui-master/dist
						</directory>
					</resource>
				</resources>
			</configuration>
		</execution>
	</executions>
</plugin>

Now if you run the following command you will see that the swagger-ui copied to your resources folder.

mvn clean install

MyDropwizardAppApplication

initialize

Now we need to configure our Dropwizard app to host the swagger-ui that we recently downloaded and modified. In our “MyDropwizardAppApplication” class that we created in the initial Dropwizard tutorial we must add the AssetsBundle for our swagger-ui.

@Override
public void initialize(final Bootstrap bootstrap) {
	bootstrap.addBundle(GuiceBundle.builder().enableAutoConfig(this.getClass().getPackage().getName())
			.modules(new ServerModule()).build());
 
	// This allows you to host swagger ui on this dropwizard app's host
	final AssetsBundle assetsBundle = new AssetsBundle("/swagger-ui", "/swagger-ui", "index.html");
	bootstrap.addBundle(assetsBundle);
	bootstrap.addCommand(new MyCommand());
}

run

Now we need to setup our Swagger scanners for our api and our models.

@Override
public void run(final MyDropwizardAppConfiguration configuration, final Environment environment) {
	this.initSwagger(configuration, environment);
}
 
private void initSwagger(MyDropwizardAppConfiguration configuration, Environment environment) {
	// Swagger Resource
	// The ApiListingResource creates the swagger.json file at localhost:8080/swagger.json
	environment.jersey().register(new ApiListingResource());
	environment.jersey().register(SwaggerSerializers.class);
 
	Package objPackage = this.getClass().getPackage();
	String version = objPackage.getImplementationVersion();
 
	// Swagger Scanner, which finds all the resources for @Api Annotations
	ScannerFactory.setScanner(new DefaultJaxrsScanner());
 
	//This is what is shown when you do "http://localhost:8080/swagger-ui/"
	BeanConfig beanConfig = new BeanConfig();
	beanConfig.setVersion(version);
	beanConfig.setSchemes(new String[] { "http" });
	beanConfig.setHost("localhost:8080");
	beanConfig.setPrettyPrint(true);
	beanConfig.setDescription("The drpowizard apis");
	beanConfig.setResourcePackage("ca.gaudreault.mydropwizardapp");
	beanConfig.setScan(true);
}

Now if we were to run our app we would be able to go to http://localhost:8080/swagger-ui/ and we would see our content but since we didn’t update any model or api then we wouldn’t see much of anything. So remember the previous tutorials on Dropwizard Guice and Dropwizard Resource. We will update those now.

Model

If you compare this to the one we did in the guice tutorial there are only a few differences. Notice we import the swagger annotations. We then add “ApiModel” annotation to the class and “ApiModelProperty” to the variable “value” and set it to be “NotNull”.

package ca.gaudreault.mydropwizardapp.models;
 
import java.io.Serializable;
 
import javax.validation.constraints.NotNull;
 
import io.swagger.annotations.ApiModel;
import io.swagger.annotations.ApiModelProperty;
 
@ApiModel(description = "My Example Model.")
public class MyModel implements Serializable {
	private static final long serialVersionUID = 1L;
	@NotNull
	@ApiModelProperty(required = true, notes = "My value")
	private Integer value;
	
	public Integer getValue() {
		return value;
	}
	public void setValue(Integer value) {
		this.value = value;
	}
}

Resource

If you compare this to the one we did in the guice tutorial there are only a few differences. Notice our class has “@SwaggerDefinition” and “@API” defined. This will help the Swagger-UI group your end points together using the tags. Also notice how our “runTest” end point has “@Path”, “@ApiResponses” and “@ApiOperation” now.

package ca.gaudreault.mydropwizardapp.resources;
 
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
 
import org.eclipse.jetty.http.HttpStatus;
 
import com.codahale.metrics.annotation.Timed;
import com.google.inject.Inject;
 
import ca.gaudreault.mydropwizardapp.models.MyModel;
import ca.gaudreault.mydropwizardapp.services.MyService;
import io.swagger.annotations.Api;
import io.swagger.annotations.ApiOperation;
import io.swagger.annotations.ApiResponses;
import io.swagger.annotations.ApiResponse;
import io.swagger.annotations.SwaggerDefinition;
import io.swagger.annotations.Tag;
 
@SwaggerDefinition(tags = { @Tag(name = "MyResource", description = "My Example Resource") })
@Api(value = "MyResource")
@Timed
@Path("/my-resource")
public class MyResource {
	private MyService myService;
 
	@Inject
	public MyResource(final MyService myService) {
		this.myService = myService;
	}
 
	@GET
	@Path("/runTest")
	@ApiOperation(value = "Run test and returns myModel", notes = "Run test and returns myModel", response = MyModel.class, tags = {
			"MyResource" })
	@ApiResponses(value = {
			@ApiResponse(code = HttpStatus.OK_200, message = "Successfully Tested", response = MyModel.class) })
	@Timed
	@Produces(MediaType.APPLICATION_JSON)
	public MyModel runTest() {
		return this.myService.runTest();
	}
}

Run our Project

If we run our project and we hit the following rest end point http://localhost:8080/my-resource/runTest we will get back the below. This shows us our rest end point is working as expected still.

{"value":123123}

Checking Swagger-UI

Now that we have started our project we can now check to see what was generated. Go to Swagger-UI. You will see the below. You are now well on your way in using Swagger.

Model Expanded

Resource Expanded

References

The following helped me build this tutorial.

https://robferguson.org/blog/2016/12/11/resteasy-embedded-jetty-fat-jars-swagger-and-swagger-ui/
https://itazuramono.com/2015/12/07/automatic-swagger-documentation-for-dropwizard-using-maven/
http://mikelynchgames.com/software-development/adding-swagger-to-your-dropwizard-application/

Java: JUnit 4 /w PowerMock

In this tutorial I will show you how to use JUnit 4 with PowerMock for mocking Static classes into your application. If you have not already done so follow JUnit 4 tutorial.

POM.xml

<dependency>
	<groupId>org.mockito</groupId>
	<artifactId>mockito-core</artifactId>
	<version>2.18.3</version>
	<scope>test</scope>
</dependency>
<dependency>
	<groupId>org.assertj</groupId>
	<artifactId>assertj-core</artifactId>
	<version>3.10.0</version>
	<scope>test</scope>
</dependency>

Static Class

We will create this class to use for our static testing.

public final class MyStaticTest {
	public static String getString() {
		return "test";
	}
}

Imports

import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.Mockito.when;
 
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.MockitoAnnotations;
import org.powermock.api.mockito.PowerMockito;
import org.powermock.core.classloader.annotations.PrepareForTest;
import org.powermock.modules.junit4.PowerMockRunner;

Test Class

Now we can run our test with PowerMock and mock our static classes methods as you can see from the below.

@RunWith(PowerMockRunner.class)
@PrepareForTest({ MyStaticTest.class })
public class AppTestStatic {
	@Before
	public void setup() {
		MockitoAnnotations.initMocks(this);
		PowerMockito.mockStatic(MyStaticTest.class);
	}
 
	@Test
	public void myTest() {
		when(MyStaticTest.getString()).thenReturn("myTest");
 
		final String returnString = MyStaticTest.getString();
 
		assertThat(returnString).isEqualTo("myTest");
	}
}

Java: JUnit 4 Example

In this tutorial I will show you how to use JUnit 4 into your application. The next tutorial I will use this class with PowerMock.

Build Path

Ensure you have added JUnit 4 to your build path and that you are using Java 8.

Imports

import org.junit.Before;
import org.junit.Test;

Test Class

package ca.gaudreault.mytestapp;
 
import org.junit.Before;
import org.junit.Test;
 
public class AppTest {
	@Before
	public void setup() {
	}
 
	@Test
	public void myTest() {
	}
}

This was a very basic example of writing a unit test with JUnit 4. In future example I will build on this with PowerMock.

Dropwizard: Resource

This entry is part 4 of 5 in the series Dropwizard

In this tutorial I will give a basic example of a resource endpoint. If you haven’t configured Guice yet please do so before continuing.

So basically now that you have Guice configured and working you can now create an api endpoint. For this we will just use a GET but you can also do POST, PUT, DELETE.

package ca.gaudreault.mydropwizardapp.resources;
 
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
 
import com.codahale.metrics.annotation.Timed;
import com.google.inject.Inject;
 
import ca.gaudraeult.mydropwizardapp.services.MyService;
import ca.gaudreault.mydropwizardapp.models.MyModel;
 
@Timed
@Path("/my-resource")
public class MyResource {
	MyService myService;
 
	@Inject
	public MyResource(final MyService myService) {
		this.myService = myService;
	}
 
	@GET
	@Timed
	@Produces(MediaType.APPLICATION_JSON)
	public MyModel runTest() {
		return this.myService.runTest();
	}
}

Once you run your application you can view the endpoint by going to http://localhost:8080/my-resource.

The output will be as follows.

{"value":123123}

If you noticed we added the “@Timed” annotation. You can now go to http://localhost:8081/metrics?pretty=true to view the metrics on our “runTest” method. The output will look like the below.

{
	"ca.gaudreault.mydropwizardapp.resources.MyResource.runTest": {
		"count": 0,
		"max": 0.0,
		"mean": 0.0,
		"min": 0.0,
		"p50": 0.0,
		"p75": 0.0,
		"p95": 0.0,
		"p98": 0.0,
		"p99": 0.0,
		"p999": 0.0,
		"stddev": 0.0,
		"m15_rate": 0.0,
		"m1_rate": 0.0,
		"m5_rate": 0.0,
		"mean_rate": 0.0,
		"duration_units": "seconds",
		"rate_units": "calls/second"
}

Dropwizard: Command

This entry is part 3 of 5 in the series Dropwizard

In this tutorial I will give a brief demonstration on how to write a custom dropwizard command.

MyCommand

So below you will see the command class and how we are creating and registering a command line param called “test” which is a Boolean.

package ca.gaudreault.mydropwizardapp;
 
import io.dropwizard.cli.Command;
import io.dropwizard.setup.Bootstrap;
import net.sourceforge.argparse4j.inf.Namespace;
import net.sourceforge.argparse4j.inf.Subparser;
 
public class MyCommand extends Command {
 
	protected MyCommand() {
		super("myCommand", "This is a sample command");
	}
 
	@Override
	public void configure(Subparser subparser) {
	    subparser.addArgument("-test").required(true).type(Boolean.class).dest("test").help("Does something really awesome");
	}
 
	@Override
	public void run(Bootstrap<?> bootstrap, Namespace namespace) throws Exception {
		System.out.println("MyCommand " + namespace.getBoolean("test"));
	}
}

MyDropwizardAppApplication

If you remember from part 1 of this series you created the based Dropwizard app. So you should have a class called “MyDropwizardAppApplication”. Open that now and modify the “initialize” like the below. Note that we are only adding the “addCommand”.

@Override
public void initialize(final Bootstrap bootstrap) {
	bootstrap.addCommand(new MyCommand());
}

Executing Command

Basically now we can just call our JAR file and pass the following arguments to it.

myCommand -test false

You will see once it runs that following

MyCommand false

Dropwizard: Guice Bundle

This entry is part 2 of 5 in the series Dropwizard

In this tutorial I will show you how to add Guice to your Dropwizard app. This will be a very basic implementation. Some things you should note is that I didn’t put in any docstrings. You should always do that!

Now there are a few Dropwizard Guice integrations available but the most active is the one I will show you today called “dropwizard-guicey“.

POM.xml

<dependency>
	<groupId>ru.vyarus</groupId>
	<artifactId>dropwizard-guicey</artifactId>
	<version>4.1.0</version>
</dependency>

Model

Now we create a model to use with our service

package ca.gaudreault.mydropwizardapp.models;
 
import java.io.Serializable;
 
import javax.validation.constraints.NotNull;
 
public class MyModel implements Serializable {
	private static final long serialVersionUID = 1L;
	private Integer value;
	
	public Integer getValue() {
		return value;
	}
	public void setValue(Integer value) {
		this.value = value;
	}
}

Service

Here you will create your service interface and class so that you can bind it in the guice module.

Interface

package ca.gaudraeult.mydropwizardapp.services;
 
import ca.gaudreault.mydropwizardapp.models.MyModel;
 
public interface MyService {
	MyModel runTest();
}

Implementation

package ca.gaudraeult.mydropwizardapp.services;
 
import ca.gaudreault.mydropwizardapp.models.MyModel;
 
public class MyServiceImpl implements MyService {
	
	public MyServiceImpl() { }
 
	@Override
	public MyModel runTest() {
		final MyModel myModel = new MyModel();
		myModel.setValue(123123);
		return myModel;
	}
}

ServerModule

Now when we create our module class you can bind the interface to the implementation. Note that if your implementation does not implement the interface this will not work.

package ca.gaudreault.mydropwizardapp;
 
import com.google.inject.AbstractModule;
 
import ca.gaudraeult.mydropwizardapp.services.MyService;
import ca.gaudraeult.mydropwizardapp.services.MyServiceImpl;
 
public class ServerModule extends AbstractModule  {
 
	@Override
	protected void configure() {
		bind(MyService.class).to(MyServiceImpl.class);
	}
}

Dropwizard Application

@Override
public void initialize(final Bootstrap bootstrap) {
	bootstrap.addBundle(GuiceBundle.builder()
		.enableAutoConfig(this.getClass().getPackage().getName())
		.modules(new ServerModule())
		.build());
}

And that is it you have configured a very basic Dropwizard Guice configuration.

ElasticSearch: High Level Client Search Scrolling

This entry is part 4 of 4 in the series ElasticSearch High Level Rest Client

In this tutorial I will show you how to perform a search scroll using the high level client. If you have not already done so please follow the search tutorial.

The reason you following the search tutorial first is that sets up the search. So you just have to do a few more steps.

Imports:

import org.elasticsearch.action.search.SearchScrollRequest;
import org.elasticsearch.common.unit.TimeValue;

Modify the “SearchRequest”. A recommended timeout is 60000 or 1m.

request.scroll(new TimeValue(60000));

Once you perform the initial search now you will get a “scrollId”. Use that to generate your new “SearchScrollRequest” using that scrollId. One thing to note is the “scrollRequest” timeout value. Set this or it may not work.

final SearchScrollRequest searchScrollRequest = new SearchScrollRequest(scrollId);
searchScrollRequest.scroll(new TimeValue(60000));

Now the searchResponse that we used initially we can repurpose to continue scrolling the results.

searchResponse = client.searchScroll(searchScrollRequest);

We know that their are no more results when the scrollId is null or when getHits length is 0.

searchResponse.getHits().getHits().length > 0

ElasticSearch: High Level Client Search

This entry is part 3 of 4 in the series ElasticSearch High Level Rest Client

In this tutorial I will show you how to perform a search using the high level client. If you have not already done so please connect to ElasticSearch.

Imports

import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.search.SearchHits;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.action.search.SearchType;

Now we can perform the search.

final SearchRequest request = new SearchRequest();
request.searchType(SearchType.QUERY_THEN_FETCH);
 
final String[] types = { "doc" };
final String[] indexes = { "index" };
 
//Specify the types that your search applies to.
//Note that this is not needed. If ommitted it will search all.
request.types(types);
 
//Specify the indexes that your search applies to.
//Note that this is not needed. If ommitted it will search all.
request.indices(indexes);
 
final SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
//You can add any type of query into this query. Adjust to what you need.
searchSourceBuilder.query(MyQuery);
request.source(searchSourceBuilder);
 
final SearchResponse searchResponse = client.search(request);
 
//This will let us know if the search was terminated early.
final Boolean terminatedEarly = searchResponse.isTerminatedEarly();
//This will let us know if it timed out.
final boolean timedOut = searchResponse.isTimedOut();
 
//Now to loop through our hits to do what we need to
final SearchHits searchHits = searchResponse.getHits();
for (final SearchHit hit : searchHits) {
  //Do work
}

ElasticSearch: High Level Client Post

This entry is part 2 of 4 in the series ElasticSearch High Level Rest Client

In this tutorial I will show you how to perform a POST request. If you have not connected first please do so before continuing.

Imports

import org.apache.http.HttpEntity;
import org.apache.http.nio.entity.NStringEntity;
import org.apache.http.entity.ContentType;
import org.apache.http.util.EntityUtils;
import org.elasticsearch.client.Response;

Now we can perform the POST to ElasticSearch.

final Integer id = 1;
final String document = "{\"key\": 1 }";
final HttpEntity httpEntity = new NStringEntity(document, ContentType.APPLICATION_JSON);
 
final Response response = restHighLevelClient.getLowLevelClient().performRequest("POST", "/indexName/indexType/" + id, Collections.<String, String>emptyMap(), httpEntity);
 
//Now you can print the response
System.out.println(EntityUtils.toString(response.getEntity()));

ElasticSearch: Low Level Client Get

This entry is part 3 of 3 in the series ElasticSearch Low Level Rest Client

In this tutorial I will show you how to put a json document into ElasticSearch. If you have not first connected to ElasticSearch please do so before continuing.

POM.xml

<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.9.5</version>
</dependency>

Imports

import org.apache.http.HttpEntity;
import org.apache.http.nio.entity.NStringEntity;
import org.apache.http.entity.ContentType;
import org.elasticsearch.client.Response;
import org.apache.http.util.EntityUtils;

Now perform the GET request using the low level client.

ObjectMapper objectMapper = new ObjectMapper();
final String document = "{\"key\": 1 }";
final JsonNode document = objectMapper.readTree("{" +
   " \"query\": {" +
   " \"match\" : {" +
   " \"key\" : 1 }}}");
final HttpEntity httpEntity = new NStringEntity(document.toString(), ContentType.APPLICATION_JSON);
final Response response = restClient.performRequest("GET", "/indexName/indexType/_search", Collections.<String, String>emptyMap(), httpEntity);
 
//Now you can print the response
System.out.println(EntityUtils.toString(response.getEntity()));
 
//OR get the content
final JsonNode content = objectMapper.readTree(response.getEntity().getContent());
System.out.println(content);

ElasticSearch: Low Level Client Put

This entry is part 2 of 3 in the series ElasticSearch Low Level Rest Client

In this tutorial I will show you how to put a json document into ElasticSearch. If you have not first connected to ElasticSearch please do so before continuing.

Imports

import org.apache.http.HttpEntity;
import org.apache.http.nio.entity.NStringEntity;
import org.elasticsearch.client.Response;
import org.apache.http.entity.ContentType;
import org.apache.http.util.EntityUtils;

Now perform the PUT request using the low level client.

final String document = "{\"key\": 1 }";
final HttpEntity httpEntity = new NStringEntity(document, ContentType.APPLICATION_JSON);
final Integer id = 1;
final Response response = restClient.performRequest("PUT", "/indexName/indexType/" + id, Collections.<String, String>emptyMap(), httpEntity);
 
//Now you can print the response
System.out.println(EntityUtils.toString(response.getEntity()));

ElasticSearch: High Level Rest Client Connection

This entry is part 1 of 4 in the series ElasticSearch High Level Rest Client

In this tutorial I will show you how to use the ElasticSearch high level rest client.

First you will need to add the low level rest to the pom.

<properties>
	<elasticSearch.version>6.2.4</elasticSearch.version>
</properties>
 
<dependency>
    <groupId>org.elasticsearch.client</groupId>
    <artifactId>elasticsearch-rest-high-level-client</artifactId>
    <version>${elasticSearch.version}</version>
</dependency>

Next you will need to specify the imports.

import java.util.List;
import java.util.ArrayList;
import java.util.Arrays;
import org.apache.http.HttpHost;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.elasticsearch.client.RestHighLevelClient;

Now you can connect to ElasticSearch.

final List hosts = new ArrayList<>(Arrays.asList("localhost"));
final Integer port = 9200;
final String scheme = "http";
		
final HttpHost[] httpHosts = hosts.stream().map(host -> new HttpHost(host, port, scheme)).toArray(HttpHost[]::new);
 
final RestClientBuilder restClientBuilder = RestClient.builder(httpHosts);
final RestHighLevelClient restHighLevelClient = new RestHighLevelClient(restClientBuilder);

Now you can do whatever you need to!

ElasticSearch: Low Level Rest Client Connection

This entry is part 1 of 3 in the series ElasticSearch Low Level Rest Client

In this tutorial I will show you how to use the ElasticSearch low level rest client.

First you will need to add the low level rest to the pom.

<properties>
	<elasticSearch.version>6.2.4</elasticSearch.version>
</properties>
 
<dependency>
    <groupId>org.elasticsearch.client</groupId>
    <artifactId>elasticsearch-rest-client</artifactId>
    <version>${elasticSearch.version}</version>
</dependency>

Next you will need to specify the imports.

import org.apache.http.HttpHost;
import org.elasticsearch.client.Response;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;

Now you can connect to ElasticSearch.

final RestClientBuilder restClientBuilder = RestClient.builder(new HttpHost("localhost", 9200, "http"));
 
final RestClient restClient = builder.build();

Now you can do whatever you need to!