Hive & Java: Connect to Remote Kerberos Hive using KeyTab

In this tutorial I will show you how to connect to remote Kerberos Hive cluster using Java. If you haven’t install Hive yet follow the tutorial.

Import SSL Cert to Java:

Follow this tutorial to “Installing unlimited strength encryption Java libraries

If on Windows do the following

#Import it
"C:\Program Files\Java\jdk1.8.0_171\bin\keytool" -import -file hadoop.csr -keystore "C:\Program Files\Java\jdk1.8.0_171\jre\lib\security\cacerts" -alias "hadoop"

#Check it
"C:\Program Files\Java\jdk1.8.0_171\bin\keytool" -list -v -keystore "C:\Program Files\Java\jdk1.8.0_171\jre\lib\security\cacerts"

#If you want to delete it
"C:\Program Files\Java\jdk1.8.0_171\bin\keytool" -delete -alias hadoop -keystore "C:\Program Files\Java\jdk1.8.0_171\jre\lib\security\cacerts"

POM.xml:

<dependency>
	<groupId>org.apache.hive</groupId>
	<artifactId>hive-jdbc</artifactId>
	<version>2.3.3</version>
	<exclusions>
		<exclusion>
			<groupId>jdk.tools</groupId>
			<artifactId>jdk.tools</artifactId>
		</exclusion>
	</exclusions>
</dependency>

Imports:

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.security.UserGroupInformation;
import java.sql.SQLException;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.Statement;
import java.sql.DriverManager;

Connect:

// Setup the configuration object.
final Configuration config = new Configuration();

config.set("fs.defaultFS", "swebhdfs://hadoop:50470");
config.set("hadoop.security.authentication", "kerberos");
config.set("hadoop.rpc.protection", "integrity");

System.setProperty("https.protocols", "TLSv1,TLSv1.1,TLSv1.2");
System.setProperty("java.security.krb5.conf", "C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\krb5.conf");
System.setProperty("java.security.krb5.realm", "REALM.CA");
System.setProperty("java.security.krb5.kdc", "REALM.CA");
System.setProperty("sun.security.krb5.debug", "true");
System.setProperty("javax.net.debug", "all");
System.setProperty("javax.net.ssl.keyStorePassword","changeit");
System.setProperty("javax.net.ssl.keyStore","C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\cacerts");
System.setProperty("javax.net.ssl.trustStore", "C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\cacerts");
System.setProperty("javax.net.ssl.trustStorePassword","changeit");
System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");

UserGroupInformation.setConfiguration(config);
UserGroupInformation.setLoginUser(UserGroupInformation.loginUserFromKeytabAndReturnUGI("hive/hadoop@REALM.CA", "c:\\data\\hive.service.keytab"));

System.out.println(UserGroupInformation.getLoginUser());
System.out.println(UserGroupInformation.getCurrentUser());

//Add the hive driver
Class.forName("org.apache.hive.jdbc.HiveDriver");

//Connect to hive jdbc
Connection connection = DriverManager.getConnection("jdbc:hive2://hadoop:10000/default;principal=hive/hadoop@REALM.CA");
Statement statement = connection.createStatement();

//Create a table
String createTableSql = "CREATE TABLE IF NOT EXISTS "
		+" employee ( eid int, name String, "
		+" salary String, designation String)"
		+" COMMENT 'Employee details'"
		+" ROW FORMAT DELIMITED"
		+" FIELDS TERMINATED BY '\t'"
		+" LINES TERMINATED BY '\n'"
		+" STORED AS TEXTFILE";

System.out.println("Creating Table: " + createTableSql);
statement.executeUpdate(createTableSql);

//Show all the tables to ensure we successfully added the table
String showTablesSql = "show tables";
System.out.println("Show All Tables: " + showTablesSql);
ResultSet res = statement.executeQuery(showTablesSql);

while (res.next()) {
	System.out.println(res.getString(1));
}

//Drop the table
String dropTablesSql = "DROP TABLE IF EXISTS employee";

System.out.println("Dropping Table: " + dropTablesSql);
statement.executeUpdate(dropTablesSql);

System.out.println("Finish!");

Eclipse/Maven: Jacoco Integration

This tutorial will guide you through configuring Jacoco in your Maven application and install the Eclipse plugin.

First Open Eclipse MarketPlace then search for “EclEmma”.

Next you need to click Install and accept the license agreement reading it first. Then it will complete and need to restart Eclipse.

Once Eclipse opens again you can edit “Code Coverage” from “Window/Preferences”.

You can now run “Code Coverage” through Eclipse by right clicking your project. As you can see below I have not written any unit tests yet :(.

 

Pom.xml

Build

<build>
	<plugins>
		<plugin>
			<groupId>org.jacoco</groupId>
			<artifactId>jacoco-maven-plugin</artifactId>
			<version>0.8.1</version>
			<configuration>
				<!-- Path to the output file for execution data. (Used in initialize 
					phase) -->
				<destFile>${project.build.directory}/target/coverage-reports/jacoco-unit.exec</destFile>
				<!-- File with execution data. (Used in package phase) -->
				<dataFile>${project.build.directory}/target/coverage-reports/jacoco-unit.exec</dataFile>
				<excludes>
				</excludes>
			</configuration>
			<executions>
				<execution>
					<id>jacoco-initialization</id>
					<phase>initialize</phase>
					<goals>
						<!-- https://www.eclemma.org/jacoco/trunk/doc/prepare-agent-mojo.html -->
						<goal>prepare-agent</goal>
					</goals>
				</execution>
				<execution>
					<id>jacoco-site</id>
					<phase>package</phase>
					<goals>
						<!-- https://www.eclemma.org/jacoco/trunk/doc/report-mojo.html -->
						<goal>report</goal>
					</goals>
				</execution>
			</executions>
		</plugin>
	</plugins>
</build>

 

 

 

Eclipse/Maven: FindBugs/SpotBugs Integration

This tutorial will guide you through configuration FindBugs/SpotBugs in your Maven application and install the Eclipse plugin.

First Open Eclipse MarketPlace then search for “SpotBugs”.

Next you need to click Install and accept the license agreement reading it first. Then it will complete and need to restart Eclipse.

Once Eclipse opens again you right click the project(s) you want to activate FindBugs/SpotBugs for and click “Properties”. Click “SpotBugs” and then make the following changes.

Now you can run SpotBugs by right clicking your project and selecting SpotBugs then “Find Bugs”.

Pom.xml

Reporting

<reporting>
	<plugins>
		<plugin>
			<groupId>com.github.spotbugs</groupId>
			<artifactId>spotbugs-maven-plugin</artifactId>
			<version>3.1.3</version>
		</plugin>
	</plugins>
</reporting>

Build

<build>
	<plugins>
		<plugin>
			<groupId>com.github.spotbugs</groupId>
			<artifactId>spotbugs-maven-plugin</artifactId>
			<version>3.1.3</version>
			<dependencies>
				<dependency>
					<groupId>com.github.spotbugs</groupId>
					<artifactId>spotbugs</artifactId>
					<version>3.1.3</version>
				</dependency>
			</dependencies>
			<configuration>
				<effort>Max</effort>
				<threshold>Low</threshold>
				<failOnError>true</failOnError>
				<plugins>
					<plugin>
						<groupId>com.h3xstream.findsecbugs</groupId>
						<artifactId>findsecbugs-plugin</artifactId>
						<version>LATEST</version>
					</plugin>
				</plugins>
			</configuration>
		</plugin>
	</plugins>
</build>

Maven Commands

mvn spotbugs:spotbugs

#Generates the report site
mvn site

Eclipse/Maven: PMD Integration

This tutorial will guide you through configuring PMD in your Maven application and install the Eclipse plugin.

First Open Eclipse MarketPlace then search for “PMD”.

Next you need to click Install and accept the license agreement reading it first. Then it will complete and need to restart Eclipse.

Once Eclipse opens again you right click the project(s) you want to activate PMD for and click “Properties”. Click “PMD” and then click “Enable PMD for this project”. You will need to create a rule set. To do that go here.

Pom.xml

Reporting

You will need both reporting plugins in your project. “maven-jxr-plugin” fixes an issue with not finding the xRef.

<reporting>
	<plugins>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-pmd-plugin</artifactId>
			<version>3.9.0</version>
		</plugin>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-jxr-plugin</artifactId>
			<version>2.5</version>
		</plugin>
	</plugins>
</reporting>

Build

You will need to configure the following to use with “mvn pmd:???” commands.

<build>
	<plugins>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-pmd-plugin</artifactId>
			<version>3.9.0</version>
			<configuration>
				<failOnViolation>true</failOnViolation>
				<verbose>true</verbose>
				<targetJdk>1.8</targetJdk>
				<includeTests>false</includeTests>
				<excludes>
				</excludes>
				<excludeRoots>
					<excludeRoot>target/generated-sources/stubs</excludeRoot>
				</excludeRoots>
			</configuration>
			<executions>
				<execution>
					<phase>test</phase>
					<goals>
						<goal>pmd</goal>
						<goal>cpd</goal>
						<goal>cpd-check</goal>
						<goal>check</goal>
					</goals>
				</execution>
			</executions>
		</plugin>
	</plugins>
</build>

Maven Commands

mvn pmd:check
mvn pmd:pmd

#cdp checks for copy paste issues

mvn pmd:cdp-check
mvn pmd:cdp

#Generates the report site
mvn site

Eclipse/Maven: CheckStyle Integration

This tutorial will guide you through configuration CheckStyle in your Maven application and install the Eclipse plugin.

First Open Eclipse MarketPlace then search for “Checkstyle”.

Next you need to click Install and accept the license agreement reading it first. Then it will complete and need to restart Eclipse.

Once Eclipse opens again you right click the project(s) you want to activate CheckStyle for and activate it. There are also properties you can configure through Eclipse’s preferences. I suggest you go there and configure it. You can also customize your checkstyle or make your own. Up to you.

Pom.xml

Build

When you run “mvn checkstyle:check” if will then run and will fail the build if you have any issues.

<build>
	<plugins>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-checkstyle-plugin</artifactId>
			<version>3.0.0</version>
			<executions>
				<execution>
					<id>validate</id>
					<phase>validate</phase>
					<configuration>
						<encoding>UTF-8</encoding>
						<consoleOutput>true</consoleOutput>
						<failsOnError>true</failsOnError>
						<linkXRef>false</linkXRef>
					</configuration>
					<goals>
						<goal>check</goal>
					</goals>
				</execution>
			</executions>
		</plugin>
	</plugins>
</build>

Reporting

You can generate a HTML report with the following by running “mvn checkstyle:checkstyle”.

<reporting>
	<plugins>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-checkstyle-plugin</artifactId>
			<version>3.0.0</version>
			<reportSets>
				<reportSet>
					<reports>
						<report>checkstyle</report>
					</reports>
				</reportSet>
			</reportSets>
		</plugin>
	</plugins>
</reporting>

Java: Command Line Arguments Parsing

This tutorial will guide you through how to do command line argument parsing easily. For this example we will use Commons-Cli package.

pom.xml

<properties>
	<commonscli.version>1.4</commonscli.version>
</properties>

<dependencies>
	<!-- https://mvnrepository.com/artifact/commons-cli/commons-cli -->
	<dependency>
		<groupId>commons-cli</groupId>
		<artifactId>commons-cli</artifactId>
		<version>${commonscli.version}</version>
	</dependency>
</dependencies>

Imports

import org.apache.commons.cli.CommandLine;
import org.apache.commons.cli.CommandLineParser;
import org.apache.commons.cli.DefaultParser;
import org.apache.commons.cli.HelpFormatter;
import org.apache.commons.cli.Option;
import org.apache.commons.cli.Options;
import org.apache.commons.cli.ParseException;

Main

public static void main(String[] args) {
	final Options options = new Options();
	Option startOption = new Option("s", "start", true, "Start the process.");
	startOption.setRequired(true);
	options.addOption(startOption);

	final HelpFormatter help = new HelpFormatter();
	final CommandLineParser parser = new DefaultParser();
	CommandLine cmd = null;

	try {
		cmd = parser.parse(options, args);
	} catch (final ParseException e) {
		help.printHelp("java -jar myApp.jar", "My Header", options, "-s must be specified");
		return;
	}

	final boolean doStart = Boolean.valueOf(cmd.getOptionValue("s"));

	if (doStart) {
		//Do my work here
	}
}

Avro & Java: Record Parsing

This tutorial will guide you through how to convert json to avro and then back to json. I suggest you first read through the documentation on Avro to familiarize yourself with it. This tutorial assumes you have a maven project already setup and a resources folder.

POM:

Add Avro Dependency

 

 

 

 

Add Jackson Dependency

Avro Schema File:

Next you need to create the avro schema file in your resources folder. Name the file “schema.avsc”. The extension avsc is the Avro schema extension.

{
    "namespace": "test.avro",
    "type": "record",
    "name": "MY_NAME",
    "fields": [
        {"name": "name_1", "type": "int"},
        {"name": "name_2", "type": {"type": "array", "items": "float"}},
        {"name": "name_3", "type": "float"}
    ]
}

Json Record to Validate:

Next you need to create a json file that conforms to your schema you just made. Name the file “record.json” and put it in your resources folder. The contents can be whatever you want as long as it conforms to your schema above.

{ "name_1": 234, "name_2": [23.34,654.98], "name_3": 234.7}

It’s Avro Time:

Imports:

import java.io.ByteArrayOutputStream;
import java.io.DataInputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;

import org.apache.avro.Schema;
import org.apache.avro.generic.GenericData;
import org.apache.avro.generic.GenericDatumReader;
import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.io.DatumReader;
import org.apache.avro.io.Decoder;
import org.apache.avro.io.DecoderFactory;
import org.apache.avro.io.Encoder;
import org.apache.avro.io.EncoderFactory;

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;

Conversion to Avro and Back:

private void run() throws IOException {
	//Get the schema and json record from resources
	final ClassLoader loader = getClass().getClassLoader();
	final File schemaFile = new File(loader.getResource("schema.avsc").getFile());
	final InputStream record = loader.getResourceAsStream("record.json");
	
	//Create avro schema
	final Schema schema = new Schema.Parser().parse(schemaFile);

	//Encode to avro
	final byte[] avro = encodeToAvro(schema, record);

	//Decode back to json
	final JsonNode node = decodeToJson(schema, avro);

	System.out.println(node);
	System.out.println("done");
}

/**
 * Encode json to avro
 * 
 * @param schema the schema the avro pertains to
 * @param record the data to convert to avro
 * @return the avro bytes
 * @throws IOException if decoding fails
 */
private byte[] encodeToAvro(Schema schema, InputStream record) throws IOException {
	final DatumReader<GenericData.Record> reader = new GenericDatumReader<>(schema);
	final DataInputStream din = new DataInputStream(record);
	final Decoder decoder = new DecoderFactory().jsonDecoder(schema, din);
	final Object datum = reader.read(null, decoder);
	final GenericDatumWriter<Object> writer = new GenericDatumWriter<>(schema);
	final ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
	final Encoder encoder = new EncoderFactory().binaryEncoder(outputStream, null);
	writer.write(datum, encoder);
	encoder.flush();

	return outputStream.toByteArray();
}

/**
 * Decode avro back to json.
 * 
 * @param schema the schema the avro pertains to
 * @param avro the avro bytes
 * @return the json
 * @throws IOException if jackson fails
 */
private JsonNode decodeToJson(Schema schema, byte[] avro) throws IOException {
	final ObjectMapper mapper = new ObjectMapper();
	final DatumReader<GenericData.Record> reader = new GenericDatumReader<>(schema);
	final Decoder decoder = new DecoderFactory().binaryDecoder(avro, null);
	final JsonNode node = mapper.readTree(reader.read(null, decoder).toString());

	return node;
}

Java: Basic Dropwizard Project

This entry is part 1 of 5 in the series Dropwizard

This tutorial will guide you through how to create a bare bones Dropwizard app. Ensure you have Eclipse installed or whatever IDE you are deciding to use. You can use their documentation as a guide. This is the first tutorial in the dropwizard series.

Setup Eclipse Archetype:

Select new Maven Project then in the select Archetype if dropwizard isn’t there you can add it by using the below settings.

Filter for Dropwizard Archetype:

Set Project Settings:

POM:

Ensure the pom has the correct “dropwizard.version” and that the dependency “dropwizard-core” is there.

Our Configuration:

Now that the project is created let’s have a look at what our configuration looks like. Pretty basic but that’s all we need right now.

package ca.gaudreault.mydropwizardapp;

import io.dropwizard.Configuration;
import com.fasterxml.jackson.annotation.JsonProperty;
import org.hibernate.validator.constraints.*;
import javax.validation.constraints.*;

public class MyDropwizardAppConfiguration extends Configuration {
    
}

Our Application:

This is what our application class looks like. It’s empty nothing yet.

package ca.gaudreault.mydropwizardapp;

import io.dropwizard.Application;
import io.dropwizard.setup.Bootstrap;
import io.dropwizard.setup.Environment;

public class MyDropwizardAppApplication extends Application {

    public static void main(final String[] args) throws Exception {
        new MyDropwizardAppApplication().run(args);
    }

    @Override
    public String getName() {
        return "MyDropwizardApp";
    }

    @Override
    public void initialize(final Bootstrap bootstrap) {

    }

    @Override
    public void run(final MyDropwizardAppConfiguration configuration,
                    final Environment environment) {

    }
}

Config.yml:

This is what our config.yml file looks like at the start.

logging:
  level: INFO
  loggers:
    ca.gaudreault: DEBUG

Setup Debug Configuration:

Setup your debug configuration like the below setup.

Running:

Once you run it you will be running two sites.

  1. http://localhost:8080
    1. Your main site
  2. http://localhost:8081
    1. Your operational site (health, etc)

Java: String.format

To format a string with parameters use String.format. There are different ways to use it depending on boolean, number, etc.

Format with Boolean

String.format("Boolean: %b", true)

Format with Number

String.format("Number: %n", 32)

Format with String

String.format("String: %s", "Information")

AWS: Java Post to Kinesis Queue

This entry is part 4 of 5 in the series AWS & Java

Posting to an AWS Kinesis Queue is rather simple and straight forward. As always you should refer to AWS Documentation.

Put Multiple Records On Queue

Import the following

import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.kinesis.AmazonKinesis;
import com.amazonaws.services.kinesis.AmazonKinesisClientBuilder;
import com.amazonaws.services.kinesis.model.PutRecordsRequest;
import com.amazonaws.services.kinesis.model.PutRecordsRequestEntry;
import com.amazonaws.services.kinesis.model.Record;

Put Records

AmazonKinesisClientBuilder clientBuilder = AmazonKinesisClientBuilder.standard().withRegion(myRegion).withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(myAccessKeyId, mySecretKey)));
AmazonKinesis kinesisClient = clientBuilder.build();
PutRecordsRequest putRecordsRequest = new PutRecordsRequest();
putRecordsRequest.setStreamName(myQueue);
List putRecordsRequestEntryList  = new ArrayList<>(); 


//You can put multiple entries at once if you wanted to
PutRecordsRequestEntry putRecordsRequestEntry  = new PutRecordsRequestEntry();
putRecordsRequestEntry.setData(ByteBuffer.wrap(myData));
putRecordsRequestEntry.setPartitionKey(myKey);
putRecordsRequestEntryList.add(putRecordsRequestEntry);


putRecordsRequest.setRecords(putRecordsRequestEntryList);
PutRecordsResult putResult = kinesisClient.putRecords(putRecordsRequest);

Put Single Record On Queue

Import the following

import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.kinesis.AmazonKinesis;
import com.amazonaws.services.kinesis.AmazonKinesisClientBuilder;
import com.amazonaws.services.kinesis.model.PutRecordRequest;
import com.amazonaws.services.kinesis.model.Record;

Put Record

AmazonKinesisClientBuilder clientBuilder = AmazonKinesisClientBuilder.standard().withRegion(myRegion).withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(myAccessKeyId, mySecretKey)));
AmazonKinesis kinesisClient = clientBuilder.build();
PutRecordRequest putRecordRequest = new PutRecordRequest();
putRecordRequest.setStreamName(myQueue);

putRecordRequest.setData(ByteBuffer.wrap(data.getBytes("UTF-8")));
putRecordRequest.setPartitionKey(myKey);

PutRecordResult putResult = kinesisClient.putRecord(putRecordRequest);

You now have put a record(s) onto the queue congratulations!

AWS: Java S3 Upload

This entry is part 3 of 5 in the series AWS & Java

If you want to push data to AWS S3 there are a few different ways of doing this. I will show you two ways I have used.

Option 1: putObject

import com.amazonaws.AmazonClientException;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;

ClientConfiguration config = new ClientConfiguration();
config.setSocketTimeout(SOCKET_TIMEOUT);
config.setMaxErrorRetry(RETRY_COUNT);
config.setClientExecutionTimeout(CLIENT_EXECUTION_TIMEOUT);
config.setRequestTimeout(REQUEST_TIMEOUT);
config.setConnectionTimeout(CONNECTION_TIMEOUT);

AWSCredentialsProvider credProvider = ...;
String region = ...;

AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withCredentials(credProvider).withRegion(region).withClientConfiguration(config).build();

InputStream stream = ...;
String bucketName = .....;
String keyName = ...;
String mimeType = ...;

//You use metadata to describe the data.
final ObjectMetadata metaData = new ObjectMetadata();
metaData.setContentType(mimeType);

//There are overrides available. Find the one that suites what you need.
try {
	s3Client.putObject(bucketName, keyName, stream, metaData);
} catch (final AmazonClientException ex) {
	//Log the exception
}

Option 2: MultiPart Upload

import com.amazonaws.AmazonClientException;
import com.amazonaws.event.ProgressEvent;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.event.ProgressEventType;
import com.amazonaws.event.ProgressListener;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.transfer.TransferManager;
import com.amazonaws.services.s3.transfer.TransferManagerBuilder;
import com.amazonaws.services.s3.transfer.Upload;

ClientConfiguration config = new ClientConfiguration();
config.setSocketTimeout(SOCKET_TIMEOUT);
config.setMaxErrorRetry(RETRY_COUNT);
config.setClientExecutionTimeout(CLIENT_EXECUTION_TIMEOUT);
config.setRequestTimeout(REQUEST_TIMEOUT);
config.setConnectionTimeout(CONNECTION_TIMEOUT);

AWSCredentialsProvider credProvider = ...;
String region = ...;

AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withCredentials(credProvider).withRegion(region).withClientConfiguration(config).build();

InputStream stream = ...;
String bucketName = .....;
String keyName = ...;
long contentLength = ...;
String mimeType = ...;

//You use metadata to describe the data. You need the content length so the multi part upload knows how big it is
final ObjectMetadata metaData = new ObjectMetadata();
metaData.setContentLength(contentLength);
metaData.setContentType(mimeType);

TransferManager tf = TransferManagerBuilder.standard().withS3Client(s3Client).build();
tf.getConfiguration().setMinimumUploadPartSize(UPLOAD_PART_SIZE);
tf.getConfiguration().setMultipartUploadThreshold(UPLOAD_THRESHOLD);
Upload xfer = tf.upload(bucketName, keyName, stream, metaData);

ProgressListener progressListener = new ProgressListener() {
	public void progressChanged(ProgressEvent progressEvent) {
		if (xfer == null)
			return;
		
		if (progressEvent.getEventType() == ProgressEventType.TRANSFER_FAILED_EVENT || progressEvent.getEventType() == ProgressEventType.TRANSFER_PART_FAILED_EVENT) {
			//Log the message
		}
	}
};

xfer.addProgressListener(progressListener);
xfer.waitForCompletion();

Java: Enums

Below is an example of how to create a enum in Java.

public enum MyEnum {
	VAL(0), NEXTVAL(1);

	private int value;

	private MyEnum(int value) {
		setValue(value);
	}

	public int getValue() {
		return value;
	}

	public void setValue(int value) {
		this.value = value;
	}
}

Java: Input/Output Streams

This page is to give some basic examples of how to convert OutputStreams to InputStreams and vise versa.

Maven:

<dependency>
	<groupId>org.logback-extensions</groupId>
	<artifactId>logback-ext-loggly</artifactId>
	<version>0.1.4</version>
</dependency>

To convert an InputStream to OutputStream we can do it using IoUtils.copy as demonstrated below.

import ch.qos.logback.ext.loggly.io.IoUtils;
import java.io.InputStream;

InputStream input = ##INPUTSTREAM##;

//Convert InputStream to OutputStream

try (FileOutputStream out = new FileOutputStream(file)) {
	IoUtils.copy(input, out);
} catch (final IOException e) {
}

To convert a ByteArrayOutputStream to a ByteArrayInputStream we can do it as demonstrated below.

final ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
//Convert to ByteArrayInputStream
final ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(byteArrayOutputStream.toByteArray());

AWS: Java Kinesis Lambda Handler

This entry is part 2 of 5 in the series AWS & Java

If you want to write a Lambda for AWS in Java that connects to a Kinesis Stream. You need to have the handler.

Maven:

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk</artifactId>
    <version>1.11.109</version>
</dependency>
<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.7.1</version>
</dependency>

This is the method that AWS Lambda will call. It will look similar to the one below.

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.events.KinesisEvent;
import com.amazonaws.services.lambda.runtime.events.KinesisEvent.KinesisEventRecord;
import com.amazonaws.services.kinesis.model.Record;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.JsonNode;

public void kinesisRecordHandler(KinesisEvent kinesisEvent, Context context) {
	final String awsRequestId = context.getAwsRequestId();
	final int memoryLimitMb = context.getMemoryLimitInMB();
	final int remainingTimeInMillis = context.getRemainingTimeInMillis();

	for (final KinesisEventRecord kinesisRec : kinesisEvent.getRecords()) {
		final Record record = kinesisRec.getKinesis();

		//We get the kinesis data information
		final JsonNode recData = new ObjectMapper().readValue(record.getData().array(), JsonNode.class);

		final String bucketName = recData.get("bucket").asText();
		final String key = recData.get("key").asText();
	}
}

The thing to note when you setup you Lambda is how to setup the “Handler” field in the “Configuration” section on AWS. It is in the format “##PACKAGE##.##CLASS##::##METHOD##”.

AWS: Java S3 Lambda Handler

This entry is part 1 of 5 in the series AWS & Java

If you want to write a Lambda for AWS in Java that connects to S3. You need to have the handler.

Maven:

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk</artifactId>
    <version>1.11.109</version>
</dependency>

This is the method that AWS Lambda will call. It will look similar to the one below.

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.events.S3Event;
import com.amazonaws.services.s3.event.S3EventNotification.S3Entity;
import com.amazonaws.services.s3.event.S3EventNotification.S3EventNotificationRecord;

public void S3Handler(S3Event s3e, Context context) {
	final String awsRequestId =  context.getAwsRequestId();
	final int memoryLimitMb = context.getMemoryLimitInMB();
	final int remainingTimeInMillis = context.getRemainingTimeInMillis();

	for (final S3EventNotificationRecord s3Rec : s3e.getRecords()) {
		final S3Entity record = s3Rec.getS3();
		
		final String bucketName = record.getBucket().getName()
		final String key = record.getObject().getKey();
	}
}

The thing to note when you setup you Lambda is how to setup the “Handler” field in the “Configuration” section on AWS. It is in the format “##PACKAGE##.##CLASS##::##METHOD##”.

Java: Jackson Json

When you want to work with JSON the package of choice is Jackson. I find it so useful and easy to use.

Maven:

<dependency>
	<groupId>com.fasterxml.jackson.core</groupId>
	<artifactId>jackson-core</artifactId>
	<version>2.7.1</version>
</dependency>
<dependency>
	<groupId>com.fasterxml.jackson.core</groupId>
	<artifactId>jackson-annotations</artifactId>
	<version>2.7.1</version>
</dependency>
<dependency>
	<groupId>com.fasterxml.jackson.core</groupId>
	<artifactId>jackson-databind</artifactId>
	<version>2.7.1</version>
</dependency>

There are so many things that we can do with Jackson. It’s actually very exciting.

Let’s start with converting a json string to ObjectNode.

import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;

//We need an object mapper to help with the conversion
final ObjectMapper mapper = new ObjectMapper();
//Next we read the json string into ObjectNode
final ObjectNode node = (ObjectNode)mapper.readTree(jsonString);

//You can check if the objectnode has a key by
node.has("my_key")

//You can get the value by
node.get("my_key")

//To get the value as a string using 
.asText() at the end of the get

//To get as int
.asInt()

//To get as boolean
.asBoolean()

//To get as double
.asDouble()

//To get as Long
.asLong()

Next let’s convert a json string to JsonNode

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;

//You need an object mapper to help with the conversion
final ObjectMapper mapper = new ObjectMapper();
//Next we read the json string into a jsonnode
final JsonNode node = mapper.readTree(jsonString);
//We can if you want to switch it to an object node fairly easily.
final ObjectNode objNode = (ObjectNode)node;

//Put value in the object node
objNode.put("my_key", 1);

//You can also set json node
objNode.set("my_key", myJsonNode);

You can also create a new object node by using JsonNodeFactory. See below.

import com.fasterxml.jackson.databind.node.JsonNodeFactory;
import com.fasterxml.jackson.databind.node.ObjectNode;

//If you want to create an instance of ObjectNode
final ObjectNode objNode = JsonNodeFactory.instance.objectNode();

If you want to work with json array’s it’s also pretty straight forward. Again we work with JsonNodeFactory to declare it out.

import com.fasterxml.jackson.databind.node.ArrayNode;
import com.fasterxml.jackson.databind.node.JsonNodeFactory;

private ArrayNode arrNode = JsonNodeFactory.instance.arrayNode();

//We can then add to it
arrNode.add();

Jackson also has generators which allow us to create json on the fly.

import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.core.JsonProcessingException;

//It throws a "JsonProcessingException'

final JsonFactory jfactory = new JsonFactory();
ByteArrayOutputStream out=new ByteArrayOutputStream();

try (final JsonGenerator jg = jfactory.createGenerator(out, JsonEncoding.UTF8)) {
	jg.setCodec(objectMapper);

	jg.writeStartObject();

	jg.writeFieldName("my_key");
	jg.writeString("hi");

	jg.writeFieldName("next_key");
	jg.writeObject(my_obj);

	jg.writeFieldName("third_key");
	jg.writeTree(my_obj_node);

	jg.writeEndObject();
}

Jackson JsonViews are really cool in my opinion. We can have an object that has multiple properties and we can set a view to each one of these properties. Which means when we serialize the object we will only get those fields that are associated to that view and vise versa on the deserialize.

import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.annotation.JsonView;

//You can however many views we want to work with in our objects. They must all be classes. 
//If you extend a class then it will contain all properties of that view and the view it extends.
public class MyClassViews {
	public static class ViewA {}
	public static class ViewB extends ViewA {}
}

//Let's create an object for our serialization and deserialization
public class MyClass {
	private final int id;
	private final String value;

	//This constructor is what the deserializer would use.
	@JsonCreator
	public MyClass(@JsonProperty("id") int id, @JsonProperty("value") String value) {
	}

	@JsonView(MyClassViews.ViewA.class)
	@JsonProperty("id") //You don't have to put the name you could leave it as @JsonProperty
	public int getId() {
		return id;
	}

	@JsonView(MyClassViews.ViewB.class)
	@JsonProperty("value")
	public String getValue() {
		return value;
	}
}

I didn’t come up with this next part but it’s so awesome that I wanted to include it. Let’s say you had an object that was populated not using any views but the object does contain views that will eventually pass up to a UI but you only want to pass using a specific view. Then you can do the following.

import javax.ws.rs.core.StreamingOutput;
import com.fasterxml.jackson.databind.ObjectMapper;

final StreamingOutput jsonStream = new StreamingOutput() {
	@Override
	public void write(OutputStream out) throws IOException {
		final ObjectMapper mapper = new ObjectMapper();
		mapper.writerWithView(MyClassViews.ViewB.class).writeValue(out, MyClass);
	}
};

If you want to write the object to string using a view. All you need to do is

final String jsonString = new ObjectMapper().writerWithView(ViewB).writeValueAsString(MyClass);

If you then want to convert our new jsonString to the object using the view do the following:

new ObjectMapper().readerWithView(ViewB.class).forType(MyClass.class).readValue(jsonString);

If you want to convert a json string to object.

new ObjectMapper().readValue(myJsonString, MyClass.class)

Java: Connect to Postgres

Below are the steps to setup a connection to a Postgres DB and some other options that you can use.

Pom.xml:

<dependency>
	<groupId>org.postgresql</groupId>
	<artifactId>postgresql</artifactId>
	<version>9.4.1208.jre7</version>
</dependency>

Import:

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;

Build The Connection:

Class.forName("org.postgresql.Driver");
Connection connection = DriverManager.getConnection(##URL##, ##USER##, ##PASS##);

Preparing the Query:
We utilise “PreparedStatement” to setup the connection. Which will allow us to use parameters in the query.

PreparedStatement ps = connection.prepareStatement("select id from table where column=?")

If your query had parameters (IE: ?) then you will need to pass in the value for each parameter. If you notice there is ##POSITION## and ##VALUE##. Position is the location of where the parameter appears in the query. You can set various types of data for example integer, json, etc.

ps.setString(##POSITION##, ##VALUE##);

After we perform a query we need to retrieve the data. We use “ResultSet” for that. Depending on how we return the data depends on how we get the data after the query succeeds. Below are examples of one line returned vs multiple rows returned.

ResultSet rs = ps.executeQuery();

if (rs.next()) {
	int variable = rs.getInt("id");
}

while (rs.next()) {
	//Do Something
}

Insert:
If you want to perform an insert you don’t need to do “executeQuery” you can call just “execute”.

ps = connection.prepareStatement("insert into mytable (column) values (?)");
ps.setInt(##POSITION##, ##VALUE##);
ps.execute();

Batching:
Sometimes we have a lot of updates or inserts to perform. We should batch that.

//You setup the preparedStatement and then add to the batch.
ps.addBatch();

//You can set a max batch size to send the batch once it hits that amount
if (++count % batchSize == 0) {
	ps.executeBatch();
}

Passing Json as a Parameter:
We create the postgres object. Set the type to json and the value as a string in json format. Next we set the preparedStatement parameter as the postgres object that we just created.

final PGobject jsonObject = new PGobject();
jsonObject.setType("json");
jsonObject.setValue(value.toString());

//Now set the json object into the preparedStatement
ps.setObject(##POSITION##, jsonObject);

Java: ExecutorService / Future

If you want to spin off a bunch of threads and manage them and their responses effectively you can do it this way.

final ExecutorService executor = Executors.newFixedThreadPool(numThreads);
final Collection<Future<JsonNode>> futures = new LinkedList<Future<JsonNode>>();

//Write a Loop if you want
final Callable<TYPE> callable = new MyClass();
futures.add(executor.submit(callable));
executor.shutdown();

// We need to monitor the queries for their data being returned.
for (final Future<?> future : futures) {
	try {
		final TYPE val = (TYPE) future.get();
	} catch (final InterruptedException e) {
	} catch (final ExecutionException e) {
	}
}

 

The callable works with a class so you will need that.

package mypackage;

import java.util.concurrent.Callable;
import org.apache.log4j.Logger;

public class MyClass implements Callable<JsonNode> {

    static final Logger logger = Logger.getLogger(MyClass.class);

    MyClass() {
    }

    /**
     * This is how each caller queries for the data. It can be called many times and runs on threads from the calling class
     * So data is returned as it gets it.
     */
    @Override
    public TYPE call() throws Exception {
        try {
                  return null;
        } catch (final Exception e) {
            return null;
        }
    }
}