Kafka: Installation (Basic)

(Last Updated On: )

To install Kafka is really straight forward. There is a quick start guide you can follow. The only thing I found was that it didn’t call out Java 8. I will be using Ubuntu 16.04 for this installation.

Install Java 8

sudo apt-get install openjdk-8-jdk

Install Kafka

wget http://apache.forsale.plus/kafka/1.1.0/kafka_2.11-1.1.0.tgz 
tar -xzf kafka_2.11-1.1.0.tgz
sudo mv kafka_2.11-1.1.0/ /usr/local/kafka
cd /usr/local/kafka/

Setup .bashrc:

 sudo nano ~/.bashrc

Add the following to the end of the file.

#KAFKA VARIABLES START
export KAFKA_HOME=/usr/local/kafka
export KAFKA_CONF_DIR=/usr/local/kafka/conf
export PATH=$PATH:$KAFKA_HOME/bin
#KAFKA VARIABLES STOP

 source ~/.bashrc

ZooKeeper

Zookeeper comes pre-installed with kafka but you can run your own. For the purposes of this we just use the built in zookeeper.

bin/zookeeper-server-start.sh config/zookeeper.properties

Kafka Server

Now we can run the kafka server and start receiving messages on topics.

bin/kafka-server-start.sh config/server.properties

List Topics

/usr/local/kafka/bin/kafka-topics.sh --list --zookeeper hadoop:2181

Create Topic

/usr/local/kafka/bin/kafka-topics.sh --create --zookeeper hadoop:2181 --replication-factor 1 --partitions 1 --topic test

Auto Start

So if you want Kafka to run at startup then do the following.

touch kafka_start.sh
sudo chmod +x kafka_start.sh
touch kafka_stop.sh
sudo chmod +x kafka_stop.sh
crontab -e

Add the following and save.

@reboot /home/kafka/kafka_start.sh

kafka_start.sh

#!/bin/bash

/usr/local/kafka/bin/zookeeper-server-start.sh -daemon /usr/local/kafka/config/zookeeper.properties
sleep 2
/usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties

kafka_stop.sh

#!/bin/bash

/usr/local/kafka/bin/zookeeper-server-stop.sh
sleep 2
/usr/local/kafka/bin/kafka-server-stop.sh