Note to Self: Getting Apache Kafka Up & Running for Local Testing

This is a quick note on starting a single-broker Kafka 2.0.1 instance (Scala 2.11) on macOS, using Terminal. This is the standalone version (no Homebrew).

Terminal window 1

$ cd /Users/yourUsrName/pathToKafka/kafka_2.11-2.0.1
$ sh bin/zookeeper-server-start.sh config/zookeeper.properties

Terminal window 2

Optional: test zookeeper connection via telnet

$ telnet localhost 2181
$ stat

Move on to starting a Broker:

$ cd /Users/yourUsrName/pathToKafka/kafka_2.11-2.0.1
$ sh bin/kafka-server-start.sh config/server.properties

Terminal window 3

Create a Topic:

$ cd /Users/yourUsrName/pathToKafka/kafka_2.11-2.0.1
$ sh bin/kafka-topics.sh --create --topic my_topic_2 --zookeeper localhost:2181 --replication-factor 1 --partitions 1

For quicker local testing, I left the –partitions and –replication-factor set to 1.

See log files directory created here:

$ cd /tmp/kafka-logs/my_topic-0

See list of available Topics:

$ cd /Users/yourUsrName/pathToKafka/kafka_2.11-2.0.1
$ sh bin/kafka-topics.sh --list --zookeeper localhost:2181

Start a Producer

$ sh bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my_topic_2

Once you get a carrot prompt, try typing some messages:

> How is this thing doing? Is it on?  
> Anyone?

Terminal window 4

Start a Consumer.

$ cd /Users/yourUsrName/pathToKafka/kafka_2.11-2.0.1
$ sh bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my_topic_2 --from-beginning

Once the Consumer’s running without errors, you should see the messages you sent via the Producer window showing up.

Apache Kafka Producer Error “zookeeper is not a recognized option”

As I’m brushing up on Apache Kafka_2.11_2.0.1 (Scala 11, Kafka 2.0.1) on macOS Mojave, I ran into this minor hick up while trying to spin up a command line Producer:

$ sh bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic my_topic --from-beginning
zookeeper is not a recognized option

Turns out they changed that option name from “–zookeeper localhost:2181” to “–bootstrap-server localhost:9092. The new command looks like so:

$sh bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my_topic --from-beginning
Your message shows up here