Thursday, July 24, 2014

Apache Kafka Tutorial

In this tutorial, we will do a basic installation of Apache Kafka and run one producer and consumer. You can follow the video here

Basic steps of installing and running apache kafka are:
  • Start Zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
  • Start Broker
bin/kafka-server-start.sh config/server.properties
  • Topic Creation
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
  • List Topic
bin/kafka-topics.sh --list --zookeeper localhost:2181
  • Start Producer
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test 
  • Start Consumer
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

12 comments:

  1. * Topic Creation should be:

    bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

    ReplyDelete
  2. Thank you Sir...It's really useful to get start..Please keep posting some advanced topics of Kafka

    ReplyDelete
  3. Hi sir you kafka tutorial was very nice. i am following same what u said but iam getting following error. please check once.

    java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:463)
    at sun.nio.ch.Net.bind(Net.java:455)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
    at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95)
    at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:111)
    at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
    at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)

    ReplyDelete
  4. Hi Dandu,

    Looks like the port that zookeeper binds to is being held by some other application. Depending on which OS you are try to find out which application is holding the port.

    In Linux you can use netstat -lnp

    ReplyDelete
  5. hellow !I would like to ask is it possible to how the ZK and brokes to one PC and run the produser to another pc (pc1) and the consumer to other (pc2)??

    ReplyDelete
    Replies
    1. Not sure if I get your question. But yes you can run all the components in different machines. In production you might be running them on different servers as per the use cases.

      Delete
    2. Thank you very much !it was really helpfull your anwser. Is there any environment so i can test the latency and other info to the messages between prod-cons? i am new to apache kafka !

      Delete
    3. I would suggest yo create virtual environments in your machine. You can use Oracle Virtual box and use ubuntu, both of which are free to use.

      Delete
  6. Hello !I would like to ask you ,is there any simplne tool so i can measure the end-to-end latency between producer and consumer? I have three machines, one running kafka server and zookeeper, one is running a producer and the last one is running a consumer.
    Thank you in advance!

    ReplyDelete
    Replies
    1. With distributed computing measuring end to end latency is difficult. Also is's an asynchronous system so when a producer put a message in kafka topics and consumer reads them are two different things. The best I can say is make your servers to synch with each other in terms of time and than put timestamps when producer is writing and when consumer is reading it.

      Delete
  7. Thank you very much Lalit, helpful as ever.

    ReplyDelete