In this tutorial, we will do a basic installation of Apache Kafka and run one producer and consumer. You can follow the video here
Basic steps of installing and running apache kafka are:
Basic steps of installing and running apache kafka are:
- Download Apache kafka from http://kafka.apache.org/
- Unzip it at a location
- Start Zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
- Start Broker
bin/kafka-server-start.sh config/server.properties
- Topic Creation
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
- List Topic
bin/kafka-topics.sh --list --zookeeper localhost:2181
- Start Producer
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
- Start Consumer
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
* Topic Creation should be:
ReplyDeletebin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Fixed. Thanks
DeleteThank you Sir...It's really useful to get start..Please keep posting some advanced topics of Kafka
ReplyDeleteHi sir you kafka tutorial was very nice. i am following same what u said but iam getting following error. please check once.
ReplyDeletejava.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:463)
at sun.nio.ch.Net.bind(Net.java:455)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:111)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Hi Dandu,
ReplyDeleteLooks like the port that zookeeper binds to is being held by some other application. Depending on which OS you are try to find out which application is holding the port.
In Linux you can use netstat -lnp
hellow !I would like to ask is it possible to how the ZK and brokes to one PC and run the produser to another pc (pc1) and the consumer to other (pc2)??
ReplyDeleteNot sure if I get your question. But yes you can run all the components in different machines. In production you might be running them on different servers as per the use cases.
DeleteThank you very much !it was really helpfull your anwser. Is there any environment so i can test the latency and other info to the messages between prod-cons? i am new to apache kafka !
DeleteI would suggest yo create virtual environments in your machine. You can use Oracle Virtual box and use ubuntu, both of which are free to use.
DeleteHello !I would like to ask you ,is there any simplne tool so i can measure the end-to-end latency between producer and consumer? I have three machines, one running kafka server and zookeeper, one is running a producer and the last one is running a consumer.
ReplyDeleteThank you in advance!
With distributed computing measuring end to end latency is difficult. Also is's an asynchronous system so when a producer put a message in kafka topics and consumer reads them are two different things. The best I can say is make your servers to synch with each other in terms of time and than put timestamps when producer is writing and when consumer is reading it.
DeleteThank you very much Lalit, helpful as ever.
ReplyDelete