kafka-python no broker available with kafka on localhost
- 25 Dec 2017: Post was created (diff)
All of a sudden I was having problems getting a script based on
kafka-python to work properly. Even the example code from the upstrem repo didn’t work. However, I still had success with my Node.js scripts, so the problem had to be with the Python lib.
Immediately when creating a new
KafkaProducer an exception was thrown stating
No brokers available
Both the consumer and producer constructors in
kafka-python accepts a long list of keyword arguments, one of which is
api_version. Apparently, when no
api_version is set, an attempt is made to automatically detect the kafka version of the broker.
The stack trace printed along with the exception also hints about where the exception originated from, and it is when attempting to detect a version.
Specify the version explicitly, and keep on programming
KAFKA_HOSTS = ['localhost:9092'] KAFKA_VERSION = (0, 10) producer = KafkaProducer(bootstrap_servers=KAFKA_HOSTS, api_version=KAFKA_VERSION) topic = "foobars" consumer = KafkaConsumer(topic, bootstrap_servers=KAFKA_HOSTS, api_version=KAFKA_VERSION)
This may be a problem between
kafka-python and the current Kafka installation I am using within a docker image, wurstmeister/kafka-docker, but I have not bothered to confirm this at this time. At least it works!