Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Measuring Kafka Cluster requirements in data pipeline
#1
As every data processing component is connected through a Kafka message queue ideally, how do we measure the Kafka cluster requirements in Data Pipeline?
Reply
#2
(05-22-2019, 06:36 AM)noopur.mehrotra Wrote: As every data processing component is connected through a Kafka message queue ideally, how do we measure the Kafka cluster requirements in Data Pipeline?

We are using standard best practices for Kafka. Apache Kafka server (with coresident Apache ZooKeeper)

Two servers 200 GB daily ingestion per server. Each server has twelve physical processors and 16 GB of RAM. Apache ZooKeeper is installed on the primary Apache Kafka server.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)