You are managing a cluster with a large number of topics, and each topic has a lot of partitions. A team wants to significantly increase the number of partitions for some topics.
Which parameters should you check before increasing the partitions?
Each Kafka partition is associated with log files and index files that are opened and maintained by the broker. Increasing the number of partitions increases the number of file descriptors (open files). Therefore, it is critical to verify that the broker's system-level max open file count (ulimit -n) is sufficiently high.
From Kafka documentation:
“Each partition uses multiple open files. When the number of partitions increases significantly, make sure to increase the max open file descriptors allowed for the Kafka process.”
C → Correct: Prevents file descriptor exhaustion.
A/B/D → Unrelated to partition count scalability.
Page Reference:
Kafka: The Definitive Guide, 1st Edition, Chapter 6, p. 192
Apache Kafka Documentation: Production Deployment and Scaling Tips
‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
Contribute your Thoughts:
Chosen Answer:
This is a voting comment (?). You can switch to a simple comment. It is better to Upvote an existing comment if you don't have anything to add.
Submit