Kafka topic per producer
<p>Lets say I have multiple devices. Each device has different type of sensors. Now I want to send the data from each device for each sensor to kafka. But I am confused about the kafka topics. For processing this real time data</p>

<p>Is it good to have kafka topic per device and all the sensors from that device will send the data to particular kafka topic, or I should create one topic and have all the devices send the data to that one topic. </p>

<p>If I go with first case where we will create topic per device then, 
</p>

<p>Device1 (sensor A, B, C) -> topic1</p>

<p>Device2 (sensor A, B, C) -> topic2</p>

<ol>
<li>how many topics I can create? </li>
<li>Will this model scale?</li>
</ol>

<p>Case 2: where, sending data to one topic</p>

<p>Device1 (sensor A, B, C), Device2 (sensor A, B, C)....DeviceN.... -> topic</p>

<ol>
<li><p>Isn't this going to be bottleneck for data. Since it will behave as queue data from some sensor will be way behind in queue and will not be processed in real time.</p></li>
<li><p>Will this model scale?</p></li>
</ol>

<p><strong>EDIT</strong></p>

<p>Lets say each device is associated with user (many to one). So I want to process data according to devices. So the way I want to process data is, each device and its sensor data will go to the user after some processing.</p>

<p>Say I do following</p>

<p>Device1 </p>

<p>-> Sensor A - Topic1 Partition 1</p>

<p>-> Sensor B - Topic1 Partition 2</p>

<p>Device2 </p>

<p>-> Sensor A - Topic2 Partition 1</p>

<p>-> Sensor B - Topic2 Partition 2</p>

<p>I want some pub/sub type of behavior. Since devices can be added or removed also sensors can be added or removed. Is there a way to create these topics and partition on the fly. </p>

<p>If not kafka, what pub/sub will be suitable for this kind of behavior.</p>