I have a use case where I get 10K to 15 K Messages/sec and it might be less than 5K also sometime and I push those into rabbitMQ now those messages I should parse,run some RE on that and do some sort aggregation and run some statistics. My product(the data pipeline) would be deployed in single machine as there are some business constrain. Have explored Spark Akka Storm and RxJava, Could you suggest me what to be used. But I don't want to do in plain java, as by this way I have to handle all the threading etc.
Best How To :
Based on my experience I would go for Akka. You can create different pools of actors to perform some tasks concurrently on different messages. Also you can leverage the power of
akka-camel to have a
RabbitMQ consumer using the
RabbitMQ Camel Component.
You might be able to do the same with Storm but I don't have that much experience with it to recommend it personally.
I wouldn't go for
Apache Spark since you would need to user
Apache Spark Streaming and you will need to learn about how to configure the buffering window correctly for your use case.