I've setup a Kestrel server and able to setup and use queues via the python pykestrel library.
We have a scenario where a python client writes to Kestrel queue(s) and a Storm spout needs to read from the queue(s). I've tried using the storm-kestrel library but running into issues. Googling seems to suggest it doesn't support the memcache port (22133).
I've added the maven bindings as provided here. Didn't use the KestrelThriftSpout spout, using Kestrel.Client. Compilation is fine but I get java.lang.NoClassDefFoundError error at runtime.
STACK TRACE
1098 [main] ERROR org.apache.zookeeper.server.NIOServerCnxn - Thread Thread[main,5,main] died
java.lang.NoClassDefFoundError: Lnet/lag/kestrel/thrift/Kestrel$Client;
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2397)
at java.lang.Class.getDeclaredField(Class.java:1946)
at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1659)
at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:72)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:480)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:468)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:468)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:365)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
at backtype.storm.utils.Utils.serialize(Utils.java:52)
at backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:94)
at com.rr.storm.ArticleTolopolgy.main(ArticleTolopolgy.java:35)
Caused by: java.lang.ClassNotFoundException: net.lag.kestrel.thrift.Kestrel$Client
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 15 more
Can someone please confirm if storm-kestrel can or cannot be used in this scenario?
If not, is there another kestrel client library that is recommended to be used from within a storm spout for reading data off of kestrel queue(s)?