Cygnus startup error: ClassNotFoundException

Question

My environment is VM with CentOS 6.6

I followed the instruction in https://github.com/telefonicaid/fiware-cygnus/blob/master/doc/quick_start_guide.md to install Cygnus. java-1.6.0-openjdk-devel was also installed and the JAVA_HOME set.

Installation went fine and it installed package cygnus.x86_64 0:0.8.2-0.g7353160

To test Cygnus I created the agent_test.conf and notification.sh files as instructed. When I start Cygnus, it outputs following stuff:

[root@localhost ~]# /usr/cygnus/bin/cygnus-flume-ng agent --conf /usr/cygnus/conf/ -f /usr/cygnus/conf/agent_test.conf -n cygnusagent -Dflume.root.logger=INFO,console
+ exec /usr/lib/jvm/java-1.6.0-openjdk.x86_64/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp '/usr/cygnus/conf:/usr/cygnus/lib/*:/usr/cygnus/plugins.d/cygnus/lib/*:/usr/cygnus/plugins.d/cygnus/libext/*' -Djava.library.path= com.telefonica.iot.cygnus.nodes.CygnusApplication -f /usr/cygnus/conf/agent_test.conf -n cygnusagent
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/cygnus/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/cygnus/plugins.d/cygnus/lib/cygnus-0.8.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2015-07-09 15:17:55,363 (main) [INFO - com.telefonica.iot.cygnus.nodes.CygnusApplication.main(CygnusApplication.java:235)] Starting a Jetty server listening on port 8081 (Management Interface)
2015-07-09 15:17:55,403 (main) [INFO - org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2015-07-09 15:17:55,404 (main) [INFO - com.telefonica.iot.cygnus.nodes.CygnusApplication.main(CygnusApplication.java:238)] Starting Cygnus application
2015-07-09 15:17:55,408 (Thread-1) [INFO - org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] jetty-6.1.26
2015-07-09 15:17:55,414 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:61)] Configuration provider starting
2015-07-09 15:17:55,421 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:133)] Reloading configuration file:/usr/cygnus/conf/agent_test.conf
2015-07-09 15:17:55,435 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:test-sink
2015-07-09 15:17:55,435 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:test-sink
2015-07-09 15:17:55,436 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: test-sink Agent: cygnusagent
2015-07-09 15:17:55,495 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [cygnusagent]
2015-07-09 15:17:55,495 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:150)] Creating channels
2015-07-09 15:17:55,520 (Thread-1) [INFO - org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Started SocketConnector@0.0.0.0:8081
2015-07-09 15:17:55,527 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:40)] Creating instance of channel test-channel type memory
2015-07-09 15:17:55,555 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:205)] Created channel test-channel
2015-07-09 15:17:55,558 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:39)] Creating instance of source http-source, type org.apache.flume.source.http.HTTPSource
2015-07-09 15:17:55,589 (conf-file-poller-0) [INFO - com.telefonica.iot.cygnus.handlers.OrionRestHandler.<init>(OrionRestHandler.java:75)] Cygnus version (0.8.2.UNKNOWN)
2015-07-09 15:17:55,635 (conf-file-poller-0) [INFO - com.telefonica.iot.cygnus.handlers.OrionRestHandler.configure(OrionRestHandler.java:141)] Startup completed
2015-07-09 15:17:55,664 (conf-file-poller-0) [ERROR - org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:113)] Builder class not found. Exception follows.
java.lang.ClassNotFoundException: com.telefonica.iot.cygnus.interceptors.DestinationExtractor$Builder
        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:191)
        at org.apache.flume.interceptor.InterceptorBuilderFactory.newInstance(InterceptorBuilderFactory.java:46)
        at org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:109)
        at org.apache.flume.channel.ChannelProcessor.configure(ChannelProcessor.java:80)
        at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
        at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:353)
        at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
        at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:701)
2015-07-09 15:17:55,670 (conf-file-poller-0) [ERROR - org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:366)] Source http-source has been removed due to an error during configuration
org.apache.flume.FlumeException: Interceptor.Builder not found.
        at org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:114)
        at org.apache.flume.channel.ChannelProcessor.configure(ChannelProcessor.java:80)
        at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
        at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:353)
        at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
        at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:701)
Caused by: java.lang.ClassNotFoundException: com.telefonica.iot.cygnus.interceptors.DestinationExtractor$Builder
        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:191)
        at org.apache.flume.interceptor.InterceptorBuilderFactory.newInstance(InterceptorBuilderFactory.java:46)
        at org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:109)
        ... 13 more
2015-07-09 15:17:55,672 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:40)] Creating instance of sink: test-sink, type: com.telefonica.iot.cygnus.sinks.OrionTestSink
2015-07-09 15:17:55,688 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:119)] Channel test-channel connected to [test-sink]
2015-07-09 15:17:55,704 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sourceRunners:{} sinkRunners:{test-sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@26651f73 counterGroup:{ name:null counters:{} } }} channels:{test-channel=org.apache.flume.channel.MemoryChannel{name: test-channel}} }
2015-07-09 15:17:55,705 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel test-channel
2015-07-09 15:17:55,822 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:110)] Monitoried counter group for type: CHANNEL, name: test-channel, registered successfully.
2015-07-09 15:17:55,822 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:94)] Component type: CHANNEL, name: test-channel started
2015-07-09 15:17:55,823 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink test-sink
2015-07-09 15:17:55,824 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.stopAllComponents(Application.java:101)] Shutting down configuration: { sourceRunners:{} sinkRunners:{test-sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@26651f73 counterGroup:{ name:null counters:{} } }} channels:{test-channel=org.apache.flume.channel.MemoryChannel{name: test-channel}} }
2015-07-09 15:17:55,824 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.stopAllComponents(Application.java:115)] Stopping Sink test-sink
2015-07-09 15:17:55,825 (conf-file-poller-0) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise(LifecycleSupervisor.java:171)] Stopping component: SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@26651f73 counterGroup:{ name:null counters:{} } }
2015-07-09 15:17:55,825 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.stopAllComponents(Application.java:125)] Stopping Channel test-channel
2015-07-09 15:17:55,825 (conf-file-poller-0) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise(LifecycleSupervisor.java:171)] Stopping component: org.apache.flume.channel.MemoryChannel{name: test-channel}
2015-07-09 15:17:55,826 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:139)] Component type: CHANNEL, name: test-channel stopped
2015-07-09 15:17:55,826 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:145)] Shutdown Metric for type: CHANNEL, name: test-channel. channel.start.time == 1436444275822
2015-07-09 15:17:55,826 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:151)] Shutdown Metric for type: CHANNEL, name: test-channel. channel.stop.time == 1436444275825
2015-07-09 15:17:55,827 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:167)] Shutdown Metric for type: CHANNEL, name: test-channel. channel.capacity == 1000
2015-07-09 15:17:55,827 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:167)] Shutdown Metric for type: CHANNEL, name: test-channel. channel.current.size == 0
2015-07-09 15:17:55,827 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:167)] Shutdown Metric for type: CHANNEL, name: test-channel. channel.event.put.attempt == 0
2015-07-09 15:17:55,827 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:167)] Shutdown Metric for type: CHANNEL, name: test-channel. channel.event.put.success == 0
2015-07-09 15:17:55,828 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:167)] Shutdown Metric for type: CHANNEL, name: test-channel. channel.event.take.attempt == 0
2015-07-09 15:17:55,828 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:167)] Shutdown Metric for type: CHANNEL, name: test-channel. channel.event.take.success == 0
2015-07-09 15:17:55,828 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sourceRunners:{} sinkRunners:{test-sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@26651f73 counterGroup:{ name:null counters:{} } }} channels:{test-channel=org.apache.flume.channel.MemoryChannel{name: test-channel}} }
2015-07-09 15:17:55,828 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel test-channel
2015-07-09 15:17:55,829 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:94)] Component type: CHANNEL, name: test-channel started
2015-07-09 15:17:55,830 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink test-sink
2015-07-09 15:17:55,830 (lifecycleSupervisor-1-4) [INFO - com.telefonica.iot.cygnus.sinks.OrionTestSink.start(OrionTestSink.java:61)] [test-sink] Startup completed

and obviously the test notification doesn't go through

[root@localhost ~]# ./notification.sh http://localhost:5050/notify
* About to connect() to localhost port 5050 (#0)
*   Trying ::1... Connection refused
*   Trying 127.0.0.1... Connection refused
* couldn't connect to host
* Closing connection #0
curl: (7) couldn't connect to host

Port 5050 should be OK as it's added with

iptables -A INPUT -p tcp --dport 5050 -j ACCEPT

I found a bit similar problem Cygnus error: class not found DestinationExtractor$Builder but that was with a much older version. Should Cygnus 0.8.2-0.g7353160 work?

Any pointers?

Answer 1

From Cygnus 0.8.1 the interceptor type for the grouping mechanism is not named DestinationExtractor anymore, but GroupingInterceptor. This is detailed in the Cygnus 0.8.2 README (showing relevant parts):

cygnusagent.sources.http-source.interceptors = ts gi
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf

Nevertheless, the Quick Start Guide must be updated, nice catch!