Warning on Cygnus avoids data persistence on Cosmo

Question

This is a cygnus agent log:

16 Sep 2015 12:30:19,820 INFO  [521330370@qtp-1739580287-1] (com.telefonica.iot.cygnus.handlers.OrionRestHandler.getEvents:236)  - Received data (<notifyContextRequest><subscriptionId>55f932e6c06c4173451bbe1c</subscriptionId><originator>localhost</originator>...<contextAttribute><name>utctime</name><type>string</type><contextValue>2015-9-16 9:37:52</contextValue></contextAttribute></contextAttributeList></contextElement><statusCode><code>200</code><reasonPhrase>OK</reasonPhrase></statusCode></contextElementResponse></contextResponseList></notifyContextRequest>)
16 Sep 2015 12:30:19,820 INFO  [521330370@qtp-1739580287-1] (com.telefonica.iot.cygnus.handlers.OrionRestHandler.getEvents:258)  - Event put in the channel (id=1145647744, ttl=0)
16 Sep 2015 12:30:19,820 WARN  [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:184)  - 
16 Sep 2015 12:30:19,820 INFO  [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:193)  - Finishing transaction (1442395508-572-0000013907)

We conserve the same configuration than this question: Fiware Cygnus Error.

Although the Cygnus agent receives data correctly from the Context Broker suscription, Cosmos doesn't receive any data.

Thanks in advance, again :)

Answer 1

Independentely of the reason that leaded you to comment the grouing rules part (nevertheless, I think it was because my own wrong advice at https://jira.fiware.org/browse/HELC-986 :)), that part cannot be commented and must be added to the configuration:

# Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts gi
# TimestampInterceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
# GroupinInterceptor, do not change
cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
# Grouping rules for the GroupingInterceptor, put the right absolute path to the file if necessary
# See the doc/design/interceptors document for more details
cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf

Once added that part, most probably another problem will arise: the performance of Cygnus will be very poor (that was the reason I wrongly adviced the user at https://jira.fiware.org/browse/HELC-986 to comment the grouping feature, in an attempt to increase the performance by removing processing steps). The reason is the latest version of Cygnus (0.8.2) is not ready to deal with the HiveServer2 running in the Cosmos side (this server was recently upgraded from old HiveServer1 to HiveServer2) and each persistence operation delays for a lot. For instance:

time=2015-09-21T12:42:30.405CEST | lvl=INFO | trans=1442832138-143-0000000000 | function=getEvents | comp=Cygnus | msg=com.telefonica.iot.cygnus.handlers.OrionRestHandler[150] : Starting transaction (1442832138-143-0000000000)
time=2015-09-21T12:42:30.407CEST | lvl=INFO | trans=1442832138-143-0000000000 | function=getEvents | comp=Cygnus | msg=com.telefonica.iot.cygnus.handlers.OrionRestHandler[236] : Received data ({  "subscriptionId" : "51c0ac9ed714fb3b37d7d5a8",  "originator" : "localhost",  "contextResponses" : [    {      "contextElement" : {        "attributes" : [          {            "name" : "temperature",            "type" : "centigrade",            "value" : "26.5"          }        ],        "type" : "Room",        "isPattern" : "false",        "id" : "Room1"      },      "statusCode" : {        "code" : "200",        "reasonPhrase" : "OK"      }    }  ]})
time=2015-09-21T12:42:30.409CEST | lvl=INFO | trans=1442832138-143-0000000000 | function=getEvents | comp=Cygnus | msg=com.telefonica.iot.cygnus.handlers.OrionRestHandler[258] : Event put in the channel (id=1966649489, ttl=10)
time=2015-09-21T12:42:30.462CEST | lvl=INFO | trans=1442832138-143-0000000000 | function=process | comp=Cygnus | msg=com.telefonica.iot.cygnus.sinks.OrionSink[128] : Event got from the channel (id=1966649489, headers={fiware-servicepath=rooms, destination=other_rooms, content-type=application/json, fiware-service=deleteme2, ttl=10, transactionId=1442832138-143-0000000000, timestamp=1442832150410}, bodyLength=460)
time=2015-09-21T12:42:30.847CEST | lvl=INFO | trans=1442832138-143-0000000000 | function=persist | comp=Cygnus | msg=com.telefonica.iot.cygnus.sinks.OrionHDFSSink[330] : [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file (deleteme2/rooms/other_rooms/other_rooms.txt), Data ({"recvTimeTs":"1442832150","recvTime":"2015-09-21T10:42:30.410Z","entityId":"Room1","entityType":"Room","attrName":"temperature","attrType":"centigrade","attrValue":"26.5","attrMd":[]})
time=2015-09-21T12:42:31.529CEST | lvl=INFO | trans=1442832138-143-0000000000 | function=provisionHiveTable | comp=Cygnus | msg=com.telefonica.iot.cygnus.backends.hdfs.HDFSBackendImpl[185] : Creating Hive external table=frb_deleteme2_rooms_other_rooms_row

(a big timeout)

A workaround is to configure as hive_host an unreachable IP address such as fake.cosmos.lab.fiware.org:

time=2015-09-21T12:44:58.278CEST | lvl=INFO | trans=1442832280-746-0000000001 | function=getEvents | comp=Cygnus | msg=com.telefonica.iot.cygnus.handlers.OrionRestHandler[150] : Starting transaction (1442832280-746-0000000001)
time=2015-09-21T12:44:58.280CEST | lvl=INFO | trans=1442832280-746-0000000001 | function=getEvents | comp=Cygnus | msg=com.telefonica.iot.cygnus.handlers.OrionRestHandler[236] : Received data ({  "subscriptionId" : "51c0ac9ed714fb3b37d7d5a8",  "originator" : "localhost",  "contextResponses" : [    {      "contextElement" : {        "attributes" : [          {            "name" : "temperature",            "type" : "centigrade",            "value" : "26.5"          }        ],        "type" : "Room",        "isPattern" : "false",        "id" : "Room1"      },      "statusCode" : {        "code" : "200",        "reasonPhrase" : "OK"      }    }  ]})
time=2015-09-21T12:44:58.280CEST | lvl=INFO | trans=1442832280-746-0000000001 | function=getEvents | comp=Cygnus | msg=com.telefonica.iot.cygnus.handlers.OrionRestHandler[258] : Event put in the channel (id=1640732993, ttl=10)
time=2015-09-21T12:44:58.283CEST | lvl=INFO | trans=1442832280-746-0000000001 | function=process | comp=Cygnus | msg=com.telefonica.iot.cygnus.sinks.OrionSink[128] : Event got from the channel (id=1640732993, headers={fiware-servicepath=rooms, destination=other_rooms, content-type=application/json, fiware-service=deleteme3, ttl=10, transactionId=1442832280-746-0000000001, timestamp=1442832298280}, bodyLength=460)
time=2015-09-21T12:44:58.527CEST | lvl=INFO | trans=1442832280-746-0000000001 | function=persist | comp=Cygnus | msg=com.telefonica.iot.cygnus.sinks.OrionHDFSSink[330] : [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file (deleteme3/rooms/other_rooms/other_rooms.txt), Data ({"recvTimeTs":"1442832298","recvTime":"2015-09-21T10:44:58.280Z","entityId":"Room1","entityType":"Room","attrName":"temperature","attrType":"centigrade","attrValue":"26.5","attrMd":[]})
time=2015-09-21T12:44:59.148CEST | lvl=INFO | trans=1442832280-746-0000000001 | function=provisionHiveTable | comp=Cygnus | msg=com.telefonica.iot.cygnus.backends.hdfs.HDFSBackendImpl[185] : Creating Hive external table=frb_deleteme3_rooms_other_rooms_row
time=2015-09-21T12:44:59.304CEST | lvl=ERROR | trans=1442832280-746-0000000001 | function=doCreateTable | comp=Cygnus | msg=com.telefonica.iot.cygnus.backends.hive.HiveBackend[77] : Runtime error (The Hive table cannot be created. Hive query='create external table frb_deleteme3_rooms_other_rooms_row (recvTimeTs bigint, recvTime string, entityId string, entityType string, attrName string, attrType string, attrValue string, attrMd array<string>) row format serde 'org.openx.data.jsonserde.JsonSerDe' location '/user/frb/deleteme3/rooms/other_rooms''. Details=Could not establish connection to fake.cosmos.lab.fiware.org:10000/default?user=frb&password=llBl3dQsMhX2sEPtPuf3izUGS92RZo: java.net.UnknownHostException: fake.cosmos.lab.fiware.org)
time=2015-09-21T12:44:59.305CEST | lvl=WARN | trans=1442832280-746-0000000001 | function=provisionHiveTable | comp=Cygnus | msg=com.telefonica.iot.cygnus.backends.hdfs.HDFSBackendImpl[210] : The HiveQL external table could not be created, but Cygnus can continue working... Check your Hive/Shark installation
time=2015-09-21T12:44:59.305CEST | lvl=INFO | trans=1442832280-746-0000000001 | function=process | comp=Cygnus | msg=com.telefonica.iot.cygnus.sinks.OrionSink[193] : Finishing transaction (1442832280-746-0000000001)

This will allow Cygnus to continue, despite the Hive tables are not automatically created, a minor problem (anyway, they would have never been created because of the current incompatibility with HiveServer2). Of course, this will be fixed in Cygnus 0.9.0 (it will be released at the end of September 2015).