-->

Cygnus not presisting data on MySql database

2019-02-25 17:13发布

站内文章 / MySQL
46 0 0

问题:

So i have read all the documentation and followed the tutorial on MySQL persistence but i can't still presist any kind of data on MySQL database.

Even though i'm puting the presistence mode = row it doesn't create any database nor table.

What am i doing wrong?

My Subscription: 

python2.7 SetSubscription.py bustest4 http://localhost:5050/notify

Output:

* Asking to http://localhost:1026/v1/subscribeContext
* Headers: {'Fiware-Service': 'fiwaretestapi', 'content-type': 'application/json                                                                                        ', 'accept': 'application/json', 'X-Auth-Token': 'NULL'}
* Sending PAYLOAD:
{
    "reference": "http://localhost:5050/notify",
    "throttling": "PT5S",
    "entities": [
        {
            "type": "",
            "id": "bustest4",
            "isPattern": "false"
        }
    ],
    "attributes": [
        "temperature"
    ],
    "duration": "P1M",
    "notifyConditions": [
        {
            "condValues": [
                "temperature"
            ],
            "type": "ONCHANGE"
        }
    ]
}

...

* Status Code: 200

{
  "subscribeResponse" : {
    "subscriptionId" : "5567332298add18cc3e183ac",
    "duration" : "P1M",
    "throttling" : "PT5S"
  }
}

My agent_a1.conf:

#=============================================
# source configuration
# channel name where to write the notification events
cygnusagent.sources.http-source.channels = hdfs-channel mysql-channel ckan-channel
# source class, must not be changed
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
# listening port the Flume source will use for receiving incoming notifications
cygnusagent.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler
# URL target
cygnusagent.sources.http-source.handler.notification_target = /notify
# Default service (service semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service = fiwaretestapi
# Default service path (service path semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service_path = /
# Number of channel re-injection retries before a Flume event is definitely discarded (-1 means infinite retries)
cygnusagent.sources.http-source.handler.events_ttl = 10
# Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts de
# Timestamp interceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
# Destination extractor interceptor, do not change
cygnusagent.sources.http-source.interceptors.de.type = es.tid.fiware.fiwareconnectors.cygnus.interceptors.DestinationExtractor$Builder
# Matching table for the destination extractor interceptor, put the right absolute path to the file if necessary
# See the doc/design/interceptors document for more details
cygnusagent.sources.http-source.interceptors.de.matching_table = /usr/cygnus/conf/matching_table.conf

# ============================================
# OrionMySQLSink configuration
# channel name from where to read notification events
cygnusagent.sinks.mysql-sink.channel = mysql-channel
# sink class, must not be changed
cygnusagent.sinks.mysql-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionMySQLSink
# the FQDN/IP address where the MySQL server runs
cygnusagent.sinks.mysql-sink.mysql_host = localhost
# the port where the MySQL server listes for incomming connections
cygnusagent.sinks.mysql-sink.mysql_port = 3306
# a valid user in the MySQL server
cygnusagent.sinks.mysql-sink.mysql_username = root
# password for the user above
cygnusagent.sinks.mysql-sink.mysql_password = ***********
# how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.mysql-sink.attr_persistence = row

#=============================================
# mysql-channel configuration
# channel type (must not be changed)
cygnusagent.channels.mysql-channel.type = memory
# capacity of the channel
cygnusagent.channels.mysql-channel.capacity = 1000
# amount of bytes that can be sent per transaction
cygnusagent.channels.mysql-channel.transactionCapacity = 100

My cygnus_instance_c1.conf:

# Who to run cygnus as. Note that you may need to use root if you want
# to run cygnus in a privileged port (<1024)
CYGNUS_USER=root

# Where is the config folder
CONFIG_FOLDER=/usr/cygnus/conf

# Which is the config file
CONFIG_FILE=/usr/cygnus/conf/agent_a1.conf

# Name of the agent. The name of the agent is not trivial, since it is the base for the Flume parameters
# naming conventions, e.g. it appears in .sources.http-source.channels=...
AGENT_NAME=cygnususer

# Name of the logfile located at /var/log/cygnus. It is important to put the extension '.log' in order to the log rotation works properly
LOGFILE_NAME=cygnus.log

# Administration port. Must be unique per instance
ADMIN_PORT=8081

My cygnus.log:

Info: Sourcing environment configuration script /usr/cygnus/conf/flume-env.sh
Warning: JAVA_HOME is not set!
+ exec /usr/bin/java -Xmx20m -Dflume.log.file=cygnus.log -cp '/usr/cygnus/conf:/usr/cygnus/lib/*:/usr/cygnus/plugins.d/cygnus/lib/*:/usr/cygnus/plugins.d/cygnus/libext/*' -Djava.library.path= es.tid.fiware.fiwareconnectors.cygnus.nodes.CygnusApplication -p 8081 -f /usr/cygnus/conf/agent_a1.conf -n root
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/cygnus/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/cygnus/plugins.d/cygnus/lib/cygnus-0.7.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

EDIT:

So after some changes i got the log file to work and i found out that the 8081 port was already in use. Can you explain me what the ADMIN_PORT is used for, what port is it recommended to be?

LOG FILE:

02 Jun 2015 05:16:40,680 INFO  [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133)  - Reloading configuration file:/usr/cygnus/conf/agent_a1.conf
02 Jun 2015 05:16:40,685 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:mysql-sink
02 Jun 2015 05:16:40,686 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:mysql-sink
02 Jun 2015 05:16:40,686 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:930)  - Added sinks: mysql-sink Agent: cygnusagent
02 Jun 2015 05:16:40,686 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:mysql-sink
02 Jun 2015 05:16:40,687 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:mysql-sink
02 Jun 2015 05:16:40,687 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:mysql-sink
02 Jun 2015 05:16:40,687 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:mysql-sink
02 Jun 2015 05:16:40,687 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:mysql-sink
02 Jun 2015 05:16:40,692 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140)  - Post-validation flume configuration contains configuration for agents: [cygnusagent]
02 Jun 2015 05:16:40,692 WARN  [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.getConfiguration:138)  - No configuration found for this host:root
02 Jun 2015 05:16:40,693 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:101)  - Shutting down configuration: { sourceRunners:{} sinkRunners:{} channels:{} }
02 Jun 2015 05:16:40,693 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138)  - Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }
02 Jun 2015 05:16:40,694 INFO  [conf-file-poller-0] (es.tid.fiware.fiwareconnectors.cygnus.nodes.CygnusApplication.startManagementInterface:85)  - Starting a Jetty server listening on port 8081 (Management Interface)
02 Jun 2015 05:16:40,695 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:101)  - Shutting down configuration: { sourceRunners:{} sinkRunners:{} channels:{} }
02 Jun 2015 05:16:40,695 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138)  - Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }
02 Jun 2015 05:16:40,695 INFO  [conf-file-poller-0] (es.tid.fiware.fiwareconnectors.cygnus.nodes.CygnusApplication.startManagementInterface:85)  - Starting a Jetty server listening on port 8081 (Management Interface)
02 Jun 2015 05:16:40,696 INFO  [Thread-26] (org.mortbay.log.Slf4jLog.info:67)  - jetty-6.1.26
02 Jun 2015 05:16:40,704 WARN  [Thread-26] (org.mortbay.log.Slf4jLog.warn:76)  - failed SocketConnector@0.0.0.0:8081: java.net.BindException: Address already in use
02 Jun 2015 05:16:40,704 WARN  [Thread-26] (org.mortbay.log.Slf4jLog.warn:76)  - failed Server@4f1b95f: java.net.BindException: Address already in use
02 Jun 2015 05:16:40,704 FATAL [Thread-26] (es.tid.fiware.fiwareconnectors.cygnus.http.JettyServer.run:63)  - Fatal error running the Management Interface. Details=Address already in use
02 Jun 2015 05:16:40,705 INFO  [Thread-27] (org.mortbay.log.Slf4jLog.info:67)  - jetty-6.1.26
02 Jun 2015 05:16:40,709 WARN  [Thread-27] (org.mortbay.log.Slf4jLog.warn:76)  - failed SocketConnector@0.0.0.0:8081: java.net.BindException: Address already in use
02 Jun 2015 05:16:40,709 WARN  [Thread-27] (org.mortbay.log.Slf4jLog.warn:76)  - failed Server@ed4c222: java.net.BindException: Address already in use
02 Jun 2015 05:16:40,709 FATAL [Thread-27] (es.tid.fiware.fiwareconnectors.cygnus.http.JettyServer.run:63)  - Fatal error running the Management Interface. Details=Address already in use

EDIT 2:

My script that updates entity on Context Broker:

BASE_URL = 'http://localhost:1026'
UPDATE_URL = BASE_URL+'/ngsi10/updateContext'

HEADERS = {
    'Content-Type': 'application/json',
    'Accept': 'application/json',
    'Fiware-Service' : 'fiwaretestapi',
    'Fiware-ServicePath': '/'
}


UPDATE_EXAMPLE = {
    "contextElements": [
        {
            "type": "",
            "isPattern": "false",
            "id": "bustest4",
            "attributes": [
            {
                "name": "temperature",
                "type": "int",
                "value": "99"
            }
            ]
        }
    ],
    "updateAction": "APPEND"
}

def post(url, data):
    """"""
    req = urllib2.Request(url, data, HEADERS)
    f = urllib2.urlopen(req)
    result = json.loads(f.read())
    f.close()
    return result

if __name__ == "__main__":
    print post(UPDATE_URL, json.dumps(UPDATE_EXAMPLE))

EDIT3:

Even though i set the admin port to be 8085 on the cygnus agent configuration it tries to bind to the 8081, is that normal?

Here are the logs from the cygnus:

time=2015-06-12T05:56:39.820EDT | lvl=INFO | trans= | function=start | comp=Cygnu                                                                                       s | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[94] : Component ty                                                                                       pe: CHANNEL, name: mysql-channel started
time=2015-06-12T05:56:39.821EDT | lvl=INFO | trans= | function=startAllComponents                                                                                        | comp=Cygnus | msg=org.apache.flume.node.Application[173] : Starting Sink mysql                                                                                       -sink
time=2015-06-12T05:56:39.821EDT | lvl=INFO | trans= | function=startAllComponents                                                                                        | comp=Cygnus | msg=org.apache.flume.node.Application[184] : Starting Source htt                                                                                       p-source
time=2015-06-12T05:56:39.821EDT | lvl=INFO | trans= | function=startManagementInt                                                                                       erface | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.nodes.CygnusAppl                                                                                       ication[85] : Starting a Jetty server listening on port 8081 (Management Interfac                                                                                       e)
time=2015-06-12T05:56:39.822EDT | lvl=INFO | trans= | function=start | comp=Cygnu                                                                                       s | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionMySQLSink[151] : [mysql-                                                                                       sink] Startup completed
time=2015-06-12T05:56:39.823EDT | lvl=INFO | trans= | function=info | comp=Cygnus                                                                                        | msg=org.mortbay.log.Slf4jLog[67] : jetty-6.1.26
time=2015-06-12T05:56:39.824EDT | lvl=INFO | trans= | function=info | comp=Cygnus                                                                                        | msg=org.mortbay.log.Slf4jLog[67] : jetty-6.1.26
time=2015-06-12T05:56:39.825EDT | lvl=INFO | trans= | function=info | comp=Cygnus                                                                                        | msg=org.mortbay.log.Slf4jLog[67] : Started SocketConnector@0.0.0.0:5050
time=2015-06-12T05:56:39.825EDT | lvl=INFO | trans= | function=start | comp=Cygnu                                                                                       s | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[94] : Component ty                                                                                       pe: SOURCE, name: http-source started
time=2015-06-12T05:56:39.827EDT | lvl=WARN | trans= | function=warn | comp=Cygnus                                                                                        | msg=org.mortbay.log.Slf4jLog[76] : failed SocketConnector@0.0.0.0:8081: java.n                                                                                       et.BindException: Address already in use
time=2015-06-12T05:56:39.827EDT | lvl=WARN | trans= | function=warn | comp=Cygnus                                                                                        | msg=org.mortbay.log.Slf4jLog[76] : failed Server@1c9c5521: java.net.BindExcept                                                                                       ion: Address already in use
time=2015-06-12T05:56:39.827EDT | lvl=FATAL | trans= | function=run | comp=Cygnus                                                                                        | msg=es.tid.fiware.fiwareconnectors.cygnus.http.JettyServer[63] : Fatal error r                                                                                       unning the Management Interface. Details=Address already in use

Log when i make a subscription:

time=2015-06-12T06:03:56.529EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu                                                                                       nction=getEvents | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.handle                                                                                       rs.OrionRestHandler[153] : Starting transaction (1434103313-190-0000000000)
time=2015-06-12T06:03:56.535EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu                                                                                       nction=getEvents | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.handle                                                                                       rs.OrionRestHandler[239] : Received data ({  "subscriptionId" : "557aae8c98add18c                                                                                       c3e183b6",  "originator" : "localhost",  "contextResponses" : [    {      "contex                                                                                       tElement" : {        "type" : "thing",        "isPattern" : "false",        "id"                                                                                        : "autocarro1",        "attributes" : [          {            "name" : "temperatu                                                                                       re",            "type" : "int",            "value" : "95",            "metadatas"                                                                                        : [              {                "name" : "TimeInstant",                "type"                                                                                        : "ISO8601",                "value" : "2015-06-03T09:17:44.046583Z"                                                                                                     }            ]          }        ]      },      "statusCode" : {        "code" :                                                                                        "200",        "reasonPhrase" : "OK"      }    }  ]})
time=2015-06-12T06:03:56.540EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu                                                                                       nction=getEvents | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.handle                                                                                       rs.OrionRestHandler[261] : Event put in the channel (id=1983722072, ttl=10)
time=2015-06-12T06:03:56.724EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu                                                                                       nction=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.Or                                                                                       ionSink[126] : Event got from the channel (id=1983722072, headers={timestamp=1434                                                                                       103436542, content-type=application/json, transactionId=1434103313-190-0000000000                                                                                       , fiware-service=fiwaretestapi, fiware-servicepath=, ttl=10, destination=autocarr                                                                                       o1_thing}, bodyLength=657)
time=2015-06-12T06:03:57.260EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu                                                                                       nction=persist | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.Or                                                                                       ionMySQLSink[227] : [mysql-sink] Persisting data at OrionMySQLSink. Database: fiw                                                                                       aretestapi, Table: autocarro1_thing, Data: 1434103436,2015-06-12T06:03:56.542,aut                                                                                       ocarro1,thing,temperature,thing,95,[{"name":"TimeInstant","type":"ISO8601","value                                                                                       ":"2015-06-03T09:17:44.046583Z"}]
time=2015-06-12T06:03:57.270EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu                                                                                       nction=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.Or                                                                                       ionSink[187] : Finishing transaction (1434103313-190-0000000000)

回答1:

As I can see in the log, most probably Cygnus is running but not starting any Flume component (any source, channel or sink). This is due to some configuration errors.

Regarding agent_a1.conf file:

  • It is missing the list of sources, channels and sinks: cygnusagent.sources = http-source cygnusagent.sinks = mysql-sink cygnusagent.channels = mysql-channel
  • cygnusagent.sources.http-source.channels value should be mysql-channel

Regarding cygnus_instance_c1.conf:

  • AGENT_NAME value must be cygnusagent

Which version have you installed? Are you reunning Cygnus as a service or as an standalone process?

In adition, you could try to start Cygnus in DEBUG mode? Simply edit the /usr/cygnus/conf/log4j.properties file.

Do the proposed changes and see how the log evolves! :)

EDIT 1

Such a "fatal error" is not so fatal. It was a bug appearing in Cygnus 0.7.1, currently fixed. Anyway, even in 0.7.1 it did not affect the normal behaviour of Cygnus since the management port is only used for retrieving the version, nothing important.

Did you try to send some update context to Orion in order a notification is received by Cygnus? Even by simulating the notification? Please, see the Cygnus Quick Start Guide for an example about how to make such a simulation.

EDIT 2

Cygnus packages names es.tid.fiware.fiwareconnectors.cygnus... were replaced by com.telefonica.iot.cygnus... from release 0.8.0 (or maybe 0.9.0).



标签： mysql fiware fiware-orion fiware-cygnus
举报

收藏的人(0)

Ta的文章 更多文章
登录 后发表评论
0条评论
还没有人评论过~