FIWARE Cygnus -> cartodb sinks.NGSISink: Persisten

2019-03-02 06:15发布

站内问答 / 移动开发
959 2 5

I'm trying to connect cygnus (1.4.0_SNAPSHOT) to cartodb. I run it locally, and I use a script to send a notification to cygnus. The script runs Ok, but cygnus says: 

ERROR sinks.NGSISink: Persistence error (The query 'INSERT INTO jcarneroatos.x002fpeoplelocation (recvtime,fiwareservicepath,entityid,entitytype,the_geom) VALUES ('2016-10-31T19:04:00.994Z','/peoplelocation','Person:1','Person',ST_SetSRID(ST_MakePoint({"coordinates":[-4.423032856,36.721290055]), 4326))' could not be executed. CartoDB response: 400 Bad Request)

Anyone knows what could be happening? Below I put my config files for information, thanks!

My username at CARTO is "jcarneroatos", and the domain is https://jcarneroatos.carto.com. This is the script I'm using to simulate the notification from Orion Context Broker:

#/bin/bash
HOST=localhost
PORT=5050
SERVICE=jcarneroatos
SUBSERVICE=/peoplelocation

#send notification
NOTIFICATION=$(\
curl http://$HOST:$PORT/notify \
    -v -s -S \
    --header "Content-Type: application/json; charset=utf-8" \
    --header 'Accept: application/json' \
    --header "Fiware-Service: $SERVICE" \
    --header "Fiware-ServicePath: $SUBSERVICE" \
    -d '
    {
        "contextResponses": [
            {
                "contextElement": {
                    "attributes": [
                        {
                            "metadatas": [
                                {
                                    "name": "location",
                                    "type": "string",
                                    "value": "WGS84"
                                }
                            ],
                            "name": "location",
                            "type": "geo:json",
                            "value": {
                                "coordinates": [
                                    -4.423032856,
                                    36.721290055
                                ],
                                "type": "Point"
                            }
                        }
                    ],
                    "id": "Person:1",
                    "isPattern": "false",
                    "type": "Person"
                },
                "statusCode": {
                    "code": "200",
                    "reasonPhrase": "OK"
                }
            }
        ],
        "originator": "localhost",
        "subscriptionId": "58178396634ded66caac35b2"
    }')
if [ -z "$NOTIFICATION" ]; then
    echo "Ok"
else
    echo $NOTIFICATION
fi

This is the structure of the dataset at cartodb:

x002fpeoplelocation
cartodb_id | the_geom | entityid | entitytype | fiwareservicepath | recvtime
  number   | geometry |  string  |   string   |      string       |   date

This is the cygnus config file:

cygnusagent.sources = http-source
cygnusagent.sinks = cartodb-sink
cygnusagent.channels =cartodb-channel

cygnusagent.sources.http-source.channels = cartodb-channel
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnusagent.sources.http-source.port = 5050
cygnusagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnusagent.sources.http-source.handler.notification_target = /notify
cygnusagent.sources.http-source.handler.default_service = jcarneroatos
cygnusagent.sources.http-source.handler.default_service_path = /peoplelocation
cygnusagent.sources.http-source.interceptors = ts gi
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /home/cygnus/APACHE_FLUME_HOME/conf/grouping_rules.conf

cygnusagent.sinks.cartodb-sink.type = com.telefonica.iot.cygnus.sinks.NGSICartoDBSink
cygnusagent.sinks.cartodb-sink.channel = cartodb-channel
cygnusagent.sinks.cartodb-sink.enable_grouping = false
cygnusagent.sinks.cartodb-sink.enable_name_mappings = false
cygnusagent.sinks.cartodb-sink.enable_lowercase = false
cygnusagent.sinks.cartodb-sink.data_model = dm-by-service-path
cygnusagent.sinks.cartodb-sink.keys_conf_file = /home/cygnus/APACHE_FLUME_HOME/conf/cartodb_keys.conf
cygnusagent.sinks.cartodb-sink.flip_coordinates = false
cygnusagent.sinks.cartodb-sink.enable_raw = true
cygnusagent.sinks.cartodb-sink.enable_distance = false
cygnusagent.sinks.cartodb-sink.batch_size = 100
cygnusagent.sinks.cartodb-sink.batch_timeout = 30
cygnusagent.sinks.cartodb-sink.batch_ttl = 10
cygnusagent.sinks.cartodb-sink.backend.max_conns = 500
cygnusagent.sinks.cartodb-sink.backend.max_conns_per_route = 100

cygnusagent.channels.cartodb-channel.type = memory
cygnusagent.channels.cartodb-channel.capacity = 1000
cygnusagent.channels.cartodb-channel.transactionCapacity = 100

And finally the cartodb_keys.conf file (without key):

{
   "cartodb_keys": [
      {
         "username": "jcarneroatos",
         "endpoint": "https://jcarneroatos.carto.com",
         "key": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"   
      }
   ]
}

Update: After executing Cygnus in DEBUG mode and check the logs, it seems that CARTO is returning: 

{"error":["syntax error at or near \"{\""]}

This the complete log: http://pastebin.com/p9VyUU8n

2条回答
我只想做你的唯一
2楼-- · 2019-03-02 06:54

Finally, after offline discussion with @Javi Carnero, we found there was a bug in Cygnus code related to the way Carto differentiates among "enterprise" and "personal" accounts. First ones allow for PostgreSQL schemas per user under the enterprise organization, while second ones have "hardcoded" schemas named public. Since the notified FIWARE service was used as the schema name, Cygnus was not properly working for "personal" accounts.

Such a bug has been fixed:

At the moment of writting this, the bug is fixed in master branch. By the end of sprint/month Cygnus 1.7.0 will be released, including this fix.

Please observe my previous answer is perfectly valid if you have an "enterprise" account. Anyway, I'll edit it in order to explain this.

查看更多
0人赞 添加讨论(0) 举报
干净又极端
3楼-- · 2019-03-02 06:59

The problem is geo:json type is not currently supported by NGSICartoDBSink. This sink understands ceratain ways of notifying geolocated attributes, according to Orion Context Broker specification; these ones:

  • Using the geo:point type, and sending the coordinates in the value field with format "latitude, longitude".
  • Using the location metadata, of type string and value WGS84, and sending the coordinates in the value field with format "latitude, longitude".

Please observe:

  • The above options are exclussive, i.e. cannot be used at the same time.
  • The location metadata is deprecated in Orion, nevertheless it can be still used.

While geo:json is supported (I'll start working on that, it could be ready during this sprint/month), I'll recommend you to use the geo:point type.

EDIT 1

I'm adding here an example of Cygnus execution, when receiving a notification involving a geolocated attribute (geo:point type).

Cygnus version:

1.6.0

Cygnus configuration:

cygnus-ngsi.sources = http-source
cygnus-ngsi.sinks = raw-sink
cygnus-ngsi.channels = raw-channel

cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnus-ngsi.sources.http-source.channels = raw-channel
cygnus-ngsi.sources.http-source.port = 5050
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnus-ngsi.sources.http-source.handler.notification_target = /notify
cygnus-ngsi.sources.http-source.handler.default_service = default
cygnus-ngsi.sources.http-source.handler.default_service_path = /
cygnus-ngsi.sources.http-source.interceptors = ts
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp

cygnus-ngsi.sinks.raw-sink.type = com.telefonica.iot.cygnus.sinks.NGSICartoDBSink
cygnus-ngsi.sinks.raw-sink.channel = raw-channel
cygnus-ngsi.sinks.raw-sink.enable_grouping = false
cygnus-ngsi.sinks.raw-sink.keys_conf_file = /usr/cygnus/conf/cartodb_keys.conf
cygnus-ngsi.sinks.raw-sink.swap_coordinates = false
cygnus-ngsi.sinks.raw-sink.enable_raw = true
cygnus-ngsi.sinks.raw-sink.enable_distance = false
cygnus-ngsi.sinks.raw-sink.enable_raw_snapshot = false
cygnus-ngsi.sinks.raw-sink.data_model = dm-by-entity
cygnus-ngsi.sinks.raw-sink.batch_size = 50
cygnus-ngsi.sinks.raw-sink.batch_timeout = 10
cygnus-ngsi.sinks.raw-sink.batch_ttl = 0
cygnus-ngsi.sinks.raw-sink.batch_retries = 5000

cygnus-ngsi.channels.raw-channel.type = com.telefonica.iot.cygnus.channels.CygnusMemoryChannel
cygnus-ngsi.channels.raw-channel.capacity = 1000
cygnus-ngsi.channels.raw-channel.transactionCapacity = 100

Create the table:

$ curl -X GET -G "https://<my_user>.cartodb.com/api/v2/sql?api_key=<my_key>" --data-urlencode "q=CREATE TABLE x002ftestxffffx0043ar1xffffx0043ar (recvTime text, fiwareServicePath text, entityId text, entityType text, speed float, speed_md text, the_geom geometry(POINT,4326))"
{"rows":[],"time":0.005,"fields":{},"total_rows":0}

Script simulating a notification:

$ cat notification.sh
#!/bin/sh

URL=$1

if [ "$2" != "" ]
then
   SERVICE=$2
else
   SERVICE=default
fi

if [ "$3" != "" ]
then
   SERVICE_PATH=$3
else
   SERVICE_PATH=/
fi

curl $URL -v -s -S --header 'Content-Type: application/json; charset=utf-8' --header 'Accept: application/json' --header 'User-Agent: orion/0.10.0' --header "Fiware-Service: $SERVICE" --header "Fiware-ServicePath: $SERVICE_PATH" -d @- 
<<EOF
{
  "subscriptionId" : "51c0ac9ed714fb3b37d7d5a8",
  "originator" : "localhost",
  "contextResponses" : [
    {
      "contextElement" : {
        "attributes" : [
          {
            "name" : "speed",
            "type" : "float",
            "value" : "$6"
          },
          {
            "name" : "the_geom",
            "type" : "geo:point",
            "value" : "$4, $5"
          }
        ],
        "type" : "Car",
        "isPattern" : "false",
        "id" : "Car1"
      },
      "statusCode" : {
        "code" : "200",
        "reasonPhrase" : "OK"
      }
    }
  ]
}
EOF

Script execution:

$ ./notification.sh http://localhost:5050/notify <my_user> /test 40.40 -3.4 120
*   Trying ::1...
* Connected to localhost (::1) port 5050 (#0)
> POST /notify HTTP/1.1
> Host: localhost:5050
> Content-Type: application/json; charset=utf-8
> Accept: application/json
> User-Agent: orion/0.10.0
> Fiware-Service: <my_user>
> Fiware-ServicePath: /test
> Content-Length: 569
> 
* upload completely sent off: 569 out of 569 bytes
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Server: Jetty(6.1.26)
< 
* Connection #0 to host localhost left intact

Cygnus logs upon notification reception:

time=2016-12-02T13:48:27.310UTC | lvl=INFO | corr=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | trans=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | srv=<my_user> | subsrv=/test | comp=cygnus-ngsi | op=getEvents | msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[282] : [NGSIRestHandler] Starting internal transaction (eb73b8d5-af9b-48ea-8ce7-ff21edc957f3)
time=2016-12-02T13:48:27.312UTC | lvl=INFO | corr=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | trans=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | srv=<my_user> | subsrv=/test | comp=cygnus-ngsi | op=getEvents | msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[299] : [NGSIRestHandler] Received data ({  "subscriptionId" : "51c0ac9ed714fb3b37d7d5a8",  "originator" : "localhost",  "contextResponses" : [    {      "contextElement" : {        "attributes" : [          {            "name" : "speed",            "type" : "float",            "value" : "120"          },          {            "name" : "the_geom",            "type" : "geo:point",            "value" : "40.40, -3.4"          }        ],        "type" : "Car",        "isPattern" : "false",        "id" : "Car1"      },      "statusCode" : {        "code" : "200",        "reasonPhrase" : "OK"      }    }  ]})
time=2016-12-02T13:48:36.404UTC | lvl=INFO | corr=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | trans=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | srv=<my_user> | subsrv=/test | comp=cygnus-ngsi | op=persistRawAggregation | msg=com.telefonica.iot.cygnus.sinks.NGSICartoDBSink[553] : [raw-sink] Persisting data at NGSICartoDBSink. Schema (<my_user>), Table (x002ftestxffffx0043ar1xffffx0043ar), Data (('2016-12-02T13:48:27.381Z','/test','Car1','Car',ST_SetSRID(ST_MakePoint(40.40,-3.4), 4326),'120','[]'))
time=2016-12-02T13:48:38.237UTC | lvl=INFO | corr=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | trans=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | srv=<my_user> | subsrv=/test | comp=cygnus-ngsi | op=processNewBatches | msg=com.telefonica.iot.cygnus.sinks.NGSISink[514] : Finishing internal transaction (eb73b8d5-af9b-48ea-8ce7-ff21edc957f3)

Getting the data:

$ curl -X GET -G "https://<my_user>.cartodb.com/api/v2/sql?api_key=<my_key>" --data-urlencode "q=select * from x002ftestxffffx0043ar1xffffx0043ar"
{"rows":[{"recvtime":"2016-12-02T13:48:27.381Z","fiwareservicepath":"/test","entityid":"Car1","entitytype":"Car","speed":120,"speed_md":"[]","the_geom":"0101000020E610000033333333333344403333333333330BC0"}],"time":0.001,"fields":{"recvtime":{"type":"string"},"fiwareservicepath":{"type":"string"},"entityid":{"type":"string"},"entitytype":{"type":"string"},"speed":{"type":"number"},"speed_md":{"type":"string"},"the_geom":{"type":"geometry"}},"total_rows":1}

EDIT 2

This answer is only valid if you own an "enterprise" Carto account. Please, see my other answer to this question.

查看更多
0人赞 添加讨论(0) 举报
登录 后发表回答
相关问题
查看全部
相关文章
查看全部
收藏的人(5)