Writing OpenTSDB to Bigtable with HTTP POST not wo

Updated with more information

I am trying to set up OpenTSDB on Bigtable, following this guide: https://cloud.google.com/solutions/opentsdb-cloud-platform

Works well, all good.

Now I was trying to open the opentsdb-write service with a LoadBalancer (type). Seems to work well, too.

Note: using a GCP load balancer.

I am then using insomnia to send a POST to the ./api/put endpoint - and I get a 204 as expected (also, using the ?details shows no errors, neither does the ?sync) (see http://opentsdb.net/docs/build/html/api_http/put.html)

When querying the data (GET on ./api/query), I don't see the data (same effect in grafana). Also, I do not see any data added in the tsdb table in bigtable.

My conclusion: no data is written to Bigtable, although tsd is returning 204.

Interesting fact: the metric is created (I can see it in Bigtable (cbt read tsdb-uid) and also the autocomplete in the opentsdb-ui (and grafana) pick the metric up right away. But no data.

When I use the Heapster-Example as in the tutorial, it all works.

And the interesting part (to me):

NOTE: It happened a few times, with massive delay or after stoping/restarting the kubernetes cluster, that the data appeared. Suddenly. I could not reproduce as of now.

I must be missing something really simple.

Note: I don't see any errors in the logs (stackdriver) and UI (opentsdb UI), neither bigtable, nor Kubernetes, nor anything I can think of.

Note: the configs I am using are as linked in the tutorial.

The put I am using (see the 204):

and if I add ?details, it indicates success:

My guess is that this relates to the opentsdb flush frequency. When a tsdb cluster is shutdown, there's an automatic flush. I'm not 100% sure, but I think that the tsd.storage.flush_interval configuration manages that process.

You can reach the team that maintains the libraries via the google-cloud-bigtable-discuss group, which you can get to from the Cloud Bigtable support page for more nuanced discussions.

As an FYI, we (Google) are actively updating the https://cloud.google.com/solutions/opentsdb-cloud-platform to the latest versions of OpenTSDB and AsyncBigtable which should improve performance at high volumes.