Application Insights purge data from custom data s

I load custom data into an application insights app via a Azure Function programatically on a schedule. The workflow is essentially loading a json file onto an azure storage account, calling post to a particular endpoint with body containing link with sas token to json file, and then the ingest cycle occurs loading the json into the custom data source. What I have noticed is that the ingestion cycle concatonates old records with the new JSON - and I need to PURGE the data before triggering a new ingestion so the JSON file always represents the complete state of the data set.

Is there any API used to purge data from an application insights custom data source programatically ?

Yes, it is possible to purge Application Insights data, but it may take a while (e.g. 2-3 days) for the operation to complete.

This is accomplished through sending a POST request to the Azure Management API as follows:

--- Request URL (POST) ---

https://management.azure.com/subscriptions/{Subscription Id (GUID)}/resourceGroups/{Resource Group Name}/providers/Microsoft.Insights/components/{Application Insights Name}/purge?api-version=2015-05-01

--- Request Body ---

{
  "table": "exceptions",
  "filters": [
    {
      "column": "timestamp",
      "operator": ">",
      "value": "2018-01-01"
    }
  ]
}

exceptions is the name of the table to where data will be deleted according to filter.

--- Request Headers ---

Authorization: Bearer {OAuth Access Token}

Navigate to the Azure Portal at http://portal.azure.com, open the Cloud Shell and run the following command to obtain an OAuth Access Token:

az account get-access-token

--- Response ---

{
    "operationId": "purge-048ccace-a6a0-41b9-80e3-fbc11a5bdd64"
}

--- Activity Log ---

An event will be recorded in the Activity Log with details about the operation.

--- Available Tables ---

The available tables (including their schema) for both Application Insights and Other Data Sources are available in the Analytics page:

Note that this process is asynchronous and it may take a while, it is possible to query its status through the following GET request:

--- Request URL (GET) ---

https://management.azure.com/subscriptions/{Subscription Id (GUID)}/resourceGroups/{Resource Group Name}/providers/Microsoft.Insights/components/{Application Insights Name}/operations/{purge-GUID (response returned in the purge POST request}?api-version=2015-05-01

--- Request Headers ---

Authorization: Bearer {OAuth Access Token}

--- Response ---

{
    "status": "pending"
}

Find more details at https://docs.microsoft.com/en-us/rest/api/application-insights/components/purge.

Here is another interesting thread about this feature https://feedback.azure.com/forums/357324-application-insights/suggestions/19254595-enable-to-clear-data-of-the-resource.