Azure Web Jobs suddenly stopped working properly

问题:

I wrote a program named "blobsUploader" that uploads a csv file to a blob container every night at 11pm.

Whenever a new csv file arrives to the blobs container, a new message in a queue named "blobsAdressQueue" appears with the address of the new blob (csv file).

This evokes the Web Job which reads the csv file and stores all it's data in Azure Table named "myDataTable".

The whole process worked great but suddenly from the past month or two, every night when a new csv is uploaded, there is an error with the Web Job process and the message from "blobsAddressQueue" moves to "blobsAddressQueue-poison" which means a message that has exceeded the maximum number of delivery attempts to the application.

I uploaded now a csv from June 2018, that worked for sure. However, now the message with the address the this blob is in "blobsAddressQueue-poison".

When I check the logs I can see 5 failing calls:

When I go into one of the attempts and open "Toggle Output", that is what I get: Which is very weird because this file was read a in June 2018!!! with no any problems! I did not change anything in my code or the csv file since then.

If more information is needed in order to answer the problem, pls let me know.

回答1:

This issue is not related to webjob, but CsvHelper library you reference to. I've checked the source code and found a field will be considered as bad data when it contains quote and the field is not quoted (escaped).

Source code:

/// <summary>
/// Gets or sets the function that is called when bad field data is found. A field
/// has bad data if it contains a quote and the field is not quoted (escaped).
/// You can supply your own function to do other things like logging the issue
/// instead of throwing an exception.
/// Arguments: context
/// </summary>
Action<ReadingContext> BadDataFound { get; set; }

The solution is

Modify the problematic fields in csv file

or

Ingnore the bad data by setting BadDataFound to null:

csv.Configuration.BadDataFound = null;

Sample code：

    static void Main(string[] args)
    {
        using (var reader = new StreamReader(@"C:\Users\toml\Desktop\test.csv"))
        using (var csv = new CsvReader(reader))
        {
            csv.Configuration.BadDataFound = null;
            var records = csv.GetRecords<Foo>();

            foreach(var item in records)
            {
                Console.WriteLine(item.Name);
            }
        }

        Console.ReadKey();
    }
}

public class Foo
{
    public int Id { get; set; }
    public string Name { get; set; }
}

Invalid CSV sample:

    Id,Name
    1,one"
    2,two

Valid CSV sample:

    Id,Name
    1,"one""
    2,two