FileSystemwatcher with Documents Indexing as Attac

Question

I'm trying to reproduce a console application which watches a folder and any new additions of documents to the folder are to be Indexed to ES .

It is working fine If I move/add 3-4 documents at a time and able to index. But if I move around 30 documents at a time, It is not indexing all the documents, instead indexing only one. But If I run the code with break points , then even 30 documents are also getting indexed. Can some one help me in solving this.

 static void OnCreated(object sender, FileSystemEventArgs e)
        {
            Console.WriteLine("File Created: Path: {0}, \n Name: {1}", e.FullPath, e.Name);
            Indexdoc(e.FullPath);
        }

If I dont call the Indexdoc(e.FullPath) method in the above code and instead print the changes, it is showing all the filenames added perfectly. so there is no problem with the filesystemwatcher. I think indexing documents is taking time to generate response and come back to onCreated method.

    public static void Indexdoc(string newFilePath)
            {
                List<Document> list = new List<Document>(); //list of Document class objects
                List<string> filesList = new List<string>(); //list of files in the path received on method call
                string path = string.Empty;

                client = ConfigSettings.connection();

                if (newFilePath == null) //for FULL Indexing
                {
                   //some code here
                }
                else //for new files indexing
                {
                    filesList.Add(newFilePath); //adds only one file everytime the method is called.
                    //the newFilePath will be of type C:/Documents/abc.txt
                }

                try
                {
                    foreach (string file in filesList)
                    {
                        Attachment attach = new Attachment
                        {
                            Name = Path.GetFileNameWithoutExtension(file),
                            Content = Convert.ToBase64String(File.ReadAllBytes(file)),
                            ContentType = Path.GetExtension(file)
                        };

                        var doc = new Document()
                        {
                            Title = Path.GetFileNameWithoutExtension(file),
                            FilePath = Path.GetFullPath(file), //added to get the path of the file
                            File = attach
                        };

                        list.Add(doc);
                    }
                }
                catch (Exception e)
                {
                    Console.WriteLine(e.Message.ToString());
                }

                var response = client.IndexMany(list, "trial");
            }

Can some one help me in solving this.

TIA

Answer 1

First, create a queue that you'll use to put the information in. This is at class scope:

private BlockingCollection<string> CreatedQueue = new BlockingCollection<string>();

At startup, you want to create a Task that watches that queue for changes:

var queueProcessor = Task.Factory.StartNew(() => ProcessQueue, TaskCreationOptions.LongRunning);

Your OnCreated event handler looks like this:

static void OnCreated(object sender, FileSystemEventArgs e)
{
    Console.WriteLine("File Created: Path: {0}, \n Name: {1}", e.FullPath, e.Name);
    CreatedQueue.Add(e.FullPath);
}

And the ProcessQueue method looks like this:

void ProcessQueue()
{
    foreach (var fileName in CreatedQueue.GetConsumingEnumerable())
    {
        Indexdoc(fileName);
    }
}

The ProcessQueue method will continue removing items from the queue (or waiting for items to be added, so that it can remove them) until the queue is marked as completed. That is, you say you're done adding items and the queue is empty. So when you shut down, you have to call CompleteAdding, and then make sure the queue is empty before you exit. To do that, your main program does this:

// Tell the queue that no more items will be added
CreatedQueue.CompleteAdding();

// Wait for the task to complete
queueProcessor.Wait();

Answer 2

I'm taking input from users to exit the application. Should I add these codes below there?

// Tell the queue that no more items will be added
CreatedQueue.CompleteAdding();

// Wait for the task to complete
queueProcessor.Wait();

This is the code I modified.

    public class FileWatcher
    {
        public static ElasticClient client;
        public static FileSystemWatcher watcher;
        public static BlockingCollection<string> CreatedQueue = new BlockingCollection<string>();

        public static void Main(string[] args)
        {
            var queueProcessor = Task.Factory.StartNew(() => ProcessQueue(), TaskCreationOptions.LongRunning);
            watch();

        }

        public static void watch()
        {
            folderPath = ConfigurationManager.AppSettings["location"];


            try
            {
                watcher = new FileSystemWatcher();
                watcher.Path = folderPath;

                watcher.EnableRaisingEvents = true;
                watcher.IncludeSubdirectories = true;
                watcher.Filter = "*.*";


                watcher.Created += OnCreated;

                Console.WriteLine("*****************************************************************");
                // Wait for user to quit program.
                Console.WriteLine("Press \'q\' to quit the watcher application at any point of time.");
                Console.WriteLine();

                //Make an infinite loop till 'q' is pressed.
                while (Console.Read() != 'q') ;
            }
            catch (IOException e)
            {
                Console.WriteLine("A Exception Occurred :" + e);
            }

            catch (Exception oe)
            {
                Console.WriteLine("An Exception Occurred :" + oe);
            }
        }

        static void OnCreated(object sender, FileSystemEventArgs e)
        {
            Console.WriteLine("File Created: Path: {0}, \n Name: {1}", e.FullPath, e.Name);
            CreatedQueue.Add(e.FullPath);
        }

        public static void ProcessQueue()
        {
            foreach (var fileName in CreatedQueue.GetConsumingEnumerable())
            {
                Indexdoc(fileName);
            }
        }
    }

    public static void Indexdoc(string newFilePath)
    {}