Loop through sub folders in a directory and import

Question

I am trying to loop through a specific directory's sub folders and import specified columns from .CSV files.

I have a coding solution that does not loop through the sub folders.

Instead, it includes a Worksheet with File Path, File Destination and Column Number in three separate columns, but the sub folders are dynamic. They are changing in name and quantity.

File Path sheet:

Code:

Dim DL As Worksheet
Dim DFI As Worksheet

Set DL = ThisWorkbook.Sheets("DataList")
Set DFI = ThisWorkbook.Sheets("DataFeedInput")

    DL.Rows("$3:$202").ClearContents

        With DL.QueryTables.Add(Connection:="TEXT;C:\Users\ ... \MQL4\Files\Hist_#Corn_1440.csv", Destination:=Range("$A$3"))
            .Name = "Hist_#Corn_1441"
            .FieldNames = True
            .RowNumbers = False
            .FillAdjacentFormulas = False
            .PreserveFormatting = True
            .RefreshOnFileOpen = False
            .RefreshStyle = xlInsertDeleteCells
            .SavePassword = False
            .SaveData = True
            .AdjustColumnWidth = True
            .RefreshPeriod = 0
            .TextFilePromptOnRefresh = False
            .TextFilePlatform = 866
            .TextFileStartRow = 1
            .TextFileParseType = xlDelimited
            .TextFileTextQualifier = xlTextQualifierDoubleQuote
            .TextFileConsecutiveDelimiter = False
            .TextFileTabDelimiter = True
            .TextFileSemicolonDelimiter = False
            .TextFileCommaDelimiter = True
            .TextFileSpaceDelimiter = False
            .TextFileColumnDataTypes = Array(9, 1, 9, 9, 9, 9, 9, 1, 9, 9, 9, 9, 9, 9, 9)
            .TextFileTrailingMinusNumbers = True
            .Refresh BackgroundQuery:=False
        End With

    Dim i As Integer

    For i = 4 To 642

    Dim FileName As String
    Dim OutputSheet As String
    Dim ColNumber As String

        FileName = DFI.Range("B" & i).Value
        OutputSheet = DFI.Range("C" & i).Value
        ColNumber = DFI.Range("D" & i).Value

            With DL.QueryTables.Add(Connection:="TEXT;" & FileName, Destination:=DL.Range(ColNumber & "3"))
                 .FieldNames = True
                 .RowNumbers = False
                 .FillAdjacentFormulas = False
                 .PreserveFormatting = True
                 .RefreshOnFileOpen = False
                 .RefreshStyle = xlInsertDeleteCells
                 .SavePassword = False
                 .SaveData = True
                 .AdjustColumnWidth = True
                 .RefreshPeriod = 0
                 .TextFilePromptOnRefresh = False
                 .TextFilePlatform = 866
                 .TextFileStartRow = 1
                 .TextFileParseType = xlDelimited
                 .TextFileTextQualifier = xlTextQualifierDoubleQuote
                 .TextFileConsecutiveDelimiter = False
                 .TextFileTabDelimiter = True
                 .TextFileSemicolonDelimiter = False
                 .TextFileCommaDelimiter = True
                 .TextFileSpaceDelimiter = False
                 .TextFileColumnDataTypes = Array(9, 9, 9, 9, 9, 9, 9, 1, 9, 9, 9, 9, 9, 9, 9)
                 .TextFileTrailingMinusNumbers = True
                 .Refresh BackgroundQuery:=True
             End With

    Next i

        DL.Cells.EntireColumn.AutoFit

The problem with this approach is that if a .CSV file is not downloaded from the external source, I get an error stating that the file is missing.

Another issue is that this approach takes decades to finish the task.

I am looking for a solution that is not dependent on the File Path sheet, loops through the sub folders and extracts solely column 6 from the .CSV file.

In each of these folders I have one .CSV file:

I need to loop through each of them and create connection to Excel sheet, while importing solely column 6 from the .CSV.

Edit 1:

This is the File Path to the Sub Folders:

C:\Users\Betty\AppData\Roaming\MetaQuotes\Terminal\B4D9BCD10BE9B5248AFCB2BE2411BA10\MQL4\Files\Export_History

Edit 2:

What I learned so far, with the help of @Jeeped, is that I can loop through the folders with FileSystemObject, probably, go in to each of the folders and import column 6 from the .CSV.

It is quite difficult for me to get into how to merge the loop trough the folders and the .CSV import. If you can give me a hand with an outline procedure, I think I will be able to put it together and add it as edit to this question, if needed.

Edit 3:

I reckon I can use something of such for completing the task:

Code from @Tim Williams' answer to this question -> VBA macro that search for file in multiple subfolders

Sub GetSubFolders()

    Dim fso As New FileSystemObject
    Dim f As Folder, sf As Folder

    Set f = fso.GetFolder("file path")
    For Each sf In f.SubFolders

        'Use a loop to import only column 6 from every .CSV file in sub folders 

    Next

End Sub

Answer 1

@QHarr: Special thanks for the guidance!

After looking in to the FileSystemObject method for the purpose of looping trough Sub Folders and importing column 6 from a .CSV file in each Sub Folder in the next blank column in Worksheet HDaER, I managed to put together this code:

    Dim fso As Object
    Dim folder As Object
    Dim subfolders As Object
    Dim CurrFile As Object
    Dim HDaER As Worksheet

With Application
    .ScreenUpdating = False
    .EnableEvents = False
    .Calculation = xlCalculationManual
End With

    Set fso = CreateObject("Scripting.FileSystemObject")
    Set folder = fso.GetFolder("C:\Users\Betty\AppData\Roaming\MetaQuotes\Terminal\B4D9BCD10BE9B5248AFCB2BE2411BA10\MQL4\Files\Export_History\")
    Set subfolders = folder.subfolders
    Set HDaER = Sheets("HDaER")

'   IMPORT Col 6 FROM EACH .CSV FILE IN EACH SubFolder    
    LastCol = HDaER.Cells(2, HDaER.Columns.Count).End(xlToLeft).Column

    For Each subfolders In subfolders

    Set CurrFile = subfolders.Files
        For Each CurrFile In CurrFile
            With HDaER.QueryTables.Add(Connection:="TEXT;" & CurrFile, Destination:=HDaER.Cells(2, LastCol + 1))
                 .TextFileStartRow = 1
                 .TextFileParseType = xlDelimited
                 .TextFileConsecutiveDelimiter = False
                 .TextFileTabDelimiter = False
                 .TextFileSemicolonDelimiter = False
                 .TextFileCommaDelimiter = True
                 .TextFileSpaceDelimiter = True
                 .TextFileColumnDataTypes = Array(9, 9, 9, 9, 9, 1, 9)
                 .Refresh BackgroundQuery:=False
                 LastCol = LastCol + 1
            End With
        Next
    Next

'   REMOVE SOURCE CONNECTIONS
    For Each Connection In HDaER.QueryTables
        Connection.Delete
    Next Connection

'   FREE MEMORY 
    Set fso = Nothing
    Set folder = Nothing
    Set subfolders = Nothing

With Application
    .ScreenUpdating = True
    .EnableEvents = True
    .Calculation = xlCalculationAutomatic
End With

The Sub Folders that I currently have in the general Folder (Export_History) are:

The output that I get from the code is:

@QHarr: Please, let me know if you see anything that can be improved, especially in the QueryTables.Add part.