Loop through CSV file with batch - Space issue

Question

I have a csv file like this

name,sex,age
venu,m,16
test,,22

[EDIT]
name could have comma also

"venu,gopal",m,16

I want to handle if sex is nothing and save it to another file.

I have a script like this

@Echo Off
For /F "usebackq tokens=1-3  delims=," %%a in (test.csv) Do (
   echo %%a, %%b, %%c >> test-new.csv
)

But for the third record, I am getting %%b as 22 which should be space. How to fix this?

[EDIT2]

I have tried as per that link. I am not sure what I am doing wrong. I am getting same issue. Please check it once.

@echo off
setlocal DisableDelayedExpansion

For /F "usebackq tokens=1-3 delims=" %%x in (C:\somefile.csv) Do (

    setlocal EnableDelayedExpansion
    set "var=%%x"   

    set "var=!var:"=""!"
    set "var=!var:^=^^!"
    set "var=!var:&=^&!"
    set "var=!var:|=^|!"
    set "var=!var:<=^<!"
    set "var=!var:>=^>!"
    set "var=!var:,=^,^,!"
    set var=!var:""="!
    set "var=!var:"=""Q!"
    set "var=!var:,,="S"S!"
    set "var=!var:^,^,=,!"
    set "var=!var:""="!"
    set "var=!var:"Q=!"


    For /F "tokens=1-3  delims=," %%a in ("!var:"S"S=","!") Do (
      endlocal
      echo %%~a, %%~b, %%~c
      setlocal EnableDelayedExpansion
      pause
    )

    endlocal

)

Answer 1

This is a bit tricky, as multiple delims will be condensed to a single delim.

So you need to replace them before to a unique delim sequence.

@Echo Off
setlocal DisableDelayedExpansion
For /F "usebackq tokens=1-3  delims=" %%a in (test.csv) Do (
    set "line=%%a"
    setlocal EnableDelayedExpansion
    set "line="!line:,=","!""
    For /F "tokens=1-3  delims=," %%a in ("!line!") Do (
        echo %%~a, %%~b, %%~c 
    )
)

This enclose each column into quotes, and with the %%~a the quotes will be removed later

EDIT: The solution for embedded commas

In this case it's only a bit different than the solution for how to split on ';' in CMD shell

@echo off
setlocal DisableDelayedExpansion

For /F "usebackq tokens=1-3 delims=" %%x in (test.csv) Do (
    set "var=%%x"
    setlocal EnableDelayedExpansion
    set "var=!var:^=^^!"
    set "var=!var:&=^&!"
    set "var=!var:|=^|!"
    set "var=!var:<=^<!"
    set "var=!var:>=^>!"
    set "var=!var:,=^,^,!"
    rem ** This is the key line, the missing quote is intention
    call set var=%%var:""="%%
    set "var=!var:"="Q!"
    set "var=!var:^,^,="C!"
    set "var=!var:,,=,!"
    set "var=!var:""="!"

    set "var="!var:,=","!""
    for /F "tokens=1-3 delims=," %%a in ("!var!") do (      
        endlocal
        set "col1=%%~a"
        set "col2=%%~b"
        set "col3=%%~c"
        setlocal EnableDelayedExpansion
        if defined col1 (
            set "col1=!col1:"C=,!"
            set "col1=!col1:"Q="!"
        )

        if defined col2 (
            set "col2=!col2:"C=,!"
            set "col2=!col2:"Q="!"
        )

        if defined col3 (
            set "col3=!col3:"C=,!"
            set "col3=!col3:"Q="!"
        )
        echo a=!col1!, b=!col2!, c=!col3!
        endlocal
    )
)

Answer 2

using a string editor like SSED, I have overcome this issue by creating a temp file where I have replaced ",," with ",-," twice, then regarding a "-" as having been an empty field in the original file...

 ssed "s/,,/,-,/ig;s/,,/,-,/ig" file1.csv > file1.tmp

Then the tokens are allocated correctly. If you need to edit the temp file and then return to a CSV, then use...

 ssed "s/,-,/,,/ig;s/,-,/,,/ig" file1.tmp > file1.csv

This seems much simpler than doing in flight string replacements by token and having subroutines/etc.