Number of tokens limit in a FOR command in a Windo

Question

I was trying to process a text file in a Windows batch script and I ran into something that looks like a limitation to 31 tokens in a FOR loop. I isolated the issue in the code below:

@ECHO OFF
SET DATA=01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

FOR /F "tokens=31* delims= " %%i IN ("%DATA%") DO (
    ECHO [%%i]
    ECHO [%%j]
)
ECHO.
FOR /F "tokens=32* delims= " %%i IN ("%DATA%") DO (
    ECHO [%%i]
    ECHO [%%j]
)

The output is:

[31]
[32 33 34 35]

[01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35]
[%j]

and I was expecting this:

[31]
[32 33 34 35]

[32]
[33 34 35]

Hoping that I haven't been doing something wrong, I couldn't find this limitation documented in the help for the FOR command. I'm using Windows XP. Do you know any workaround for this, aside from chopping off parts of the data?

Thank you.

Answer 1

There is more than one way to loop through the DATA array:

@ECHO OFF
setlocal ENABLEDELAYEDEXPANSION
SET DATA=01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101

for /l %%a in (0,3,150) do @echo %%a data: !data:~%%a,2!

This is possible only because of the two character variables in the %DATA% array. As you will see, the limit is now 99 characters, rather than 31. When you get to the 100th and beyond, the number is truncated.

Rob

Answer 2

A token is the smallest unit of syntax that counts as one chunk. And a Batch command line in Windows 95/98/ME has a maximum limit of 64 tokens. Any more, and the command line will generate a Bad command or file name error.

which is why you're probably limited to 31 within DATA.

Answer 3

I came up with a solution. It's not elegant, but it solves my problem. When the commmand line interpreter cannot go further with the tokens, I pass the remaning of the data to a CALL :label command. Here is an example:

@ECHO OFF

SET DATA=01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100

FOR /F "tokens=1,31* delims= " %%i IN ("%DATA%") DO (
    ECHO  1st token: %%i
    ECHO 31th token: %%j
    CALL :processdatatokens32-62 %%k
)

:processdatatokens32-62
SET DATA=%*
FOR /F "tokens=1,31* delims= " %%i IN ("%DATA%") DO (
    ECHO 32nd token: %%i
    ECHO 62th token: %%j
    CALL :processdatatokens63-83 %%k
)
GOTO :EOF

:processdatatokens63-83
SET DATA=%*
FOR /F "tokens=1,31* delims= " %%i IN ("%DATA%") DO (
    ECHO 63th token: %%i
    ECHO 93th token: %%j
)
GOTO :EOF

The output is:

 1st token: 01
31th token: 31
32nd token: 32
62th token: 62
63th token: 63
93th token: 93

Answer 4

I've created a typewriter function for batch (just for fun) and also encountered this limitation. Below a stripped-down version to show you how I fixed the problem. You can use up to 8,191 chars including space-separators when executed from the command line. When using a variable the maximum length is 32,767.

@echo off
::#############################################################################
 :_typeWriter <stringVariable> <skipLinefeed>
::#############################################################################
::
:: Copyleft Denny Lenselink 2018
::

%= Set local environment =%
( call ) & 2>nul setlocal EnableDelayedExpansion || exit /b 99

:: Set vars
set "_CNT=0" && set "_LEN=0" && set "_STR= %~1" && set "_TOT=0"

:_typeWriter_Loop
set /a "_CNT+=1"

:: 31 tokens limit fix. Cut the used part of the string and set counter to 1
if !_CNT! equ 32 ( set "_CNT=1" && set "_STR=!_STR:~%_TOT%!" && set "_TOT=0" )

:: Go through string (seeking words)
for /f "tokens=%_CNT% delims= " %%* in ( "%_STR%" ) do (

    :: Set word var
    set "_WRD=#%%*"

    :: Calculate word length
    for /l %%I in ( 12, -1, 0 ) do (

        set /a "_LEN|=1<<%%I"

        for %%J in ( !_LEN! ) do ( if "!_WRD:~%%J,1!"=="" set /a "_LEN&=~1<<%%I" )
    )

    :: Strip first char (used before to calculate correct word length)
    set "_WRD=!_WRD:~1!"

    :: Count chars including spaces
    set /a "_TOT=!_TOT!+!_LEN!+1"

    :: Type word or use echo
    <nul set /p "=!_WRD! "

    :: Do a loop
    goto _typeWriter_Loop
)

:: No linefeed when specified
if "%~2"=="" echo.

endlocal
exit /b

Execute: C:\Temp> typewriter "this is a test that works"

Result:

this is a test that works

Changing the line <nul set /p "=!_WRD! " to echo !_WRD! will result in:

this
is
a
test
that
works

Answer 5

From for /?:

%i is explicitly declared in the for statement and the %j and %k are implicitly declared via the tokens= option. You can specify up to 26 tokens via the tokens= line, provided it does not cause an attempt to declare a variable higher than the letter 'z' or 'Z'. Remember, FOR variables are single-letter, case sensitive, global, and you can't have more than 52 total active at any one time.