Set timeout for InternetReadFile (WinInet + De

Question

My application needs to process a list of urls, by retrieving their text content. My internet connection is started by

myTimeOut:= 2000;
InternetSetOption(nil, INTERNET_OPTION_CONNECT_TIMEOUT, Pointer(@myTimeOut), SizeOf(myTimeOut));
InternetSetOption(nil, INTERNET_OPTION_SEND_TIMEOUT, Pointer(@myTimeOut), SizeOf(myTimeOut));
InternetSetOption(nil, INTERNET_OPTION_RECEIVE_TIMEOUT, Pointer(@myTimeOut), SizeOf(myTimeOut));

I'm launching separate threads for each url reading

(...)
read_url_threads[thread_counter]:= TReadURLThread.Create(false);
read_url_threads[thread_counter].FreeOnTerminate:= False;
read_url_threads[thread_counter].myURL:= target_url_list[i];
read_url_threads[thread_counter].output_filename:= local_output_filename;
read_url_threads[thread_counter].NetHandle:= NetHandle;
read_url_threads[thread_counter].limit_text_size:= max_length_url;
hArr[thread_counter]:= read_url_threads[thread_counter].Handle;
(...)
if (thread_counter >= max_threads) or
   (thread_counter >= (target_url_list.Count)) then

   repeat
     rWait:= WaitForMultipleObjects(thread_counter, @hArr,True, 100);
     Application.ProcessMessages;
   until rWait <> WAIT_TIMEOUT;
(...)

Inside the thread execution I retrieve the url content by:

   if Assigned(NetHandle) and (not EndsText('.pdf',url)) then
    try
      UrlHandle := InternetOpenUrl(NetHandle, PChar(url), nil, 0, 0, 0);
      if Assigned(UrlHandle) then
        try
          repeat
            InternetReadFile(UrlHandle, @Buffer, SizeOf(Buffer), BytesRead);
            SetString(StrBuffer, PAnsiChar(@Buffer[0]), BytesRead);
            Result := Result + StrBuffer;
          until BytesRead = 0;
        finally
          InternetCloseHandle(UrlHandle);
        end;

My problem is setting the timeout for the InternetReadFile routine. With some urls it gets stuck sometimes for whole minutes. What would be the best way to force a timeout in this cycle?

Answer 1

If the configured timeout has no effect (which is a known bug), then you should close the connection handle, which will force the associated function to fail, thereby returning control to your program. Microsoft article Q224318, "How to control connection timeout value by creating second thread," describes how to do that. (You need a separate thread because the other thread is stuck waiting for InternetReadFile to return.)