My application needs to process a list of urls, by retrieving their text content. My internet connection is started by
myTimeOut:= 2000;
InternetSetOption(nil, INTERNET_OPTION_CONNECT_TIMEOUT, Pointer(@myTimeOut), SizeOf(myTimeOut));
InternetSetOption(nil, INTERNET_OPTION_SEND_TIMEOUT, Pointer(@myTimeOut), SizeOf(myTimeOut));
InternetSetOption(nil, INTERNET_OPTION_RECEIVE_TIMEOUT, Pointer(@myTimeOut), SizeOf(myTimeOut));
I'm launching separate threads for each url reading
(...)
read_url_threads[thread_counter]:= TReadURLThread.Create(false);
read_url_threads[thread_counter].FreeOnTerminate:= False;
read_url_threads[thread_counter].myURL:= target_url_list[i];
read_url_threads[thread_counter].output_filename:= local_output_filename;
read_url_threads[thread_counter].NetHandle:= NetHandle;
read_url_threads[thread_counter].limit_text_size:= max_length_url;
hArr[thread_counter]:= read_url_threads[thread_counter].Handle;
(...)
if (thread_counter >= max_threads) or
(thread_counter >= (target_url_list.Count)) then
repeat
rWait:= WaitForMultipleObjects(thread_counter, @hArr,True, 100);
Application.ProcessMessages;
until rWait <> WAIT_TIMEOUT;
(...)
Inside the thread execution I retrieve the url content by:
if Assigned(NetHandle) and (not EndsText('.pdf',url)) then
try
UrlHandle := InternetOpenUrl(NetHandle, PChar(url), nil, 0, 0, 0);
if Assigned(UrlHandle) then
try
repeat
InternetReadFile(UrlHandle, @Buffer, SizeOf(Buffer), BytesRead);
SetString(StrBuffer, PAnsiChar(@Buffer[0]), BytesRead);
Result := Result + StrBuffer;
until BytesRead = 0;
finally
InternetCloseHandle(UrlHandle);
end;
My problem is setting the timeout for the InternetReadFile routine. With some urls it gets stuck sometimes for whole minutes. What would be the best way to force a timeout in this cycle?