Apache HttpClient - post request to ETools.ch with

Question

The code works fine if the query does not contain any utf-8 chars. As soon as there is one utf-8 char then ETools provides results I do not expect. For example for "trees" I get correct result and for "bäume" (german word for trees) I get strange results. It looks like that ETools receives the query as "b%C3%A4ume" and looks for exact that query with exact those chars and not for "bäume". I think the problem may be solved if I set some header parameters but I dont know what parameters are possible there.

String query = "some+query+with+utf8+chars";

HttpClient client = new DefaultHttpClient();
HttpPost request = new HttpPost();

List<NameValuePair> parameters = new ArrayList<NameValuePair>();
parameters.add(new BasicNameValuePair("query", query));
parameters.add(new BasicNameValuePair("country", "web"));
parameters.add(new BasicNameValuePair("language", "all"));
parameters.add(new BasicNameValuePair("dataSourceResults", String.valueOf(40)));
parameters.add(new BasicNameValuePair("pageResults", String.valueOf(40)));
request.setEntity(new UrlEncodedFormEntity(parameters, "UTF-8"));
request.setHeader("Content-Type", "application/x-www-form-urlencoded");
request.setURI("http://www.etools.ch/searchAdvancedSubmit.do?page=2");

MyResponse myResponse = client.execute(request, myResponseHandler);

request.reset();
client.getConnectionManager().shutdown();

Answer 1

You should add your charset into the Content-Type at least (the default is latin1):

request.setHeader("Content-Type", "application/x-www-form-urlencoded; charset=UTF-8");

If that doesn't work, it could be a server bug. You may want to try submitting the form as multipart/form-data (RFC 2388) instead of URL encoded. There is already a StackOverflow answer with an example that you can use.