WSO2ESB Rest API error: Invalid UTF-8 middle byte

Question

I developed a REST API in wso2esb 5.0.0 and I am facing some problems when client send a request that contains ã, ç or é for example, the first mediator on inSequence pipeline, Raise an error about Invalid UTF-8 middle byte.

When we force charset of message in HTTP header as ISO-8859-1, works fine. But we need to work with UTF-8.

Somebody already face this issue and could help me?

Following the stack trace off error:

TID[-1234] [ESB] [2017-07-04 15:12:26,738] ERROR {API_LOGGER.ResourceAPI} - Could not build full log message: [com.ctc.wstx.exc.WstxLazyException] Invalid UTF-8 middle byte 0x6f (at char #143, byte #127) com.ctc.wstx.exc.WstxLazyException.throwLazily(WstxLazyException.java:45) com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:720) com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3677) com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:858) org.apache.axiom.util.stax.wrapper.XMLStreamReaderWrapper.getText(XMLStreamReaderWrapper.java:164) org.apache.axiom.om.impl.builder.StAXBuilder.createOMText(StAXBuilder.java:289) org.apache.axiom.om.impl.builder.StAXBuilder.createOMText(StAXBuilder.java:250) org.apache.axiom.om.impl.builder.StAXOMBuilder.next(StAXOMBuilder.java:252) org.apache.axiom.om.impl.llom.OMElementImpl.buildNext(OMElementImpl.java:653) org.apache.axiom.om.impl.llom.OMElementImpl.getFirstOMChild(OMElementImpl.java:670) org.apache.axiom.om.impl.llom.OMElementImpl.getChildren(OMElementImpl.java:352) org.apache.axiom.om.impl.util.OMSerializerUtil.serializeChildren(OMSerializerUtil.java:553) org.apache.axiom.om.impl.llom.OMElementImpl.internalSerialize(OMElementImpl.java:875) org.apache.axiom.om.impl.util.OMSerializerUtil.serializeChildren(OMSerializerUtil.java:555) org.apache.axiom.om.impl.llom.OMElementImpl.internalSerialize(OMElementImpl.java:875) org.apache.axiom.om.impl.util.OMSerializerUtil.serializeChildren(OMSerializerUtil.java:555) org.apache.axiom.om.impl.llom.OMElementImpl.internalSerialize(OMElementImpl.java:875) org.apache.axiom.soap.impl.llom.SOAPEnvelopeImpl.internalSerialize(SOAPEnvelopeImpl.java:230) org.apache.axiom.om.impl.llom.OMSerializableImpl.serialize(OMSerializableImpl.java:125) org.apache.axiom.om.impl.llom.OMSerializableImpl.serialize(OMSerializableImpl.java:113) org.apache.axiom.om.impl.llom.OMElementImpl.toString(OMElementImpl.java:988) java.lang.String.valueOf(String.java:2994) java.lang.StringBuffer.append(StringBuffer.java:265) org.apache.synapse.mediators.builtin.LogMediator.getFullLogMessage(LogMediator.java:203) org.apache.synapse.mediators.builtin.LogMediator.getLogMessage(LogMediator.java:138) org.apache.synapse.mediators.builtin.LogMediator.mediate(LogMediator.java:101) org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:97) org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:59) org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:158) org.apache.synapse.rest.Resource.process(Resource.java:343) org.apache.synapse.rest.API.process(API.java:399) org.apache.synapse.rest.RESTRequestHandler.apiProcess(RESTRequestHandler.java:123) org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:101) org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:69) org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:304) org.apache.synapse.core.axis2.SynapseMessageReceiver.receive(SynapseMessageReceiver.java:75) org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180) org.apache.axis2.transport.http.util.RESTUtil.invokeAxisEngine(RESTUtil.java:144) org.apache.axis2.transport.http.util.RESTUtil.processXMLRequest(RESTUtil.java:89) org.apache.synapse.transport.nhttp.util.RESTUtil.processPOSTRequest(RESTUtil.java:213) org.apache.synapse.transport.nhttp.ServerWorker.processEntityEnclosingMethod(ServerWorker.java:468) org.apache.synapse.transport.nhttp.ServerWorker.run(ServerWorker.java:291) org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) 

Note: When we use common SOAP proxy and messages works fine. No problem with charset.

[EDITED]

Note: Like @weibeld said, my language is Portuguese.

My REST API has a resource with POST method and I am sending the following request to server:

POST http://myHost:8280/api/v1/myResource HTTP/1.1
Accept-Encoding: gzip,deflate
Content-Type: application/json;charset=utf-8
Content-Length: 741
Host: myHost:8280
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)

 {
"customerName": "CONSULTORIA EMPRESARIAL EIRELI - ME",
"serviceId": "05948517",
"productName": "Banda Larga Expansão Sul",
"localityCode": "51000",
"status": "OCUPADO",
"isChangeOfAddress": "FALSO",
"serial": "0047848517",
"vlanType": "1Q",
"ont_id": "0047848517",
"modelo_ont": "ZHONE-2425",
"accessType": "GPON",
"streetCode": "4535",
"neighbourhood": "NEIGHBOURHOOD",
"streetNumber": "4428",
"complement1": "2"

}

Note: In this line "productName": "Banda Larga Expansão Sul", I have an ão.

Summarizing, I am sending a POST, wiht application/json;charset=utf-8 and the wso2esb is raising an error about the ão, and when I sent an ISO-8859-1 works fine. (when I use soap with utf-8 works properly).

Answer 1

It seems that the the HTTP request that your client sends to the server is encoded with ISO-8859-1 rather than with UTF-8.

The server then tries to decode this request with UTF-8. This works well for all ASCII characters. However, at some point the server encounters the ISO 8859-1 code of one of your reported non-ASCII characters:

• ã ==> 0xE3 = 11100011
• ç ==> 0xE7 = 11100111
• é ==> 0xE9 = 11101001

(Read the above listing as: [character] ==> [ISO 8859-1 code hex.] = [ISO 8859-1 code bin.])

Now UTF-8 interprets this byte as the leading byte of a 3-byte UTF-8 code because it starts with 1110 (see here). Consequently, UTF-8 expects the next byte to be a UTF-8 continuation byte, which must start with 10. However, the next byte is 0x6f, the ISO 8859-1 code for the ASCII character o, which is 01101111 in binary, i.e. does not start with 10.

Hence the error message Invalid UTF-8 middle byte 0x6f.

So, I guess that your request contains the substring ão, ço, or éo, which is likely if your language is Portuguese.

In summary: you need to make sure that the client encodes the request with UTF-8. I don't know how you create the request, but there should be a possibility to set the encoding in whatever function you use.