How to efficiently define End-of-Transmission for

Question

I want to develop a text protocol based on XML and transmitted via TCP/IP sockets. Let's say I have a simple request/response mechanism to be send over a persistent TCP/IP connection between client and server like this:

<?xml version="1.0" encoding="UTF-8"?>
<request id="1" command="get.answer">
    <value type="string">Answer to the Ultimate Question of Life, the Universe, and Everything</value>
</request>

<?xml version="1.0" encoding="UTF-8"?>
<response id="1" command="get.answer">
    <value type="int32">42</value>
</response>

When should each side start to process the incoming data or in other words when would the server know that the incoming client data is fully transfered and possible to process to create a response?

Of course I made some research about that topic: I found this answer which points in the right direction based on an HTTP example: So using a kind of 'Transfer Protocol' on top of the XML messages would certainly help.

But I also looked at the purely XML-based XMPP protocol which doesn't use any 'Transfer Protocol' like HTTP at least as far as I have seen.

From RFC 6120 at "2.4. Structured Data" it reads:

The basic protocol data unit in XMPP is not an XML stream (which simply provides the transport for point-to-point communication) but an XML "stanza", which is essentially a fragment of XML that is sent over a stream. The root element of a stanza includes routing attributes (such as "from" and "to" addresses), and the child elements of the stanza contain a payload for delivery to the intended recipient.

So they send basically small XML chunks over TCP/IP w/o 'Transfer Protocol' and from my wireshark traces I can see that there is also no special End-Of-Transmission character at the end of each XML stanza like two times \r\n or something like that. So how do they know about the end of a message (stanza)?

Answer 1

Actually, XMPP uses an XML stream to transfer data. The data unit you are referring to is the actual exchange of individual messages, but they are all contained within an XML stream that define the start and endpoint of the communication for an XMPP session.

This would be where the End Of Transmission occurs, as in end of all transmission. Within that stream, there are 3 defined packet types (IQ, Message and Presence) which would indicate the start and end of individual messages (for client to server comms).

Although the basic case is done over a TCP connection, there are extensions to support different wireline protocols as well, such as HTTP which is useful for allowing XMPP through a firewall.

If you want to do something similar, then you can follow the same approach, which is to start and end you XML stream when your connection is established and dropped. Then you simply need to define the individual message types, so your endpoints will know what constitutes a complete message.

Or you could just use XMPP which seems to fit your use case perfectly.

Answer 2

An XMPP endpoint have to parse the XML. By doing so, it knows when the end is, as there is only allowed to be 1 document (top level) element (I'm unsure if they can possibly be preceeded by XML processor instructions)

<?xml version="1.0" encoding="UTF-8"?>
<request id="1" command="get.answer">
    <value type="string">Answer to the Ultimate Question of Life, the Universe, and Everything</value>
</request>

This is self delimited, in that once you've parsed the <request markup, you know that this XML documents ends when you hit the matching </request>.

(Personally, I'd place a framing protocol at the protocol level below, instead of stuffing raw xml on top of a (TCP) stream, perhaps just preceed every message with a 4 byte big endian length field.)

Answer 3

XMPP has a transport over XML streams as said by @Robin. It also can use HTTP as a transport with BOSH.

In the second (HTTP) case, things are easy. Strophe for instance, a js library using BOSH, requests are essentially HTTP requests, and thus have Content-Length. It looks like this:

POST /webclient HTTP/1.1
Content-Type: text/xml; charset=utf-8
Content-Length: 240

<body rid='1573741825'
      sid='SomeSID'
      xmlns='http://jabber.org/protocol/httpbind'>
  <iq id='bind_1'
      type='set'
      xmlns='jabber:client'>
    <bind xmlns='urn:ietf:params:xml:ns:xmpp-bind'>
      <resource>httpclient</resource>
    </bind>
  </iq>
</body>

In the first case (XML streams) though things are different. A well-performing, long time in existence and tested python library I use, Twisted, uses a python wrapper on the Expat XML parser. The parser is a fast, non-validating parser that throws useful events indicating the start or end of "root" elements for instance. The elements are then parsed one by one as received.

Answer 4

As it is mentioned in here there mainly two methods: Have a delimiter or the length in header. Your delimiter could simply be the end of your beginning tag and that's what XMPP is doing. This means as long as your XML messages are wrapped in a tag which starts and ends properly you are set to go. If you wanna have a sort of validation on the chunk of data you receive, what you need to do is to make sure that there is an end for all of your tags. Most of the parser packages do this for you. If you pass them a non-parsable package they will throw you a sort of exception. If you wanna write your own parser, then you need to study more about parsers rather than the transfer/XML protocol.