Logstash expecting \\n for it to process an XML

问题:

I am trying to parse through an XML file and read only a single tag out of the entire XML using the Grok Patterns.

My grok pattern looks like this. Its able to parse through an XML when its properly indented, as there's a new line after each closing tag. But when the file comes with no spaces in between consecutive tags, this pattern does not work. Could anyone help here?

 input {
      beats {
        port => 5045
        type => 'iis'
      }

    }    
    filter {
        #ignore log comments
        if [message] =~ "^#" {
        drop {}
        }

        grok {
        patterns_dir => "./patterns"
        match => ["message", "%{DATA:extras}<LoadID%{DATA:extra}>%{DATA:ASNNumber}%{GREEDYDATA:behind}"]
        }
        date {
        match => [ "timestamp", "yyyy-MM-dd HH:mm:ss" ]
        locale => "en"
        }
        }

        Second filter
        filter {
        if "_grokparsefailure" in [tags] {
        drop { }
        } else {
        # on success remove the message field to save space
        mutate {
        remove_field => ["message", "timestamp", "extra", "extras", "behind"]
        }
        }
        }

This fails:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?><tns:ASNAck xmlns:tns='http://www.xyx.com/YYY/logistics/mxg/xnsds/V1_0' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'><MessageHeader><MessageID>3031999515</MessageID><MessageTimeStamp>2017-09-12T06:37:36Z</MessageTimeStamp><SenderID>XBHSNS</SenderID><ReceiverID>GOLF_DAO</ReceiverID><MessageType>ACKACKACK</MessageType><CorrelationID>2d323537383935353034383933383135</CorrelationID></MessageHeader><Masterbill>G829441</Masterbill><LoadID>Jitesh555</LoadID><Accept>true</Accept><ReasonCode/><ReasonDescription/></tns:ASNAck>

This works:

<tns:ASNAck xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://www.xyx.com/YYY/logistics/mxg/xnsds/V1_0">
    <MessageHeader>
        <MessageID>20170704080189</MessageID>
        <MessageTimeStamp>2017-07-04T20:17:30Z</MessageTimeStamp>
        <SenderID>KNN_DAO_MXC</SenderID>
        <ReceiverID>GOLF_DAO</ReceiverID>
        <MessageType>InboundASNAck</MessageType>
        <CorrelationID>2d383736363033383337333530313338</CorrelationID>
    </MessageHeader>
    <MasterWaybill>C211</MasterWaybill>
    <LoadID>10112275912A02</LoadID>
    <Accept>true</Accept>
    <ReasonCode>0</ReasonCode>
    <ReasonDescription/>
</tns:ASNAck>

This works too:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?><tns:ASNAck xmlns:tns='http://www.xyx.com/YYY/logistics/mxg/xnsds/V1_0' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'><MessageHeader><MessageID>3031999515</MessageID><MessageTimeStamp>2017-09-12T06:37:36Z</MessageTimeStamp><SenderID>XBHSNS</SenderID><ReceiverID>GOLF_DAO</ReceiverID><MessageType>ACKACKACK</MessageType><CorrelationID>2d323537383935353034383933383135</CorrelationID></MessageHeader><Masterbill>G829441</Masterbill><LoadID>Jitesh555</LoadID>
<Accept>true</Accept><ReasonCode/><ReasonDescription/></tns:ASNAck>

回答1:

You don't need to parse the whole string, you can just grab the part that interest you.

 grok {
    patterns_dir => "./patterns"
    match => ["message", "<LoadID>%{GREEDYDATA:ASNNumber}</LoadID>"]
 }

I also dropped the "extra", "extras", "behind" fields since you don't keep them.