How come same logs processed in two ways? The logs which have which are saved as json in a file are stopped by td-agent saying pattern (which is json) not match
This is my log4j.xml configuration for logs which saves (see below first BOLD comment) the logs to /opt/tpa/logs/SPSServer.log and also, forwards them to fluentd (see below second BOLD comment) at 8514 port.
```
<!-- ================================= -->
<!-- **Preserve messages in a local file** -->
<!-- ================================= -->
<appender name="SPSServerFileAppender"
class="com.alcatel.tpapps.common.server.logging.log4jappenders.RollingZipFileAppender">
<errorHandler class="org.apache.log4j.helpers.OnlyOnceErrorHandler"/>
<param name="File" value="/opt/tpa/logs/SPSServer.log"/>
<param name="Folding" value="true"/>
<param name="Debug" value="true"/>
<param name="MaxFileSize" value="30MB"/>
<param name="CompressLogs" value="true"/>
<param name="MaxBackupIndex" value="15"/>
<param name="MaxBackupDir" value="-1"/>
<!-- retain backup folders for 3 months (90 days)
(units of the following parameter are ms)-->
<param name="MaxBackupTime" value="7776000000"/>
<param name="SavePreviousLogs" value="true"/>
<param name="MaxArchiveFileNumber" value="500"/>
<param name="MaxArchiveTime" value="-1"/>
<param name="BackupDirTag" value="SPSServer"/>
<param name="ConsoleLog" value="server_console.log"/>
<layout class="com.alcatel.tpapps.common.server.logging.log4jappenders.UnifiedLoggingFormat">
</layout>
<filter class="com.alcatel.tpapps.common.utils.AddMDCFilter"/>
</appender>
<!--**Use to forward logs to Fluentd**-->
<appender name="SocketAppender" class="com.alcatel.tpapps.common.server.logging.log4jappenders.UnifiedFormatSocketAppender">
<param name="ReconnectionDelay" value="10000"/>
<param name="Application" value="SPS"/>
<param name="Port" value="8514"/>
<param name="RemoteHost" value="localhost"/>
</appender>
So logs in SPSServer.log are:
{"host":"sps-me1-diameterapp-0","level":"WARN","log":{"classname":"com.nokia.as.mux.reactor.Agent:76","message":"ReactorProvider Mux Agent started!","stacktrace":"","threadname":"Main-reactor"},"process":"init","service":"","time":"2021-01-08T01:21:33.470Z","timezone":"Asia/Kolkata","type":"log","system":"SPS_ME_AI_Punit","systemid":"CBAM-11ee306d636b4177b3d78c65cc15d06e"}
{"host":"sps-me1-comsvc-0","level":"INFO","log":{"classname":"com.nokia.csf.cdr.common.util.Env:17","message":"Env is not initialized.","stacktrace":"","threadname":"FelixDispatchQueue"},"process":"isDockerEnv","service":"","time":"2021-01-08T01:19:54.815Z","timezone":"Asia/Kolkata","type":"log","system":"SPS_ME_AI_Punit","systemid":"CBAM-11ee306d636b4177b3d78c65cc15d06e"}
{"host":"sps-me1-oame-0","level":"INFO","log":{"classname":"com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient:245","message":"HB-sps-me1-R8292-sps-me2-O> Unable to contact service for 'sps-me2'.","stacktrace":"[com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: java.net.ConnectException: Connection refused (Connection refused), at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:117), at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:208), at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:130), at com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:124), at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Fiber.java:1121), at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Fiber.java:1035), at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Fiber.java:1004), at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Fiber.java:862), at com.sun.xml.internal.ws.client.Stub.process(Stub.java:448), at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(SEIStub.java:178), at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:93), at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:77), at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:147), at com.sun.proxy.$Proxy267.getHeartBeatStatus(Unknown Source), at com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient.getMemberStatusesInternal(MemberStatusOAMClient.java:160), at com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient.getMemberStatuses(MemberStatusOAMClient.java:130), at com.nokia.tpapps.spscommon.heartbeat.common.HeartBeatMonitorSystemIntegrator.poll(HeartBeatMonitorSystemIntegrator.java:226), at com.nokia.tpapps.spscommon.heartbeat.common.HeartBeatMonitorSystemIntegrator.pollOAM(HeartBeatMonitorSystemIntegrator.java:194), at com.nokia.tpapps.spscommon.heartbeat.common.HeartBeatMonitor.lambda$poll$0(HeartBeatMonitor.java:219), at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511), at java.util.concurrent.FutureTask.run(FutureTask.java:266), at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149), at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624), at java.lang.Thread.run(Thread.java:748), Caused by: java.net.ConnectException: Connection refused (Connection refused), at java.net.PlainSocketImpl.socketConnect(Native Method), at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350), at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206), at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188), at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392), at java.net.Socket.connect(Socket.java:607), at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:660), at sun.net.NetworkClient.doConnect(NetworkClient.java:175), at sun.net.www.http.HttpClient.openServer(HttpClient.java:463), at sun.net.www.http.HttpClient.openServer(HttpClient.java:558), at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264), at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367), at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191), at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1205), at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056), at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177), at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1340), at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1315), at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:264), at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:104), ... 23 more]","threadname":"Heartbeat-4"},"process":"getMemberStatusesInternal","service":"","time":"2021-01-08T09:42:24.081Z","timezone":"Asia/Kolkata","type":"log","system":"SPS_ME_AI_Punit","systemid":"CBAM-11ee306d636b4177b3d78c65cc15d06e"}
Now, my td-agent configuration to handle those logs at 8514 is
<source>
@type tcp
port 8514
bind 0.0.0.0
format /(^(?<header>[^{]+)?(?<message>{.+type.+})$)|(^(?<log>[^{].+))/
tag system
</source>
</worker>
<match system syslog.**>
@type rewrite_tag_filter
<rule>
key message
pattern .+
tag nokia.logging.json
</rule>
<rule>
key log
pattern .+
tag nokia.logging.clear
</rule>
</match>
<filter nokia.logging.json>
@type parser
key_name message
<parse>
@type json
time_key time
time_type string
keep_time_key true
</parse>
</filter>
But td-agent gives following errors when parsing the same logs which are saved in SPSServer.log by log4j.xml (the same above logs.)
2021-01-08 06:51:33 +0530 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not match with data '{{"host":"sps-me1-diameterapp-0","level":"WARN","log":{"classname":"com.nokia.as.mux.reactor.Agent:76","message":"ReactorProvider Mux Agent started!","stacktrace":"","threadname":"Main-reactor"},"process":"init","service":"SPS","time":"2021-01-08T01:21:33.470Z","timezone":"Asia/Kolkata","type":"log","system":"SPS_ME_AI_Punit","systemid":"CBAM-11ee306d636b4177b3d78c65cc15d06e"}'" location=nil tag="nokia.logging.json" time=2021-01-08 06:51:33.471639265 +0530 record={"header"=>"ytx01", "message"=>"{{"host":"sps-me1-diameterapp-0","level":"WARN","log":{"classname":"com.nokia.as.mux.reactor.Agent:76","message":"ReactorProvider Mux Agent started!","stacktrace":"","threadname":"Main-reactor"},"process":"init","service":"SPS","time":"2021-01-08T01:21:33.470Z","timezone":"Asia/Kolkata","type":"log","system":"SPS_ME_AI_Punit","systemid":"CBAM-11ee306d636b4177b3d78c65cc15d06e"}"}
2021-01-08 09:41:33 +0530 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not match with data '{{"host":"sps-me1-comsvc-0","level":"INFO","log":{"classname":"com.nokia.csf.cdr.common.util.Env:17","message":"Env is not initialized.","stacktrace":"","threadname":"FelixDispatchQueue"},"process":"isDockerEnv","service":"SPS&quo
