PureConnect

 View Only
Discussion Thread View
  • 1.  Restarting StatServer or StatServerAgent - PMDB Error at the end of most intervals

    Posted 08-31-2022 09:29
    Good morning all,

    Starting yesterday we are running into an issue that we haven't seen before. This is happening not at every 30 minute interval, but it's happening more than it isn't. Reprocessing PMQ manually seems to work fine and clear all the .ERR files. Have any of you ran into this before? We know of nothing that changed yesterday to cause it and our SQL dba says everything looks fine on his end. We're running 2021 R4 Patch 4 and everything has been fine for months until yesterday.

    We have also thought about restarting StatServer and/or StatServer Agent. Do you guys know if that'll cause a switchover event and/or potentially clear this error?

    The common error we see at most intervals is:

    Data for StatServerWorkgroupU-w64:IC Report Logging7 failed processing with this message: PMDB ERROR - SQueueMsg [ SVer(4) ReportLog(85) Action(EICLogQueueInsert) Data[(100) rows with (0) fields, filter-() ] [I3ODBCCONNECTION.CPP, 1063] A database error was generated in function SQLEndTran:    DBMS error code = -2147467259; SQLSTATE = 08S01;    Message = '[Microsoft][SQL Server Native Client 11.0]Communication link failure'

    Any insight or suggestions are greatly appreciated.

    Thanks as always.
    #PlatformAdministration
    #Reporting/Analytics
    #SystemAdministration

    ------------------------------
    Shane
    ------------------------------


  • 2.  RE: Restarting StatServer or StatServerAgent - PMDB Error at the end of most intervals

    Posted 09-01-2022 02:13
    PMQ is handled by transaction server. So you should look at the transaction server logs to get more information about the SQL error. But the message you posted says it all: "Communication link failure". This meens the link to the database is broken. Perhaps you have a problem in your network or on the databsase server?

    ------------------------------
    Andreas Tikart
    Fiebig GmbH
    ------------------------------



  • 3.  RE: Restarting StatServer or StatServerAgent - PMDB Error at the end of most intervals

    Posted 09-01-2022 10:59
    Thanks Andreas! After troubleshooting this issue with our network team for many hours yesterday we discovered that they had made recent firewall adjustments which had introduced latency and some packet loss. This was leading to our issue w StatServer. Once they reversed those changes, the issue immediately went away.

    Thanks again for the reply!

    ------------------------------
    Shane
    ------------------------------



  • 4.  RE: Restarting StatServer or StatServerAgent - PMDB Error at the end of most intervals

    Posted 10-31-2022 10:00
    All,

    This issue has appeared again. Detail data is getting written consistently, StatServer Workgroup and Agent is timing out with the error in previous post and below.
    Can someone explain why or how StatServer data is different than say Detail data? We're trying to determine why this issue ONLY seems to be impacting StatServer.

    We have gracefully switched over to our secondary IC server, same issue occurs there. We have also switched to another SQL server in our cluster.

    Data for StatServerWorkgroupU-w64:IC Report Logging7 failed processing with this message: PMDB ERROR - SQueueMsg [ SVer(4) ReportLog(85) Action(EICLogQueueInsert) Data[(100) rows with (0) fields, filter-() ] [I3ODBCCONNECTION.CPP, 1063] A database error was generated in function SQLEndTran:    DBMS error code = -2147467259; SQLSTATE = 08S01;    Message = '[Microsoft][SQL Server Native Client 11.0]Communication link failure'

    Manually reprocessing PMQ does work intermittently.

    Anyone seen something similar to this or can explain why only statserver may be failing here?

    Thanks,
    Shane

    ------------------------------
    Shane
    ------------------------------



  • 5.  RE: Restarting StatServer or StatServerAgent - PMDB Error at the end of most intervals

    Posted 10-31-2022 10:31
    I see you are using "SQL Server Native Client 11.0". Perhaps you should upgrade your SQL client and then adjust the IC data sources configuration. We are using "ODBC Driver 13 for SQL Server" which has a better handling for encrypted SQL connections, but you need to check which SQL client (!) release is supported by which CIC version, especially ICBM historical repoorting and Dialer are the limitations here.

    ------------------------------
    Andreas Tikart
    Fiebig GmbH
    ------------------------------



  • 6.  RE: Restarting StatServer or StatServerAgent - PMDB Error at the end of most intervals

    Posted 10-31-2022 17:07
    Hi Andreas,

    Thanks for replying!

    A little earlier today we finally discovered the root cause of our statserver data issue. We had a recurring SQL db script that queried for data to push to a CRM system. That db script wasn't leveraging (nolock) thus it was locking the database and preventing stat server data from getting written. We have since addressed that and now all seems well. Just so happened the timing of the recurring job also conflicted with the 30 minute stat server data interval.

    This may help others down the road. We've ran into this similar issue twice now, first resolution was firewall related and this most recent was due to the above.

    Thanks all!

    ------------------------------
    Shane
    ------------------------------