We are wanting to automate out windows patching process, I see a few items as being required:

1. check server is in a good state (for media server, CIC server, CCS server & Marquee server)
2. check switchover pair is in a good state (ready for switchover), both CIC & CCS switchover pairs
3. check which server is active in switchover pair. (both CIC & CCS switchover pairs)
4. initiate switchover (both CIC & CCS switchover pairs)
5. check server has restarted successfully, for each server type

Some of these e.g. 5. above for CIC server could be done by checking for a recent occurrence of event 15008 in windows app log (event id 34000? on media server, on marquee server event id 0? but on CCS server???

Does anyone know of useful documentation around this?

Thanks
#PlatformAdministration

------------------------------
Tarquin Bell
Precision Administration Services (Pty) Ltd
------------------------------

Tarquin,

Many years ago the company I was at we had a similar requirement. We needed to reboot the CIC environment and no one liked getting up early to do it. We wrote a script in VB script that would check server status and initiate switchovers. The VB script used sendcustomnotification to check server status and initiate switchovers as needed along with monitoring the service statuses of the various CIC applications. I didn't write the script but I can tell you that what you are asking for is totally possible with a little scripting work and some handler customization.

Thanks,

------------------------------
Mark Tatera
ConvergeOne

Opinions are my own and not the views of my employer. Any suggestions or programming changes I suggest come with no warranty and should be tried at your own risk.
------------------------------

Original Message:
Sent: 08-14-2019 21:14
From: Tarquin Bell
Subject: Monitoring

We are wanting to automate out windows patching process, I see a few items as being required:

1. check server is in a good state (for media server, CIC server, CCS server & Marquee server)
2. check switchover pair is in a good state (ready for switchover), both CIC & CCS switchover pairs
3. check which server is active in switchover pair. (both CIC & CCS switchover pairs)
4. initiate switchover (both CIC & CCS switchover pairs)
5. check server has restarted successfully, for each server type

Some of these e.g. 5. above for CIC server could be done by checking for a recent occurrence of event 15008 in windows app log (event id 34000? on media server, on marquee server event id 0? but on CCS server???

Does anyone know of useful documentation around this?

Thanks
#PlatformAdministration

------------------------------
Tarquin Bell
Precision Administration Services (Pty) Ltd
------------------------------

As Mark said, there are possibilities here and I have written them. We actually have code in a package for automating sync to non-production environments, from production, e.g. DEV, UAT, etc. However, I would caution use of these on production. We do not recommend this as more success validation is needed when dealing with production. A simple switchover event is one thing, but applying Microsoft patches or any other system changes should include testing and validation of at least critical call and routing functions.

An example might be that you patch one of the Switchover member servers, switch to it as active and test. Before patching the 2nd member server, you validate the patches and do not wholly commit the entire Switchover pair until fully validated. This logic could also be applied to peripheral servers, as available in multiples.

Good luck and happy patching! :-)

------------------------------
Chris Mayo
Director, Technical Services
Altivon
------------------------------

Original Message:
Sent: 08-15-2019 10:06
From: Mark Tatera
Subject: Monitoring

Tarquin,

Many years ago the company I was at we had a similar requirement. We needed to reboot the CIC environment and no one liked getting up early to do it. We wrote a script in VB script that would check server status and initiate switchovers. The VB script used sendcustomnotification to check server status and initiate switchovers as needed along with monitoring the service statuses of the various CIC applications. I didn't write the script but I can tell you that what you are asking for is totally possible with a little scripting work and some handler customization.

Thanks,

------------------------------
Mark Tatera
ConvergeOne

Opinions are my own and not the views of my employer. Any suggestions or programming changes I suggest come with no warranty and should be tried at your own risk.

Original Message:
Sent: 08-14-2019 21:14
From: Tarquin Bell
Subject: Monitoring

We are wanting to automate out windows patching process, I see a few items as being required:

1. check server is in a good state (for media server, CIC server, CCS server & Marquee server)
2. check switchover pair is in a good state (ready for switchover), both CIC & CCS switchover pairs
3. check which server is active in switchover pair. (both CIC & CCS switchover pairs)
4. initiate switchover (both CIC & CCS switchover pairs)
5. check server has restarted successfully, for each server type

Some of these e.g. 5. above for CIC server could be done by checking for a recent occurrence of event 15008 in windows app log (event id 34000? on media server, on marquee server event id 0? but on CCS server???

Does anyone know of useful documentation around this?

Thanks
#PlatformAdministration

------------------------------
Tarquin Bell
Precision Administration Services (Pty) Ltd
------------------------------

Original Message------

As Mark said, there are possibilities here and I have written them. We actually have code in a package for automating sync to non-production environments, from production, e.g. DEV, UAT, etc. However, I would caution use of these on production. We do not recommend this as more success validation is needed when dealing with production. A simple switchover event is one thing, but applying Microsoft patches or any other system changes should include testing and validation of at least critical call and routing functions.

An example might be that you patch one of the Switchover member servers, switch to it as active and test. Before patching the 2nd member server, you validate the patches and do not wholly commit the entire Switchover pair until fully validated. This logic could also be applied to peripheral servers, as available in multiples.

Good luck and happy patching! :-)

------------------------------
Chris Mayo
Director, Technical Services
Altivon
------------------------------

Tarquin,

I worked for a company who rolled out Empirix Hammer. The tool is very slick and seems to have quite a few nice features for testing IVRs for regression tests or hourly/daily/weekly tests. I ended up leaving the company before the product was fully deployed but from what I saw the product seems to allow you to do a good amount of testing of the IVR. Obviously this would not be able to test your internal things like calls to and from phones and the CIC applications work properly.

Thanks,

------------------------------
Mark Tatera
ConvergeOne

Opinions are my own and not the views of my employer. Any suggestions or programming changes I suggest come with no warranty and should be tried at your own risk.
------------------------------

Original Message:
Sent: 08-15-2019 19:45
From: Tarquin Bell
Subject: Monitoring

Thanks Mark & Chris

 

We run the MS patching through our Integration & Acceptance environments 2 weeks before Production.

So I think I'll test with using 'SendCustomNotification' and a successful 'GetProcessInformation' to give a basic "Is Up" test.

 

Have you tried using automated call flow testers (like Cyara or StarTrinity) for voice service monitoring?

 

Best Regards

 

Tarquin Bell

Senior Telephony Administrator

 

GPO Box 2924 Brisbane Qld 4001

30 Little Cribb Street Milton Qld 4064
T 07 3333 7033 | M 0427 667 110 | W sunsuper.com.au

 

 

IMPORTANT INFORMATION: This email is from Sunsuper Pty Ltd (ABN 88 010 720 840; AFSL 228975) or one of its wholly owned subsidiaries - Sunsuper Financial Services Pty Ltd (ABN 50 087 154 818; AFSL 227867) or Precision Administration Services Pty Ltd (ABN 47 098 977 667; AFSL 246604). It may be confidential or privileged. If you do not wish to receive future e-mail messages please send an e-mail to privacy_officer@sunsuper.com.au with the subject line UNSUBSCRIBE. Our full email footer is available at http://www.sunsuper.com.au/transmission_disclaimer/ For ratings and awards information refer to sunsuper.com.au/ratingsagencies

Original Message------

As Mark said, there are possibilities here and I have written them. We actually have code in a package for automating sync to non-production environments, from production, e.g. DEV, UAT, etc. However, I would caution use of these on production. We do not recommend this as more success validation is needed when dealing with production. A simple switchover event is one thing, but applying Microsoft patches or any other system changes should include testing and validation of at least critical call and routing functions.

An example might be that you patch one of the Switchover member servers, switch to it as active and test. Before patching the 2nd member server, you validate the patches and do not wholly commit the entire Switchover pair until fully validated. This logic could also be applied to peripheral servers, as available in multiples.

Good luck and happy patching! :-)

------------------------------
Chris Mayo
Director, Technical Services
Altivon
------------------------------

There are many options to verify a successful startup of PureConnect, including testing services like Cyara and TekVision, as well as using ININ SNMP to monitor and receive system level details.

However, for our call center customers there is no tolerance for unexpected service impacts at the start of a business day. Even with the most extensive strategy, it's ultimately up to the business as to how tolerable they may be to service impacting events. We have some customers that chose to either let automation apply patches or a seperate server team (no PureConnect savvy) and get many tickets from these, most often as emergencies. In 99% of these instances, more manual testing with PureConnect savvy personnel could have avoided the issues.

I wish you the best of look in this but I recommend you carefully consider all the risks and exposures.

Thanks.

------------------------------
Chris Mayo
Director, Technical Services
Altivon
------------------------------

Original Message:
Sent: 08-15-2019 19:45
From: Tarquin Bell
Subject: Monitoring

Thanks Mark & Chris

 

We run the MS patching through our Integration & Acceptance environments 2 weeks before Production.

So I think I'll test with using 'SendCustomNotification' and a successful 'GetProcessInformation' to give a basic "Is Up" test.

 

Have you tried using automated call flow testers (like Cyara or StarTrinity) for voice service monitoring?

 

Best Regards

 

Tarquin Bell

Senior Telephony Administrator

 

GPO Box 2924 Brisbane Qld 4001

30 Little Cribb Street Milton Qld 4064
T 07 3333 7033 | M 0427 667 110 | W sunsuper.com.au

 

 

IMPORTANT INFORMATION: This email is from Sunsuper Pty Ltd (ABN 88 010 720 840; AFSL 228975) or one of its wholly owned subsidiaries - Sunsuper Financial Services Pty Ltd (ABN 50 087 154 818; AFSL 227867) or Precision Administration Services Pty Ltd (ABN 47 098 977 667; AFSL 246604). It may be confidential or privileged. If you do not wish to receive future e-mail messages please send an e-mail to privacy_officer@sunsuper.com.au with the subject line UNSUBSCRIBE. Our full email footer is available at http://www.sunsuper.com.au/transmission_disclaimer/ For ratings and awards information refer to sunsuper.com.au/ratingsagencies

Original Message------

As Mark said, there are possibilities here and I have written them. We actually have code in a package for automating sync to non-production environments, from production, e.g. DEV, UAT, etc. However, I would caution use of these on production. We do not recommend this as more success validation is needed when dealing with production. A simple switchover event is one thing, but applying Microsoft patches or any other system changes should include testing and validation of at least critical call and routing functions.

An example might be that you patch one of the Switchover member servers, switch to it as active and test. Before patching the 2nd member server, you validate the patches and do not wholly commit the entire Switchover pair until fully validated. This logic could also be applied to peripheral servers, as available in multiples.

Good luck and happy patching! :-)

------------------------------
Chris Mayo
Director, Technical Services
Altivon
------------------------------

Thanks again guys.

I'm just having to work things through with server admins and their ethos of script/automate everything.

------------------------------
Tarquin Bell
Precision Administration Services (Pty) Ltd
------------------------------

Original Message:
Sent: 08-16-2019 11:06
From: Chris Mayo
Subject: Monitoring

There are many options to verify a successful startup of PureConnect, including testing services like Cyara and TekVision, as well as using ININ SNMP to monitor and receive system level details.

However, for our call center customers there is no tolerance for unexpected service impacts at the start of a business day. Even with the most extensive strategy, it's ultimately up to the business as to how tolerable they may be to service impacting events. We have some customers that chose to either let automation apply patches or a seperate server team (no PureConnect savvy) and get many tickets from these, most often as emergencies. In 99% of these instances, more manual testing with PureConnect savvy personnel could have avoided the issues.

I wish you the best of look in this but I recommend you carefully consider all the risks and exposures.

Thanks.

------------------------------
Chris Mayo
Director, Technical Services
Altivon

Original Message:
Sent: 08-15-2019 19:45
From: Tarquin Bell
Subject: Monitoring

Thanks Mark & Chris

 

We run the MS patching through our Integration & Acceptance environments 2 weeks before Production.

So I think I'll test with using 'SendCustomNotification' and a successful 'GetProcessInformation' to give a basic "Is Up" test.

 

Have you tried using automated call flow testers (like Cyara or StarTrinity) for voice service monitoring?

 

Best Regards

 

Tarquin Bell

Senior Telephony Administrator

 

GPO Box 2924 Brisbane Qld 4001

30 Little Cribb Street Milton Qld 4064
T 07 3333 7033 | M 0427 667 110 | W sunsuper.com.au

 

 

IMPORTANT INFORMATION: This email is from Sunsuper Pty Ltd (ABN 88 010 720 840; AFSL 228975) or one of its wholly owned subsidiaries - Sunsuper Financial Services Pty Ltd (ABN 50 087 154 818; AFSL 227867) or Precision Administration Services Pty Ltd (ABN 47 098 977 667; AFSL 246604). It may be confidential or privileged. If you do not wish to receive future e-mail messages please send an e-mail to privacy_officer@sunsuper.com.au with the subject line UNSUBSCRIBE. Our full email footer is available at http://www.sunsuper.com.au/transmission_disclaimer/ For ratings and awards information refer to sunsuper.com.au/ratingsagencies

Original Message------

As Mark said, there are possibilities here and I have written them. We actually have code in a package for automating sync to non-production environments, from production, e.g. DEV, UAT, etc. However, I would caution use of these on production. We do not recommend this as more success validation is needed when dealing with production. A simple switchover event is one thing, but applying Microsoft patches or any other system changes should include testing and validation of at least critical call and routing functions.

An example might be that you patch one of the Switchover member servers, switch to it as active and test. Before patching the 2nd member server, you validate the patches and do not wholly commit the entire Switchover pair until fully validated. This logic could also be applied to peripheral servers, as available in multiples.

Good luck and happy patching! :-)

------------------------------
Chris Mayo
Director, Technical Services
Altivon
------------------------------