Shaheem | 2018-03-17 03:27:26 UTC | #1
Hi guys,
We have an application that registers for Real Time notifications for all users for the following Topics: Presence, Routing Status, Call Summary and Queue Conversations
The application has been running fine for all customers in the last 6 months or so but lately one of the customer has started complaining that the user status are not changing.
I've been trying to figure out where the issue is or has anything changed on the Pure Cloud api.
The application uses C# and REST API (not .Net SDK).
- It authenticates with pure cloud using the client credentials (OAuth Application)
- Loads and caches all the lookup data such as Users, Presence definitions, Queue details etc
- Creates 4 notification channels for each notification topics as mentioned above
- Registers all users and queues for notifications
- Starts a web socket connection for each channel
- Sets up internal heartbeat monitoring in the application (using "channel.metadata" sent every 30 seconds over the web socket connection)
- Starts the re-subscription timer to re-register all subscriptions every 23 hours.
The application monitors all the web socket connections for any errors and would restart the re-subscription process immediately if an error occurs to prevent stale data from being displayed to the end user. It will also re-subscribe all topics if a heartbeat breaches a pre configured threshold of 60 seconds.
I have done a lot of debugging and code analysis to diagnose the issue and feel that it might something to do with the limit of 5 channels per authenticated token limit.
My questions:
- I thought the limit was 10 channels previously. Is it 5 channels now? When did this happen?
- Is the 5 channel limit per authenticated token/user same as authentication token/OAuth Application
- Is the 5 Channel limit per authenticated token or per user/application? i.e. regardless of the authenticated token issued.
It seems channels are being bumped when a new instance of the the application is started with the same credentials simultaneously? I still have to confirm this via some testing. I think this might be the issue with our application?
The thing is if this is true shouldn't the heartbeat of the bumped channel stop. In our case the heartbeat messages are still being received but no actual data (i.e. presence , routing status etc)
References:
Thank you very much for your help in advance.
Regards Shaheem
tim.smith | 2018-03-19 16:13:09 UTC | #2
I thought the limit was 10 channels previously. Is it 5 channels now? When did this happen?
The limit was changed to 5 channels earlier this year.
Is the 5 channel limit per authenticated token/user same as authentication token/OAuth Application Is the 5 Channel limit per authenticated token or per user/application? i.e. regardless of the authenticated token issued.
The enforcement of the limit used to be per auth token, but that has now been changed to per combination of user ID and oauth client ID. In the case of client credentials, the user ID is the same as the client ID because there is no user, so this means you get 5 channels for all instances of the application using those client credentials.
It seems channels are being bumped when a new instance of the the application is started with the same credentials simultaneously?
This is the intended behavior. When the 6th channel is created, the first is deleted.
The thing is if this is true shouldn’t the heartbeat of the bumped channel stop. In our case the heartbeat messages are still being received but no actual data (i.e. presence , routing status etc)
No, the heart beat only indicates that your connection with the server is still open. It does not indicate the status of any notification subscriptions for the channel. PureCloud does not close the web socket if a channel is discarded. I'll pass this feedback along to the dev team.
Shaheem | 2018-03-19 22:09:47 UTC | #3
Hi Tim,
Thanks for your reply.
This seems to be the cause of our issue.
Unfortunately this change seemed to have happened silently and our customer thinks that this was an issue on our end.
What is the recommended way to determine a channel has been bumped since we cannot rely on the heartbeat?
I would recommend that the heartbeat be stop so that we do not require any separate polling to figure out the status of the channel and the status would be known in real time.
Also does this mean we have to have a separate credentials for every 5 channels (5000 subscriptions) needed?
Our biggest customer has got 10000+ users registered with pure cloud and we monitor 4 topics for each user. This means we need 11 x 4 (channels) = 44 channels to get all updates. This translate to around 9 oauth credentials. We need to modify the application to allow entering of n number of credentials and n number of authentications which seems highly unnecessary and impractical. Is this the new intended way of integrating with pure cloud or is this going to change again as I would be required to implement this change into our application?
Thanks Shaheem
tim.smith | 2018-03-20 13:51:21 UTC | #4
Shaheem, post:3, topic:2632
What is the recommended way to determine a channel has been bumped since we cannot rely on the heartbeat?
You can use GET /api/v2/notifications/channels/{channelId}/subscriptions to see if there are any topics subscribed on the channel. If you get an empty array (i.e. { "entities": [] }) and you expect there to be notification topics, the most likely reason is that the channel has been deleted.
Shaheem | 2018-03-20 23:18:28 UTC | #5
Hi Tim,
Thanks for your reply.
Wouldn't you agree that this would put so much unnecessary pressure on your system and also our system to process all these requests to maintain the health of the channels. And at best it would still have a minute latency for each channel if we do this poll every minute.
For the example of the 10000+ users plus 4 channels this would amount to 1440 (Requests per channel per day) * 44 (Channels) = 63,360 Total Requests per day and the health monitoring would still not be in real time. This is not even taking into account the processors running on the client server and the resources available and also not taking into account the requirement to add more topics for monitoring.
Compare this with my suggestion of stopping the heartbeat of the channels that have been bumped. It would be a 100% gain in performance and the health monitoring would be in real time as the heartbeats are already being sent via web socket. That's 63,360 requests compared to 0 requests and the benefit of real time heartbeat monitoring.
Can we put this enhancement request in for a future release? Thanks.
Anyways lets discuss in detail via email as suggested.
Regards Shaheem
system | 2018-04-20 23:20:46 UTC | #6
This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.
This post was migrated from the old Developer Forum.
ref: 2632