James_Tarling | 2022-03-21 15:13:42 UTC | #1
Hi,
We're using a data action to call a Cloud Function on GCP
This has started to return 429 errors
This is the error from GCP
{
"error": {
"code": 429,
"message": "Quota exceeded for quota metric 'Test function' and limit 'Test function per minute' of service 'cloudfunctions.googleapis.com' for consumer 'project_number:XXXXXXXXXXXX'.",
"status": "RESOURCE_EXHAUSTED",
"details": [
{
"@type": "type.googleapis.com/google.rpc.ErrorInfo",
"reason": "RATE_LIMIT_EXCEEDED",
"domain": "googleapis.com",
"metadata": {
"quota_metric": "cloudfunctions.googleapis.com/test_call_requests",
"service": "cloudfunctions.googleapis.com",
"consumer": "projects/XXXXXXXXXXXX",
"quota_limit": "TestCallRequestsPerMinutePerProject"
}
}
]
}
}
It looks like we are using the test URL rather than the trigger URL, after following the example in the Genesys docs. (See GCP docs about rate limits for calling the test URL).
If we switch to using the trigger URL we get a 401 error, or a 403 if we set the Cloud Function ingress to Internal Only. The Data Action is set up to use an integration which has GCP service account credentials
Are you able to advise how to configure the data action, integration, service account or Cloud Function so that we can call the Trigger URL, please
Also, please can you update the docs so the example doesn't use the Cloud Function's test URL
Many thanks
John_Carnell | 2022-03-21 15:19:31 UTC | #2
James_Tarling, post:1, topic:13987
": "cloudfunctions.googleapis.com",
"consumer": "projects/XXXXXXXXXXXX",
"quota_limit": "TestCallRequestsPerMinutePerProject"
}
}
Hi James,
I am not quite sure what you are asking for here, but the error you are getting is a rate-limit error being returned by your Google Cloud provider endpoint. Usually, a 429 indicates you are calling the endpoint too many times within a minute and the service starts throwing a 429 indicating you should back off. A 403 would be an authorization issue so you need to setup your auth (usually some kind of auth token in the header) in the data action to pass the auth token to the endpoint.
Hope that helps.
Thanks, John Carnell Manager, Developer Engagement
James_Tarling | 2022-03-21 17:17:47 UTC | #3
Hi John, As far as I can tell, you can call a Cloud Function with two URLs
The test URL is rate limited and is not meant for production use. The Genesys docs show the test URL being used. We have set up a custom integration with GCP service account credentials and our data action, using the test URL, works fine until we use it in production since we start breaching the test URL quota and getting 429 errors.
But if instead we use the trigger URL, we get a 401 error
So
- I think the docs are wrong to suggest using the test URL since it is not suitable for production
- I wondered if you were able to advise how to configure a Data Action to use a Cloud Function with GCP service account credentials since although the method suggested in the docs works with test URLs it doesn't seem to work with trigger URLs
Regards, James
EDIT: When intercepting the request using ngrok, I can see that there is a Bearer token in the Authorization header. The response from Google includes this in the headers:
Www-Authenticate: Bearer error="invalid_token" error_description="The access token could not be verified"
The response is:
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>401 Unauthorized</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Unauthorized</h1>
<h2>Your client does not have permission to the requested URL <code>/FUNCTION_NAME</code>.</h2>
<h2></h2>
</body></html>
John_Carnell | 2022-03-21 17:19:59 UTC | #4
Hi James,
I will pass on your request to update the document to the data actions team. I am also going to see if anyone on my team has worked with GCP before (most of them have worked only with AWS). I would suggest you take data actions out of the picture (for now) and just try to get the endpoint invoked via something like CURL or POSTMAN. Usually, with this type of thing, it's missing a parameter or the auth token has not been properly encoded (e.g. base64). POSTman is a heck of a lot easier to debug with than data actions because you can more easily manipulate the headers.
Let me talk with a few people and see if I can stir up anything.
Thanks, John Carnell Manager, Developer Engagement
James_Tarling | 2022-03-22 14:36:29 UTC | #5
Thanks John.
Please bear in mind that the authentication, using the GCP service account credentials, is done by the Genesys Cloud Function integration, so if there is a problem with the format of the auth token this isn't something that we can fix at our end
Greg_Cunningham | 2022-03-24 18:51:36 UTC | #6
You seem to be encountering the issue we did when we tried to use triggers. We could only get permissions on a function to work with triggers if it ran without any permission schecks by being public. We could not find any way to get a permission set on the FUNCTION that matched our authentication. Only the FUNCTION call endpoint seemed to work correctly for security. This is why we recommend that triggers were unsuitable for production usage.
We are working on getting a contact at Google to work on this as we did not realize that the FUNCTION:call endpoint was being restricted by Google.
Greg_Cunningham | 2022-03-24 19:21:06 UTC | #7
Following up, I looked into the permissions docs, and there are a couple things you could try.We will try them to and change the docs if they work. The Function documentation, https://cloud.google.com/functions/docs/securing/authenticating, describes a
- User permission, cloudfunctions.functions.invoke
- Service permission, roles/cloudfunctions.invoker.
Try setting those on the IAM role assigned to the service account and it's permissions to your function.
Greg_Cunningham | 2022-03-29 21:39:20 UTC | #8
We found what we need. It is a different type of JWT auth than what they are using for Cloud API endpoints, or GSuite endpoints. https://cloud.google.com/functions/docs/securing/authenticating#functions-bearer-token-example-java
The JWT auth for this is not like the auth we use for GSuite, or GooglCloud endpoints. We will need to implement new code to make this work correctly.
Jason_Mathison | 2022-05-16 14:00:34 UTC | #9
Hey James,
We have a fix for this ready to test. Please shoot me a private message with your OrgId and I will set you up to test it out.
--Jason
James_Tarling | 2022-04-06 09:35:10 UTC | #10
Hi Jason, Greg,
Great news - sorry I missed these replies. I'll be in touch
Thanks, James
James_Tarling | 2022-04-26 15:45:49 UTC | #11
Hi @Jason_Mathison
Have DMed you with the results of our test
Thanks, James
system | 2022-05-27 15:46:44 UTC | #12
This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.
This post was migrated from the old Developer Forum.
ref: 13987