BM4291 | 2022-05-18 16:29:50 UTC | #1
Hi,
I'm using the guidance from here: Conversation Detail job (genesys.cloud) to retrieve a jobId ahead of paging the results.
I've run the below and all I get returned is the 'FULFILLED' response which based on the documentation is the next step once I've acquired the jobId... Any ideas why the below isn't returning what I'm expecting?
When I run this via the API Explorer, I get a jobId returned as expected.
FYI - I'm using Azure Data Factory to run a web call:
Input { "url": "<<EU West>>/api/v2/analytics/conversations/details/jobs", "method": "POST", "headers": { "Content-Type": "application/json", "Authorization": "Bearer <<TOKEN>>” }, "body": "{\"interval\":\"2022-05-17T00:00:00.000Z/2022-05-18T00:00:00.000Z\"}" }
Output { "state": "FULFILLED", "expirationDate": "2022-06-18T00:00:00Z", "submissionDate": "2022-05-18T15:56:42.994Z", "ADFWebActivityResponseHeaders": { "Connection": "keep-alive", "ININ-Correlation-Id": "882eff13-9e73-444f-a70e-f0d705e4dc2c", "Strict-Transport-Security": "max-age=600; includeSubDomains", "X-Cache": "Miss from cloudfront", "X-Amz-Cf-Pop": "LHR62-C3", "X-Amz-Cf-Id": "t9Hwd18j6vjqApYaBYdMYmHRuq-sJpPEGGBlK2KoHmcV3wkWEApSdg==", "Cache-Control": "no-store, must-revalidate, no-cache", "Date": "Wed, 18 May 2022 16:11:59 GMT", "Via": "1.1 25ef9a7ab53a2ce8c41854fb4c1b243e.cloudfront.net (CloudFront)", "Content-Length": "105", "Content-Type": "application/json" } ... }
ralegner | 2022-05-18 16:44:31 UTC | #2
https://developer.genesys.cloud/analyticsdatamanagement/analytics/jobs/conversation-details-job
A POST request will submit the job for execution, since it's already fullfilled it will just keep responding that it's ready. To get the results, you need to submit a GET request. The documentation I linked walks you through how this works.
BM4291 | 2022-05-19 06:10:46 UTC | #3
Thanks for the response; I don't quite follow however, the documentation says that the first step i.e. the POST to the jobs endpoint should return a jobId... If I don't have the Id, how can I use the GET request, I'll just be hitting a blank endpoint, what if I have several queries running, how does it distinguish?
I used the same guidance as linked above and specifically, these steps would mean I expect to see a jobId whereas I just get the completed status.
Jerome.Saint-Marc | 2022-05-19 08:09:11 UTC | #4
Hello,
I am not familiar with Azure Data Factory and Web Activities. But the output you are showing is the expected result (in 200 OK) of the HTTP GET /api/v2/analytics/conversations/details/jobs/{jobId} -> a JSON which contains "state", "expirationDate" and "submissionDate".
Out of curiosity, wouldn't you have defined a pipeline with 2 chained Web Activities (in Azure Data Factory)? One to create the job (i.e. POST /api/v2/analytics/conversations/details/jobs -> jobId received in 202 Accepted). And a second one (i.e. GET /api/v2/analytics/conversations/details/jobs/{jobId}) to check the status, waiting for state == FULFILLED to retrieve results, and which takes the jobId retrieved in previous Web Activity as input.
Regards,
BM4291 | 2022-05-19 08:22:46 UTC | #5
Thanks Jerome,
This is the point I'm making, I'm seeing the expected output from the GET call when I'm making a POST call!
Below is a snippet from the pipeline where @BaseURL = 'https://api.euw2.pure.cloud/api/v2/analytics/conversations/details/jobs'
The scenario we have is essentially what you'd described; I'm now re-working the template to include the polling step to get the status of the job, but falling short because I can't actually get the jobId.
Jerome.Saint-Marc | 2022-05-19 08:49:09 UTC | #6
I think I may have found where it is coming from. Just trying to read few pages to make sure as I don't have an Azure env/account to verify.
BM4291 | 2022-05-19 08:52:19 UTC | #7
Let me know and happy to assist/test my end!
Jerome.Saint-Marc | 2022-05-19 09:18:18 UTC | #8
I learned something today :slight_smile:
It is just a guess as I can't check. This has to do with the use of WebActivity and the response to the POST request (to create the job).
The result to the POST request is an HTTP 202 Accepted, with the jobId in the response body (JSON), but also in an HTTP Header named "Location" which points to the job status endpoint (and contains the jobId as well). I mean https://......//api/v2/analytics/conversations/details/jobs/123456
The WebActivity implements an Async Request-Reply pattern. See AZURE DATA FACTORY – WEB HOOK VS WEB ACTIVITY And Asynchronous Request-Reply Pattern
So my guess is that your Web Activity is in fact sending both POST request to create the job, and then polls the GET status. As the status is defined in the GET response (contains state=FULFILLED), your Web Activity is possibly completing on timeout. I mean that the Azure Async Request-Reply pattern may expect a 302 when the data is ready. Also found this on timeout improvements.
As I have never used Azure Data Factory, I don't know if in the Web Activity, there is still a way to get access to the jobId (from the first POST response or from the GET url used for status polling). Or if you would have to use a WebHook Activity as explained in the blog I posted above.
Regards,
BM4291 | 2022-05-20 13:59:22 UTC | #9
Apologies for the delay, have been testing this one back and forth for a while.
I wasn't able to get the webhook activity to function as expected so have gone down a slightly different route by calling the query endpoint and paging the results in a loop.
I'm able to get the desired results here, although it's having to run the data in batches of 100 due to the max page size. This seems okay for now, but our system only went to production yesterday so not clear on what the anticipated volumes will be once it's fully operational.
I'll have another look at the webhook approach to get the jobId; assuming I'd be able have this trigger, pointing at the jobs endpoint... Then run a web call in parallel and grab the jobId from there.
Jerome.Saint-Marc | 2022-05-20 14:20:30 UTC | #10
According to the documentation on conversation detail job, the max pageSize for GET /api/v2/analytics/conversations/details/jobs/{jobId}/results is 10000.
So you could reduce the number of your requests to retrieve the job results.
There might be another limit triggered on the max size of an HTTP Response - like if you ask next 10000 results and each conversation details is large (e.g. a lot of participant attributes - which are retrieved in case of the Async Conversation Details query).
Regards,
BM4291 | 2022-05-20 14:29:58 UTC | #11
Unfortunately that only applies to the jobs endpoint; for the query endpoint it looks to be 100 results per page:
So until I can work out a way to get that jobId, I'll have to make do with the 100 per page max.
Jerome.Saint-Marc | 2022-05-20 14:35:25 UTC | #12
Ah ok. I thought you were talking about the job endpoints.
BM4291 | 2022-05-31 08:24:29 UTC | #13
Just wanted to check something on this as we're hoping to wrap this development up soon.
If we went down the WebHook route, am I passing the same parameters as to the web activity i.e. the URL would point to the jobs endpoint, and I would just need to trigger a web activity to call it? On calling that end point, the web hook should then respond with the expected output?
Jerome.Saint-Marc | 2022-05-31 08:46:23 UTC | #14
Hello,
If the question is for me, as I mentioned, I am not familiar with Azure Data Factory. But from what I understand of webhook (the short article/page I read), you would have: A first webhook to create the job (i.e. POST /api/v2/analytics/conversations/details/jobs) Looping on a second webhook to check job status, using the jobId obtained in the 302 response from previous webhook (i.e. GET /api/v2/analytics/conversations/details/jobs/{jobId}) until state = FULFILLED. And then invoking 3rd webhook to get the results - one or more calls depending on how many data you retrieve - if it is will be in a single page or over multiple ones (i.e. GET /api/v2/analytics/conversations/details/jobs/{jobId}/results using cursor query parameter if there are multiple pages to retrieve).
Regards,
BM4291 | 2022-05-31 09:00:12 UTC | #15
Thanks, was more a general pattern that I needed a bit of guidance with, and to ensure I'm pointing at the right endpoint throughout.
I'll see how I go.
BM4291 | 2022-07-25 09:36:35 UTC | #16
Looks like this one won't work via the webhook method; calling the same query via webhook just times out as there's nothing to point at, the web activity just returns the 200 status without polling so think we'll have to stick with the current method and see how it goes for now.
Tried a $top=1 and that's not supported on the jobs endpoint, unless I'm missing something?
Alternatively we could port this to PowerAutomate/Logic Apps but less keen on that given it separates out our platform.
UPDATE
After revisiting this one I've found that the solution was staring me in the face the whole time and leaving this comment here to hopefully help others... Out of the box there is an advanced setting on the web activity in ADF to switch async on/off:
Firing a request at this returns a jobId and location header with the 202 response.
system | 2022-07-02 10:34:17 UTC | #17
This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.
Jerome.Saint-Marc | 2022-07-25 12:59:40 UTC | #18
Glad you found the necessary setting on Azure Data Factory!
Regards,
This post was migrated from the old Developer Forum.
ref: 14796