mkc210342 | 2022-11-29 09:57:54 UTC | #1
I am trying to fetch the historical adherence report in a Java Application.
My approach is to call: /api/v2/workforcemanagement/adherence/historical to request a report, then call: /api/v2/workforcemanagement/adherence/historical/jobs/{jobId} in which there will be multiple downloadUrls for reading the historical adherence report, which is a json file.
However, whenever I tried to use HTTPClient in Java to read the downloadUrls, although I am getting a status code 200, my application always fails to decode it. I tried different types of encoding like "UTF-8", "ISO-8859-1" but none of this works. I tried other url which returns a JSON and all of them can be decoded successfully.
Here is my Java code:
HttpClient client = HttpClient.newHttpClient(); HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(url)) .header("Content-Type", "application/json; charset=UTF-8") .header("Accept", "application/json") .header("Accept-Encoding", "gzip, deflate, br") .GET() .build();
HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); System.out.println(response.body());
And here is an exmaple of the download url (The URL has already expired. I just posted it for reference): https://historical-adherence-prod-apne1-wemagents.s3.ap-northeast-1.amazonaws.com/historicalAdherence/userDownloads/historicalAdherence/6d24809f-c5c4-4837-a417-ffeeab4d34fe/13508579-8027-41a6-853d-4f366347494e?response-content-disposition=inline%3Bfilename%3D%22historicalAdherenceResult.json%22&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEJz%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDmFwLW5vcnRoZWFzdC0xIkgwRgIhANrvTfkP2zmRDQ81bFaNZtk0WCsoOrGhWP5mVnVDpO0IAiEAyZI7lgKnot7WB59rUzVn%2Bc7aVkMpqdHfzW%2BFIyrHdEkq5QQIpP%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARAAGgw3NjU2Mjg5ODU0NzEiDKO%2BfMvzYsk0HoYYpyq5BD%2Bun1wnuEljhFtbIlGmh5UmJaa2G1LwlALW6Whf2xHRL5K9EhY22xHZmIc8eEtI%2B3lQC0xx3TeyleurXA3mTr4B%2F0bsnxfH%2B%2Fcfi2Iogh8de5qk98c5WlREE0MENOynwrriJdGtGiUv1qFtFnl15DIapQKWxO3Zo9NaNa2%2BaCZTjAbD95qjMNNJYOquHEZT9IAAvwt35uHjI1fnND2HgrU7GZFqvg29D%2FjOIsYn%2BwHdxuoUMB6z2lJdc4%2B6jEQdtbW40pnFfTCTjg3x%2BtzG5caI0NpElYPwsvwhEPLmm3ijJjQZfE9RLsjj1xejvt%2FK655bSZyyZrSRGGmpBwf6g34IqJVXtbi1ptzBQFWqerA8EmbWN1gyru%2FiwcYEjZDDjeXY8v%2BshGQ5cfU5biQIOOwhdXe5lmqW5V1ALyDUgWMd566wd7beI3p6TyCWKPXl1myVLsW41Ac5aeCxj%2FfV1dqBEQ%2BzDtLEbaekj41GeoLTNCXYfVNtA5FLmyrfHlBFneRjD63A7ngJ38yJqtTpPGfNk2HLqLIAw%2FPZzW8l5NJdP1hBLCbrU3k1P%2FCYZv2F2lhgzZOFrwchJtPPr0cYAlqro7SROVNh1u3FDIHqhSvwgLHJtfs4J07v46MormcLyKMnp7rk1IbcgRxalsnlmObc8PdIMDo6xricJOiNAkSdGhhPnVvnJ1PvQ9vMdl%2Bj3lOdmrCeHLkC7NzisDNZK8H3rHW4FA2aGlk9zU%2BIYFDAKI%2B%2Fsi1fAJkPMP708psGOqgBHyQ59FRlRA3%2FaJfZcTYeRAT2CQ%2Bwru20yvou2jf3K8d19vih2YMiKbO76jLooWET37rBmQJo3dx6wGL3wKOb%2FM%2BI8S2vktXpyUOOIAv2soXboWD5iSgdFUSuiIVLbMonGO9AJS87FQ17JeK8OVqpfYnPdD9dRwIbA2%2BC66wisYyXJQQaR9t3EdaPhz4ZUS8QBbHGzjjCBIj6E0Q0U%2B4w7A%2FWVpBUNSNu&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221122T123751Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=ASIA3EQYLGB76GODUQ6J%2F20221122%2Fap-northeast-1%2Fs3%2Faws4_request&X-Amz-Signature=c43f3738c2d924ae6078ea2b92b7d22a1ba360de050d4e78775516b52a53ae85
Weirdly enough, I don't have this issue in Python using Requests library. Does anyone know why this happens?
Here is what I see in Java:
Eos_Rios | 2022-11-30 03:42:57 UTC | #3
It's entirely up to the implementation of the browser you use. .NET's default webclient has a similar issue and has to be replaced with a version that overrides it's internal GetWebRequest with this logic to force it to auto decompress;
switch (_request ) { case HttpWebRequest httpRequest: httpRequest.AllowAutoRedirect = AutoRedirect; httpRequest.CookieContainer = CookieContainer; httpRequest.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
Setup?.Invoke(httpRequest); break;
You may want to try a DecompressingHttpClient or https://hc.apache.org/httpcomponents-client-4.5.x/current/httpclient/apidocs/org/apache/http/impl/client/HttpClientBuilder.html to see if those behave as expected.
mkc210342 | 2022-11-30 03:42:59 UTC | #4
Thank you very much! It is working now! In case anyone wants to know here is the code that works. (I am using java.net.HttpClient)
HttpClient client = HttpClient.newHttpClient(); HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(url)) .header("Content-Type", "application/json; charset=UTF-8") .header("Accept", "application/json") .header("Accept-Encoding", "gzip, deflate, br") .version(HttpClient.Version.HTTP_2) .GET() .build();
HttpResponse<InputStream> response = client.send(request, HttpResponse.BodyHandlers.ofInputStream()); GZIPInputStream gzis = new GZIPInputStream(response.body()); InputStreamReader reader = new InputStreamReader(gzis); BufferedReader in = new BufferedReader(reader); String data = in.readLine(); System.out.println(data);
Apparently java.net library does not handle compressed data by default, and that's why we need to implement decompression within the application. Reference: Does Java HTTP Client handle compression - Stack Overflow
system | 2022-12-31 03:40:18 UTC | #5
This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.
This post was migrated from the old Developer Forum.
ref: 17417