We're consuming the Genesys Cloud Data Extraction API and wanted to ask the community how you approach consuming and processing this data, given the volume it generates.
For context on volume: we poll the API every 10 minutes, which generates roughly 260 GB per month spread across up to 1,440 Parquet files a day (individual files can be up to 1 GB), with 26 active schemas. Event latency is around 5 to 10 minutes, with a 72-hour retention window on the API side.
Do you apply any transformation to this raw data before using it for reporting (aggregations, joins across schemas, deduplication), or do you query it directly as it lands? If you do transform it, what tools or approach do you use (Spark, scheduled SQL, something serverless), and how often do those transformations run?
On the architecture side, does your pipeline stop at extraction and raw storage, or do you have an additional processing/curation layer before data reaches dashboards? What would you recommend for someone who currently only has extraction plus raw storage and wants to move toward data that's more "ready to consume"?
#PlatformAPI------------------------------
Danna Espinosa Arenas
------------------------------