Happy Business Starts Here

Valued Scholar

Large files download through API


I have a technical problem/question. We use the ZOQL Export for integrating our customer's data for analytics.
I am having a problem with larger exports, though. (We basically load all customers , invoices etc.) They are often interrupted by network errors, so I have to restart the download. I've looked at the responses, but it looks like the endpoint:
doesn't support resume (Accept-Ranges)
Do you have any recommendation how to tackle this problem?
Thank you

Re: Large files download through API

Hi @kopacpetr


I understand that it is a fundamental problem that the size of the file you are going to download is large.
If the size of the file gets smaller, your problem may be solved.


> We basically load all customers , invoices etc.

You write as above, but your request may be satisfied by Incremental Load.

You can do Incremental Load using Zuora's AQuA Stateful Mode:
> When you first call AQuA in stateful mode, AQuA uses the combination of partner and project as the unique identifier of a new session. The results file of the first call contains a full load of data for each of the requested queries. If you make subsequent calls with the same values of partner and project, either requesting the same queries or without any queries specified, the results file contains an incremental load of data for each query.


Unfortunately, I have never used Stateful Mode.
Therefore, I can not provide any further advice ...

Valued Scholar

Re: Large files download through API

Thank you, @yamasaki1ma, for your reply.


I've done the stateful mode, but its perk is in the time domain, eg. you load all data up to today and next week you load the updates. Whereas my problem is that "first call contains a full load of data" - which is too much.


If I go with your proposed solution to lower the length of the file, I'd need Zuora to have pagination implemented (doesn't have) or manually iterate over some partitioning key, like Account ID, which is autoincremental and therefore predictable. And I am afraid that this pretty awkward solution is the only one possible :/


Note: Also, I'll have to guess where they use DB indexes, because some queries take 5 hours to complete, so would be "fun" having N 5-hour queries.


Note 2: The new REST API doesn't have the methods that we need for the integration, like "list all accounts", "list all payments" etc.