Solution for processing lots of data with CRON/API, dealing with memory/timeout issues

Using a custom CRON job set up in my Wordpress site I am connecting to an external API to refresh some data. The API only allows for a certain amount of records to be read at a time so I have to page through, once one set of data is loaded then make another request to the API and load more, etc.

The problem is that this can quite often run out of memory or just go on too long and error out for large batches of data.

I am wondering if there is any kind of solution for jobs or workers in Wordpress, where after the first page of results is processed then submit another job for the next page and it's run as a second request instead of one long string.