Is there any background process that I can run from plugin without depending on page hits on a website without affecting page-load speed?

I am developing a WordPress plugin and wants to run a background task that calls an API and updates database table. Now, the API can only give results for 5 DB entries in one go and for 500 entries in my table, I have to make 100 API call. The API has allowed TPS quota of 1 and also in every 40 minutes, its old response expires which means I need to update my table if any entry is older than 40 minutes by making a new API call. And, all these DB entries have to be shown to the page viewer with latest data.

The solution that I came up with is scheduling a cron task that runs every minute and does API calls one after another for 25 seconds and then dies so that it doesn't exceed PHP max execution time limit. And at time a customer comes, he has not to wait for API call or be throttled by API

But the problem is I can't rely on wp-cron as it will be called only when page hit occurs(not like actual cron and since I am plugin developer I can't schedule a system cron on my customer's WordPress hosting environment). Also, if the person receives only one page hit in 3 hours, that first person in 3 hours has to either wait for those API calls to finish(not at all desirable or better to say feasible) or else he will not be shown the data.

Is there any other way to solve this problem of updating the DB entries via some background process that is neither slowing down the client and nor depending on page load. So even if 4 clients come in a day they all get latest data (data updated within last 40 minutes) as its already been updated in the background?

Topic wp-remote-get wp-cron cron plugin-development Wordpress

Category Web

1
answered at

Disable WP Cron from running on users' traffic. Then run cron through a cron or timed job somewhere else. You can achieve this by adding this to your wp-config.php

define('DISABLE_WP_CRON', true);

Then set up a cron job to hit this URL - https://domain.com/wp-cron.php?doing_wp_cron at your desired time intervals. A quick wget code below:

wget -q -O - https://domain.com/wp-cron.php?doing_wp_cron >/dev/null 2>&1

I suggest every minute, as all other cron jobs in the WordPress installation will be dependent on it

1
answered at

There is no reason for your design, but the core of your problem is that (I assume) people are trying to use a free tier of the API instead of properly paying for its use and removing the ridiculous restrictions.

Anyway, the solution is fairly simple, run your cron every half an hour, allocate to your import process unlimited time (and you might need to instruct your users to increase memory limit), and just loop until you got all the requested data with a second delay http://php.net/manual/en/function.sleep.php between each request.

Even the worst case of each request taking 5 seconds to complete, the whole thing should be done in 5 minutes.

As for the user view, you should keep new info in some kind of limbo state until you got all of it, and only then make it presentable.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.