My developer recently disappeared and I need to make a small change to my website.
Here's the code I'll be referring to:
my $process = scalar(@rows); $process = 500 if $process > 500; my $pm = Parallel::ForkManager->new($process);
This is code from a Perl script that scrapes data from an API system through a cron job. Every time the cron job is run, it's opening a ton of processes for that file. For example,
cron-job.pl will be running 100+ times.
The number of instances it opens is based on the amount of data that needs to be checked so it's different every time, but never seems to exceed 500. Is the code above what would be causing this to happen?
I'm not familiar with using ForkManager, but from the research I've done it looks like it runs the same file multiple times, that way it'll be extracting multiple streams of data from the API system all at the same time.
The problem is that the number of instances being run is significantly slowing down the server. To lower the amount of instances, is it really as simple as changing
500 to a lower number or am I missing something?