I have hundreds of file names in an array. I want to create a child process for every 4 files in the array, and have that child do some stuff to each of those 4 files. (So with 100 files, I'll create 25 processes.)
I'm having some trouble understanding the order in which lines are processed when there's a fork. I was thinking I could do something like this, but I'm getting stuck:
foreach $file (@files) {
if ($f++ % 4 == 0) {
my $pid = fork();
if ($pid) {
push(@childs, $pid);
}
elsif ($pid == 0) {
... do stuff to $file ...
}
}
I don't think this is right, and I'm hoping someone can point me in the right direction. Thanks.
In addition to your trouble using fork
, you also seem to have trouble partitioning your @files
array into smaller sets of four files. Maybe something like this:
for (my $i = 0; $i < @files; $i += 4) {
# take a slice of 4 elements from @files
my @files4 = @files[$i .. $i + 3];
# do something with them in a child process
if (fork() == 0) {
... do something with @files4 ...
exit; # <--- this is very important
}
}
# wait for the child processes to finish
wait for 0 .. @files/4;
Use Parallel::ForkManager
use Parallel::ForkManager qw( );
my $pm = Parallel::ForkManager->new(int(@files/4));
for my $file (@files) {
my $pid = $pm->start and next;
... do something with $file ...
$pm->finish; # Terminates the child process
}
Note that this still creates 100 processes, it simply limits it to 25 concurrent.
If you truly want only 25 processes, you can use the following:
use List::Util qw( min );
use Parallel::ForkManager qw( );
my $pm = Parallel::ForkManager->new(0+@files);
while (@files) {
my @batch = @files[0..min(4, $#files)];
my $pid = $pm->start and next;
for my $file (@batch) {
... do something with $file ...
}
$pm->finish; # Terminates the child process
}
i would group to an array, and let the child handle that group
my $group = []
foreach my $file (@files) {
push @$group, $file;
if(scalar(@$group) % 4 == 0) {
my $pid = fork;
die "Unable to fork!" unless defined $pid;
push @childs, $pid if $pid;
children_work($group) unless $pid;
$group = [];
}
}
sub children_work {
my $group = shift;
// child, work with $group
exit(0);
}