Perl - How to make a library specific to individua

Question

I am writing a multi threaded script in perl. In which I am using a library Net::Netconf::Manager which inturn uses Net::SSH2. This Net::SSH2(libssh2) doesn't seem to be thread safe when 'shared handles' simulataneously.

I quote as in libssh2 website

Thread-safe: just don't share handles simultaneously

1. I am not sure what this "sharing handles" mean. Also I would like to know how to 'not share handles'.

When I run my script, occasionally i see error trace with backtrace and memory map denoting *** glibc detected *** perl: double free or corruption (out): 0x00007f0320012d70 *** error. This error is because of the thread safeness of the Net::SSh2 library.

2. How to make this Net::Netconf::Manager available to every thread instead of having it declared globally with 'use' . I want all the threads to have their own access to this library independent of other threads.

Please let me know your views.

Answer 1

I am the current maintainer for Net::SSH2.

I have never went after thread safeness for that module but a shallow inspection of its code, shows that probably, that double free error is caused by the Perl side of the Net::SSH2 objects being cloned on thread creation while the C side is not. That results in libssh2 objects being destroyed and released twice which results in the program crashing.

So, if you want to use Net::SSH2 in a multithread application you should ensure that threads are never created from threads where objects of this module exist.

Even then there may be other bugs lurking on the module.

Obviously the right thing to do would be to fix the module. If you want to do it yourself, I would try to help you. Just get in touch with me so that we could discuss the details first... Otherwise, now that you have brought that issue to my attention, well, probably at some point I will fix it myself... but this is not going to happen overnight.

Answer 2

The general workaround to 'thread safety' problems is 'require' and 'import'. These must be called after any sort of thread instantiation takes place, and no threads may be created after (from wherever loaded the modules - it's ok within 'main').

So - because you haven't given us any code, I've used samples out of the module. You'll need to amend accordingly.

#!/usr/bin/env perl

use strict;
use warnings;
use threads;
use Thread::Queue;

my $num_workers = 10;

my %generalargs = (
   'access'              => 'ssh',
   'server'              => 'netconf',
   'command'             => 'junoscript netconf',
   'debug_level'         => 1,
   'client_capabilities' => [
      'urn:ietf:params:xml:ns:netconf:base:1.0',
      'urn:ietf:params:xml:ns:netconf:capability:candidate:1.0',
      'urn:ietf:params:xml:ns:netconf:capability:confirmed-commit:1.0',
      'urn:ietf:params:xml:ns:netconf:capability:validate:1.0',
      'urn:ietf:params:xml:ns:netconf:capability:url:1.0?protocol=http,ftp,file',
      'http://xml.juniper.net/netconf/junos/1.0',
   ]
);

my @host_list = (
   {  'hostname' => 'routername',
      'login'    => 'loginname',
      'password' => 'secret',
   },
   {  'hostname' => 'routername2',
      'login'    => 'differentname',
      'password' => 'anotherpassword',
   },
);

my $work_q = Thread::Queue->new;


sub some_helper_sub_that_isnt_a_thread {
   my ( $input, $process ) = @_;
   return "$input";
}

sub do_netconf_stuff {
   require 'Net::NetConf::Manager';
   Net::NetConf::Manager->import;

   while ( my $item = work_q->dequeue ) {
      my $device = Net::NetConf::Manager->new( %{$item} );
      print 'Could not create Netconf device' unless $device;
      some_helper_sub($device);
   }
}

threads->create( \&do_netconf_stuff ) for 1 .. $num_workers;

foreach my $host (@host_list) {
   $work_q->enqueue( { %$host, %generalargs } );
}
$work_q->end;

$_->join for threads->list;

What happens here is that each thread independently - and at runtime - imports the Net::NetConf::Manager and that means they're each separately instantiated. You can then call other subs from within the thread, and they'll work fine - you've loaded into the global namespace for that thread.

The thing you must not do is then start additional threads that will 'inherit' that imported environment.

Note - this isn't 100% sure to work - there's other reasons that threads might clash (like trying to listen on the same port number, lock the same files etc.). But you will avoid problems within modules due to sharing of file handles etc.