I’m doing some Ruby development on servers that I have inherited (aka: I never originally set up.) that haven’t been maintained in a while and noticed that the installed MongoDB version was 1.8 when a 2.4 series upgrade was available. Since the box is running a RedHat/CentOS variant that uses yum to install RPMs, I went ahead and do what I usually do to upgrade. First, stop the running MongoDB instance like this:
sudo service mongod stop
And then upgrade the packages from the repo.
sudo yum install mongodb mongodb-server libmongodb
All went well including dependencies being installed as well. But when I went to restart MongoDB bia this command:
sudo service mongod start
Nothing appeared to happen. Connections were dead. Checking the MongoDB log showed the following one sad error line:
/usr/bin/mongod: symbol lookup error: /usr/bin/mongod: undefined symbol: _ZN7pcrecpp2RE4InitEPKcPKNS_10RE_OptionsE
What the heck is that about? I saw this question and answer thread that recommending rebuilding from the RPM source as well as other posts online that advise some variant of the same: Download source code to recompile or download the RPM directly from the MongoDB site. But all of those solutions seem to radical for what should be a simple package installer update? What could be happening?
I figured it out. Somewhat accidentally, but fairly certain this is the solution. The short answer? If you get that /usr/bin/mongod: symbol lookup error: /usr/bin/mongod: undefined symbol: _ZN7pcrecpp2RE4InitEPKcPKNS_10RE_OptionsE then you should install pcre and pcre-devel from the repository like this:
sudo yum install pcre pcre-devel
Details on how I discovered this are basically, I was resigning myself to recompiling from scratch as outlined in this answer. That is something I do not want to do unless there is a very good reason. But as the answerer states, before recompiling one should install the following compiler items and related libraries:
sudo yum install rpm-build redhat-rpm-config gcc gcc-c++ make yum install openssl-devel snappy-devel v8-devel boost-devel python-devel python-nose scons pcre-devel readline-devel libpcap-devel gperftools-devel
Okay, so I did that to lay down the groundwork for a source code rebuild. But also noticed in the install that pcre was being installed as it was apparently missing and required dependency for pcre-devel; this is key. As I was getting ready to recompile I just decided to attempt to start mongod again like this:
sudo service mongod start
And checked. Lo and behold, the MongoDB install was running again! But why? This answer here holds the clue:
The error was caused by libpcre changing the signature of
RE::Init() to only take a std::string, rather than a char*. This
is fixed if you get a newer version of libpcrecpp, which adds the
old interface for backwards compat.
That answer also recommended recompiling from source, but now that made little sense since it was clear my MongoDB install was up and running again. So I ran lsof on the development box and saw this:
sudo lsof | grep pcre
nginx 892 deploy mem REG 253,2 97140006 (deleted)/lib64/libpcre.so.0.0.1 (stat: No such file or directory)
nginx 893 deploy mem REG 253,2 97140006 (deleted)/lib64/libpcre.so.0.0.1 (stat: No such file or directory)
nginx 1369 root mem REG 253,2 97140006 (deleted)/lib64/libpcre.so.0.0.1 (stat: No such file or directory)
mongod 26841 mongodb mem REG 253,2 1052673 /usr/lib64/libpcrecpp.so.0.0.0 (path dev=0,53)
mongod 26841 mongodb mem REG 253,2 97126735 /lib64/libpcre.so.0.0.1 (path dev=0,53)
grep 28590 deploy mem REG 253,2 97126735 /lib64/libpcre.so.0.0.1 (path dev=0,53)
Note how the mongod user is loading /lib64/libpcre.so.0.0.1. That has to be it, right?
I confirmed this by jumping onto the partner/twin production box of this setup—where I did not upgrade MongoDB on yet—and ran the same lsof command and this was the result:
sudo lsof | grep pcre
nginx 922 root mem REG 253,2 81795343 (deleted)/lib64/libpcre.so.0.0.1 (stat: No such file or directory)
nginx 923 deploy mem REG 253,2 81795343 (deleted)/lib64/libpcre.so.0.0.1 (stat: No such file or directory)
nginx 924 deploy mem REG 253,2 81795343 (deleted)/lib64/libpcre.so.0.0.1 (stat: No such file or directory)
grep 8067 deploy mem REG 253,2 81791051 /lib64/libpcre.so.0.0.1 (path dev=0,61)
Note how in comparison there is 100% no instance of mongod loading /lib64/libpcre.so.0.0.1. So the solution to this issue was not to recompile from source—and thus dealing with the headaches of a no-RPM install—but rather just to install pcre from the repository.
