My developer committed a huge mistake and we cannot find our mongo database anyone in the server. Rescue please!!!
He logged into the server, and saved the following shell under ~/crontab/mongod_back.sh
:
And then he run ./mongod_back.sh
, then there were lots of permission denied
, then he did Ctrl+C
. Then the server shut down automatically.
He tried to restart the server, then he got an grub error:
He then contacted AliCloud, the engineer connected the disk to another working server, so that he could check the disk. Then, he realized that some folders have gone, including /data/
where the mongodb is!!!
1) We just don't understand how the bash could destroy the disk including /data/
;
2) And of course, is it possible to get the /data/
back?
PS: he did not take a snapshot of the disk before.
Question 1
1) We just don't understand how the bash could destroy the disk including /data/;
Reason: $OUT_DIR
was unset
In bash
and sh
comments are written as # comment
, not // comment
.
The following line will have the following effects
someVariable=someValue // not a comment
- Assign
someValue
to variable someVariable
, but only for that one line. After that line the variable will go back to its old value, which is null in this case.
- Execute the "command"
// not a comment
, that is the program //
with the parameters not
, a
, and comment
. Since //
is just a directory (the same as /
) this will cause an error message and nothing more.
Right now this behavior might seem strange, but you may have already used it in well known idioms like IFS= read -r line
or LC_ALL=C sort
.
Looking at your script the following lines probably caused the problem:
OUT_DIR=/data/backup/mongodb/tmp // ...
...
rm -rf $OUT_DIR/*
I'm sorry to bring this to you, but you basically executed rm -rf /*
since $OUT_DIR
expanded to the empty string.
Potential Risk On Other Systems
Even if $OUT_DIR
wasn't empty the effect could have been the same since there is a //
"comment" after rm
. Consider the command
rm -rf some // thing
This is supposed to delete the three files/directories some
, //
, and thing
. As already pointed out //
is the same directory as /
.
However, most implementations of rm
on Linux have a guard for this case and won't delete /
so easily. On Ubuntu you will get the following warning (don't try this at home. Would suck if your rm
differs.)
$ rm -rf //
rm: it is dangerous to operate recursively on '//' (same as '/')
rm: use --no-preserve-root to override this failsafe
Question 2
2) And of cause, is it possible to get the /data/ back?
This is off-topic for StackOverflow. However, you can find many answers to this question on other stackexchange sites.
There are recovery tools you can try, but there is no guarantee that you can restore your data if you don't have a backup.
Switching between languages can be tricky! //
is not a comment starter in shell, it's #
. All the commands with these "comments" were parsed wrongly and skipped:
$ VAR=whatever // comment
bash: //: Is a directory
[$?=126]
$ echo "($VAR)"
()
Therefore, OUT_DIR=...
was ignored and $OUT_DIR
was empty. It's easy to guess what
rm -rf $OUT_DIR/*
then did. It was basically equivalent to
rm -rf /*
Use your backups to restore the database.
I can read the Chinese wordings in comment field, from line 10, the user wants to create a temp folder but used cd
, so if /data/backup/mongodb/tmp
does not exist in the first place, then $OUT_DIR is empty or null, after that line 11 became rm -rf /*
.