Recovering Deleted Files With lsof

One of the more neat things you can do with the versatile utility lsof is use it to recover a file you've just accidentally deleted.

A file in Linux is a pointer to an inode, which contains the file data (permissions, owner and where its actual content lives on the disk). Deleting the file removes the link, but not the inode itself – if another process has it open, the inode isn't released for writing until that process is done with it.

To try this out, create a test text file, save it and then type less test.txt. Open another terminal window, and type rm testing.txt. If you try ls testing.txt you'll get an error message. But! less still has a reference to the file. So:

# lsof | grep testing.txt
less 4607 juliet 4r REG 254,4 21
8880214 /home/juliet/testing.txt (deleted)

The important columns are the second one, which gives you the PID of the process that has the file open (4607), and the fourth one, which gives you the file descriptor (4). Now, we go look in /proc, where there will still be a reference to the inode, from which you can copy the file back out:

# ls -l /proc/4607/fd/4
lr-x------ 1 juliet juliet 64 Apr 7 03:19
/proc/4607/fd/4 -> /home/juliet/testing.txt (deleted)
# cp /proc/4607/fd/4 testing.txt.bk

Note: don't use the -a flag with cp, as this will copy the (broken) symbolic link, rather than the actual file contents.

Now check the file to make sure you've got what you think you have, and you're done!

Cheers
Nobs

Difference Between Soft Link and Hard Link

Hard Links :

1. All Links have same inode number.

2.ls -l command shows all the links with the link column(Second) shows No. of links.

3. Links have actual file contents

4.Removing any link ,just reduces the link count , but doesn’t affect other links.

Soft Links(Symbolic Links) :

1.Links have different inode numbers.

2. ls -l command shows all links with second column value 1 and the link points to original file.

3. Link has the path for original file and not the contents.

4.Removing soft link doesn’t affect anything but removing original file ,the link becomes “dangling” link which points to nonexistant file.

Regards
Nobs

Migrating LVM Volumes Over Network (using snapshots)

We run a big share of Xen virtual servers spanned over multiple servers and if you want to use the full or best capability of Xen, I would suggest LVM (Logical Volume Manager), it makes life a lot easier, especially for those who do not run a RAID setup (We run RAID10 on all VM nodes) as you can split the partition over multiple hard drives. I’m not going to cover setting up the LVM as there are loads of tutorials on how to do that but I will rather cover the best way to migrate a LVM volume.

First, we will need to create a snapshot of the LVM volume as we cannot create an image of the live version, we run the following line:

lvcreate -L20G -s -n storageLV_s /dev/vGroup/storageLV

The 20G part is the size of the snapshot LVM, I would suggest looking up the size of the real original LV and making it the same, you can find out the size of the LV by using this command:
lvdisplay /dev/vGroup/storageLV —
There will be a “LV Size” field, get it from there and put it in the command, the -n switch is for the name, usually I name them the same as the LV with a trailing _s for snapshot, the last argument is simply the real LV that we want to make a snapshot of.

Afterwards, we will use dd in different way, usually if you use dd in one line, it’s either reading or it’s either writing which makes it crawl, to bypass this, we will read the LV and pipe it to one that writes so the minimum speed is the fastest speed of the slowest hard drive (I could re-phrase that but it’s 11:10 PM!) — To speed it up a bit more, we used a block size of 64K.

dd if=/dev/vGroup/storageLV_s conv=noerror,sync bs=64k | dd of=/migrate/storageLV_s.dd bs=64k


I won’t cover the file transfer process as there are multiple methods, if you want to use SCP, I would suggest disabling encryption or anything as it really slows it down, our node usually has httpd installed on them so I simply changed the configuration to listen on a different port (for security) and changed the DocumentRoot to /migrate

Once you got your file on the server, you’ll need to re-create the LV on the target server, you’ll need to run this

lvcreate -L20G -n storageLV vGroup

You’ll have to keep the same size, bring the same name (this time without a trailing _s as it won’t be a snapshot) and the volume group at the end.

The last step is to actually restore the image using dd, again using our block-size & pipe tweak for better performance.

dd if=/migrate/storageLV_s.dd conv=noerror,sync bs=64k | dd of=/dev/vGroup/storageLV bs=64k


I have migrated around 16 LVs with this method without any problems, 13 of them were 20G each, 2 40G and 1 75G — So far every part is fast however I have to admit that the slowest part was the file transfer, I would suggest using a Gbit crossover or even better if you have a Gbit switch, if you don’t but you’re right next to the server, might consider using a spare USB 2.0 HDD as they are much faster compared to 100mbps (USB2.0 is around 480Mbps).

Thanks
Nobs

Load Alert Script for Server

The following script will send a mail to the email address mentioned, if the server load goes above 5. The mail will contain the top 10 cpu consuming processes, memory consuming processes, memory and swap status, disk space information and the uptime.

1 . Login to the server as root

# vi /root/loadalert

2. And the below script
-------------------------------------------

#!/bin/bash
#Wednesday, December 06 2006
EMAIL="test@gmail.com"
EMAIL1="test@yahoo.com"
SUBJECT="$(hostname) load is"
TEMPFILE="/tmp/$(hostname)"
echo "Load average has crossed the limits..." >> $TEMPFILE
echo "Hostname: $(hostname)" >> $TEMPFILE
echo "Local Date & Time : $(date)" >> $TEMPFILE
echo "| Uptime status: |" >> $TEMPFILE
echo "------------------" >> $TEMPFILE
/usr/bin/uptime >> $TEMPFILE
echo "------------------" >> $TEMPFILE
echo "| Top 20 CPU consuming processes: |" >> $TEMPFILE
ps aux | head -1 >> $TEMPFILE
ps aux --no-headers | sort -rn +2 | head -20 >> $TEMPFILE
echo "| Top 10 memory-consuming processes: |" >> $TEMPFILE
ps aux --no-headers| sort -rn +3 | head >> $TEMPFILE
echo "---------------------------" >> $TEMPFILE
echo "| Memory and Swap status: |" >> $TEMPFILE
/usr/bin/free -m >> $TEMPFILE
echo "------------------------------" >> $TEMPFILE
echo "| Disk Space information: |" >> $TEMPFILE
echo "---------------------------" >> $TEMPFILE
/bin/df -h >> $TEMPFILE
echo "------THE END----------------" >> $TEMPFILE
L05="$(uptime|awk '{print $(NF-2)}'|cut -d. -f1)"
if test $L05 -gt 5
then
mail -s "$SUBJECT $L05" "$EMAIL" < $TEMPFILE
mail -s "$SUBJECT $L05" "$EMAIL1" < $TEMPFILE
fi
rm -f $TEMPFILE

-----------------------------------
Change permission

3 . chmod +x /root/loadalert

4. Add cron

# vi /var/spool/cron/root

* * * * * /root/loadalert >/dev/null 2>&1

5. Restart Cron

# /etc/init.d/crond restart

6. Check cron log for error mesage...

# tail -f /var/log/cron

Thanks
Nobs

Volume Labels

Volume labels make it possible for partitions to retain a consistent name. Each can be a maximum of 16 characters long. There are three tools to make volume labels: mke2fs, tune2fs and e2label.

e2label /dev/hda1 omega

tune2fs -L omega /dev/hda1

The above 2 commands will label the first partition of the drive “omega”. That label stays with that particular partition, even if the drive is moved to another controller or even another computer.

mke2fs omega /dev/hda1

mke2fs -L omega /dev/hda1

The above command also will do the label but only after they make the file system. This means that either of these last two commands will delete any existing data in the partition.

Cheers
Nobs

CSF Installation

Install
-------
rm -fv csf.tgz
wget http://www.configserver.com/free/csf.tgz
tar -xzf csf.tgz
cd csf
sh install.sh

If you would like to disable APF+BFD (which you will need to do if you have
them installed otherwise they will conflict horribly):

sh disable_apf_bfd.sh

That’s it. You can then configure csf and lfd in WHM, or edit the files
directly in /etc/csf/*

CSF is pre configured to work on a cPanel server with all the standard cPanel
ports open. It also auto-configures your SSH port if it’s non-standard on
installation.

You should ensure that kernel logging daemon (klogd) is enabled. Typically, VPS
servers have this disabled and you should check /etc/init.d/syslog and make
sure that any klogd lines are not commented out. If you change the file,
remember to restart syslog.

Uninstallation
------------------
Removing csf and lfd is even more simple:

cd /etc/csf
sh uninstall.sh

Thanks
Nobs

About Kernel Versioning

Command to show the running kernel version:

[root@wordsworth modules]# uname -r
2.6.9-42.0.3.ELsmp

Kernel Version Numbers:

The Linux kernel version numbers consist of three numbers separated by decimals, such as 2.2.14. The first number is the major version number. The second number is the minor revision number. The third number is the patch level version.

At any given time there is a group of kernels that are considered “stable releases” and another group that is considered “development.” If the second number of a kernel is even, then that kernel is a stable release. For example, the 2.2.14 kernel is a stable release because the second number is even. If the second number is odd, then that kernel is a development release. For example, the 2.3.51 is a development release because the second nubmer is odd.

Once the 2.3.x branch is considered finished, then it will become the 2.4.0 kernel. Patches will then appear for the 2.4.x branch and development work will begin on the 2.5.x branch. If the 2.3.x advancements are significant enough to be considered a major revision, the 2.3.x branch will become 3.0.0 and development work will begin on the 3.1.x branch.

Thanks
Nobs