recursive grep started at / hangs

I have used following grep search pattern on multiple platforms:

grep -r -I -D skip 'string_to_match' /

For example on FreeBSD 8.0, FreeBSD 6.4 and Debian 6.0(squeeze). Command does a recursive search starting from root directory, assumes that binary files do not have the 'string_to_match' and skips devices, sockets and named pipes. FreeBSD 8.0 and FreeBSD 6.4 use GNU grep version 2.5.1 and Debian 6.0 uses GNU grep version 2.6.3. On FreeBSD 6.4, last information printed to stderr was "grep: /dev/cuad0: Device busy". After this grep just idles as according to "top -m io -o total" the I/O usage of grep is nonexistent. Same behavior is true under FreeBSD 8.0, but last information sent to stderr is "grep: /tmp/.wine-0: Permission denied" on my installation. In case of Debian, last output to stderr is "grep: /proc/sysrq-trigger: Input/output error". If I check the I/O usage of grep process under Debian, it is following:

root@Debian:~# iotop -bp 22439
Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
22439 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % grep -r -I
-D skip 10.10.10.99 /
Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
22439 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % grep -r -I
-D skip 10.10.10.99 /
Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
22439 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % grep -r -I
-D skip 10.10.10.99 /
^Croot@Debian:~#

What might cause this? Is there a way to view which file grep is currently processing in case lsof is not present? I'm able to use lsof under Debian and looks like the problematic file name there is "0xc6b2c230 file struct, ty=0, op=0xc0d34120". I'm not sure what this is.. I'm not able to use lsof or fstat under FreeBSD.

PS: I know I could use find utility, but this is not the question.

Solution

On Linux, instead of lsof you can list the contents of /proc/$(pgrep grep)/fd. You should see a symlink for each open file descriptor for that process and it will point to the file that it corresponds to. Here is a sample of what this did for me:

$ ls -gG /proc/$(pgrep grep)/fd
total 0
lrwx------ 1 64 Oct 18 19:39 0 -> /dev/pts/2
lrwx------ 1 64 Oct 18 19:39 1 -> /dev/pts/2
lrwx------ 1 64 Oct 18 19:39 2 -> /dev/pts/2
lr-x------ 1 64 Oct 18 19:39 3 -> /usr/share/groff/1.18.1.4/font/devdvi/generate/texb.map

Comments (2)

+0 – Unfortunately, looks like this is not possible under FreeBSD. I executed grep -r -I -D skip 'string_to_match' / under OpenSUSE 12.1 and there it got stuck to /proc/kmsg file, which is a file used for holding messages generated by kernel. — Oct 19, 2012 at 08:43
+0 – /proc/kmsg is likley to always be a problem so you may want to consider the -x option. Other than that, isn't it possible to attach to the process with a debugger and figure out the file it is stuck on? — Oct 21, 2012 at 11:39

References

https://serverfault.com/questions/439705