Unix & Linux
files logs open-files file-descriptors
Updated Wed, 05 Oct 2022 14:09:07 GMT

How can a log program continue to log to a deleted file?


From the Unix Power Tools, 3rd Edition: Instead of Removing a File, Empty It section:

If an active process has the file open (not uncommon for log files), removing the file and creating a new one will not affect the logging program; those messages will just keep going to the file thats no longer linked. Emptying the file doesnt break the association, and so it clears the file without affecting the logging program.

(emphasis mine)

I don't understand why a program will continue to log to a deleted file. Is it because the file descriptor entry not getting removed from the process table?

Solution

When you delete a file you really remove a link to the file (to the inode). If someone already has that file open, they get to keep the file descriptor they have. The file remains on disk, taking up space, and can be written to and read from if you have access to it.

The unlink function is defined with this behaviour by POSIX:

When the file's link count becomes 0 and no process has the file open, the space occupied by the file shall be freed and the file shall no longer be accessible. If one or more processes have the file open when the last link is removed, the link shall be removed before unlink() returns, but the removal of the file contents shall be postponed until all references to the file are closed.

This piece of advice because of that behaviour. The daemon will have the file open, and won't notice that it has been deleted (unless it was monitoring it specifically, which is uncommon). It will keep blithely writing to the existing file descriptor it has: you'll keep taking up (more) space on disk, but you won't be able to see any of the messages it writes, so you're really in the worst of both worlds. If you truncate the file to zero length instead then the space is freed up immediately, and any new messages will be appended at the new end of the file where you can see them.

Eventually, when the daemon terminates or closes the file, the space will be freed up. Nobody new can open the file in the mean time (other than through system-specific reflective interfaces like Linux's /proc/x/fd/...). It's also guaranteed that:

If the link count of the file is 0, when all file descriptors associated with the file are closed, the space occupied by the file shall be freed and the file shall no longer be accessible.

So you don't lose your disk space permanently, but you don't gain anything by deleting the file and you lose access to new messages.

Comments (5)

  • +1 – What will occur if a user (let's say root here) attempt to unlink /proc/x/fd/y? Would that cause the process to fail to write to the file descriptor, or is that an illegal operation? — Jul 28, 2014 at 11:11  
  • +0 – @hexafraction /proc/*/fd/* are symlinks to real files, so removing them won't delete the file. I'd suggest you to experiment :) (not on production system of course!) — Jul 28, 2014 at 11:19  
  • +1 – @MichaelHomer Perhaps you could clarify in your answer that once a file is unlinked, the process having a file descriptor pointing to it can link it again, at the same path or not. This can sometimes be useful. — Jul 28, 2014 at 14:08  
  • +0 – @hexafraction Well, these are just representations (in the filesystem space) of process state and objects. If you remove those representations in the filesystem space, nothing should happen to the actual process - unless it (or some other process) relies on that representation being there. Not sure you can use rm incontinently inside /proc or /sys without getting told off by the system anyway. — Jul 28, 2014 at 17:54  
  • +0 – @lgeorget How is that accomplished? — Jul 28, 2014 at 18:30  

External Links

External links referenced by this document:

References