moreutils: A collection of useful command-line tools
April 15th, 2007 edited by TinchoEntry submitted by John Beisley. DPOTD needs your help, please contribute !
Moreutils contains a suite of utilities for command-line users. Written by multiple authors, they are individually quite modest, and perhaps too limited in scope to exist in a package in their own right, but together they form a useful accompaniment to existing command-line tools.
Following are a few examples of the utilities in moreutils, and how they can be used.
Sponge - Soaking file redirects
Many users of the command line have made a mistake such as the following:
$ grep -v someuser /etc/passwd > /etc/passwd
As the shell runs this command it will immediately truncate the destination file, and then run the grep command on it. The end result is an empty file - certainly not what was intended! After learning from their mistake, the user might instead redirect the output to a temporary file, then move the file on top of the original:
$ grep -v someuser /etc/passwd > /etc/passwd.tmp
$ mv /etc/passwd.tmp /etc/passwd
Sponge is a simple, but convenient, command that exists to roll this process up into a single step:
$ grep -v someuser /etc/passwd | sponge /etc/passwd
Sponge ’soaks up’ its standard input, waiting for it to complete before writing to the specified output file. In this way the result is what was intended, and without the need to fiddle with temporary files.
Vipe - Interactively editing pipes
Sometimes it is desirable to put yourself in the middle of a pipe processing chain, where it will be quicker to just edit the pipe content, rather than create complex filters with sed and friends. Maybe the pipe content will be unknown, perhaps a list of files that the user would like to manually filter.
Say that you would like to touch all files in the current directory, but would like to do so selectively, you could do the following:
ls | vipe | xargs -d ‘\n’ touch
In this example a list of files is opened up in your editor (as set by either the EDITOR or VISUAL environment variables), in which you may edit the list of files to be touched at your leisure. When you are satisfied, you can save the file and quit your editor, and vipe will regurgitate the content you edited to its standard-out - touching only the files retained in the editor.
Combine - Boolean/Set operations with text files
Performing set operations with lines in two text files can also be useful. This can be useful in knowing what file names are common to two directories, for example:
- Directory foo contains the files:
- tom and harry
- Directory bar contains the files:
- tom and dick
The file lists can be written to two files ready for use with combine with the following bash command:
ls /path/to/foo > foo_list && ls /path/to/bar > bar_list
It can be complicated to find which files are present in both directories, which is where combine really comes into its own:
$ combine foo_list and bar_list
tom
How about files that are in foo, but not in bar?
$ combine foo_list not bar_list
harry
A note of caution, however: while using combine, ‘or’ can be used to find the complete list of files, it has the slightly odd effect of listing files common to both files twice:
$ combine foo_list or bar_list
harry
tom
dick
tom
Obtaining the union of the two lists of file names is therefore best performed with the standard sort command:
$ cat foo_list bar_list | sort -u
dick
harry
tom
Summary
This introduction to moreutils has only touched upon some of the commands that it provides. It is worth experimenting with the others, very briefly described here:
- isutf8
- Checks if a file consists of valid UTF-8
- ts
- Reproduces standard input on standard output, with a timestamp prefix - good for annotating non-timestamped log files.
- vidir
- Enables a user to “edit” a directory with their text editor. That is, to conveniently delete and rename files.
- ifdata
- Outputs requested information about a specific Linux network interface, without the need to parse ifconfig output.
- pee
- Acts like tee, but pipes to given commands, instead of standard out and files.
- zrun
- Automatically recognises compressed files passed to a given command, uncompresses them to a temporary file and passes the uncompressed files in place of the original compressed file arguments.
- mispipe
- Connects two commands with a pipe, as in command1 | command2, but returns the exit value of the first command, rather than the second.
moreutils has been available in Debian in testing and unstable distributions, and in Ubuntu from Edgy onwards.
Posted in Debian, Ubuntu | 5 Comments »
