Debian Package of the Day (static archived copy)

Entry submitted by John Beisley. DPOTD needs your help, please contribute !

Moreutils contains a suite of utilities for command-line users. Written by multiple authors, they are individually quite modest, and perhaps too limited in scope to exist in a package in their own right, but together they form a useful accompaniment to existing command-line tools.

Following are a few examples of the utilities in moreutils, and how they can be used.

Sponge - Soaking file redirects

Many users of the command line have made a mistake such as the following:

$ grep -v someuser /etc/passwd > /etc/passwd

As the shell runs this command it will immediately truncate the destination file, and then run the grep command on it. The end result is an empty file - certainly not what was intended! After learning from their mistake, the user might instead redirect the output to a temporary file, then move the file on top of the original:

$ grep -v someuser /etc/passwd > /etc/passwd.tmp
$ mv /etc/passwd.tmp /etc/passwd

Sponge is a simple, but convenient, command that exists to roll this process up into a single step:

$ grep -v someuser /etc/passwd | sponge /etc/passwd

Sponge ’soaks up’ its standard input, waiting for it to complete before writing to the specified output file. In this way the result is what was intended, and without the need to fiddle with temporary files.

Vipe - Interactively editing pipes

Sometimes it is desirable to put yourself in the middle of a pipe processing chain, where it will be quicker to just edit the pipe content, rather than create complex filters with sed and friends. Maybe the pipe content will be unknown, perhaps a list of files that the user would like to manually filter.

Say that you would like to touch all files in the current directory, but would like to do so selectively, you could do the following:

ls | vipe | xargs -d ‘\n’ touch

In this example a list of files is opened up in your editor (as set by either the EDITOR or VISUAL environment variables), in which you may edit the list of files to be touched at your leisure. When you are satisfied, you can save the file and quit your editor, and vipe will regurgitate the content you edited to its standard-out - touching only the files retained in the editor.

Combine - Boolean/Set operations with text files

Performing set operations with lines in two text files can also be useful. This can be useful in knowing what file names are common to two directories, for example:

Directory foo contains the files:

tom and harry

Directory bar contains the files:

tom and dick

The file lists can be written to two files ready for use with combine with the following bash command:

ls /path/to/foo > foo_list && ls /path/to/bar > bar_list

It can be complicated to find which files are present in both directories, which is where combine really comes into its own:

$ combine foo_list and bar_list
tom

How about files that are in foo, but not in bar?

$ combine foo_list not bar_list
harry

A note of caution, however: while using combine, ‘or’ can be used to find the complete list of files, it has the slightly odd effect of listing files common to both files twice:

$ combine foo_list or bar_list
harry
tom
dick
tom

Obtaining the union of the two lists of file names is therefore best performed with the standard sort command:

$ cat foo_list bar_list | sort -u
dick
harry
tom

Summary

This introduction to moreutils has only touched upon some of the commands that it provides. It is worth experimenting with the others, very briefly described here:

isutf8

Checks if a file consists of valid UTF-8

ts

Reproduces standard input on standard output, with a timestamp prefix - good for annotating non-timestamped log files.

vidir

Enables a user to “edit” a directory with their text editor. That is, to conveniently delete and rename files.

ifdata

Outputs requested information about a specific Linux network interface, without the need to parse ifconfig output.

pee

Acts like tee, but pipes to given commands, instead of standard out and files.

zrun

Automatically recognises compressed files passed to a given command, uncompresses them to a temporary file and passes the uncompressed files in place of the original compressed file arguments.

mispipe

Connects two commands with a pipe, as in command1 | command2, but returns the exit value of the first command, rather than the second.

moreutils has been available in Debian in testing and unstable distributions, and in Ubuntu from Edgy onwards.