Chapter 1 : Introduction to the command line

In Chapter 4.0 you were shown how to launch a shell. In this chapter we will show you how to put it to work.

The shell's main asset is the number of existing utilities: there are thousands of them, and each one is devoted to a particular task. We will only look at a small number of them here. One of Unix's greatest assets is the ability to combine these utilities, as we shall see later.

File Handling Utilities

File handling here means copying, moving and deleting files. Later, we will look at ways of changing their attributes (owner, permissions).

mkdir, touch: creating empty directories and files (MaKe DIRectory)

mkdir is used for creating directories. Its syntax is simple:

mkdir [options] <directory> [directory ...]

Only one option is worth noting: the option -p. If this option is set, mkdir will create parent directories if these did not exist before. If this option is not specified and the parent directories do not exist, mkdir will display an error. Examples:

• mkdir foo: creates a directory foo in the current directory;
• mkdir -p images/misc docs: creates a directory misc in directory images by first creating the latter if it does not exist, together with a directory docs.

Initially, the touch command is not intended for creating files but for updating file access and modification times[3]. However, one of its side-effects is to create the files mentioned if they did not exist before. The syntax is:

touch [options] file [file...]

So running the command:

touch file1 images/file2

will create a file called file1 in the current directory and a file file2 in directory images.

rm: deleting files or directories (ReMove)

This command replaces the DOS commands del and deltree, and more. Its syntax is as follows:

rm [options] <file|directory> [file|directory...]

Options include:

• -r, or -R: Delete recursively. This option is mandatory for deleting a directory, empty or not. However, there is also the command rmdir for deleting empty directories.
• -i: Request confirmation before each deletion. It is recommended to alias the bare rm word to rm -i in your shell, and the same goes for cp and mv commands.
• -f: The opposite of -i, forces deletion of the files or directories, even if the user has no write authorisation on the files[4].

Some examples:

• rm -i images/*.jpg file1: Deletes all files which name ends with .jpg in directory images and file file1 in the current directory, requesting confirmation for each file. Answer 'y' to confirm deletion, 'n' to cancel.
• rm -Rf images/misc/ file*: Deletes without requesting confirmation the whole directory misc/ in directory images/ together with all files in the current directory which name begins with file.

Warning: a file deleted using rm is deleted irrevocaby. There is no way of restoring the files! Don't hesitate to use the -i option...

mv: moving or renaming files (MoVe)

The syntax of the mv command is as follows:

mv [options] <file|directory> [file|directory ...] <destination>

Some options:

• -f: Forces file moving -- no warning if an existing file is overwritten by the operation.
• -i: The opposite -- ask the user for confirmation before overwriting an existing file.
• -v: Verbose mode, report all changes.

Some examples:

• mv -i /tmp/pics/*.gif .: Move all files in directory /tmp/pics/ which name ends with .gif to the current directory (.), requesting confirmation before overwriting any files.
• mv foo bar: Rename file foo as bar.
• mv -vf file* images/ trash/: Move, without requesting confirmation, all files in the current directory which name begins with file together with the entire directory images/ to directory trash/, and show each operation carried out.

cp: copying files and directories (CoPy)

cp replaces the DOS commands copy, xcopy and more. Its syntax is as follows:

cp [options] <file|directory> [file|directory ...] <destination>

It has a bunch of options. These are the most common:

• -R: Recursive copy; mandatory for copying a directory, even empty.
• -i: Request confirmation before overwriting any files which might be overwritten.
• -f: The opposite of -i, replace any existing files without requesting confirmation.
• -v: Verbose mode, records all actions performed by cp.

Some examples:

• cp -i /tmp/images/* images/: Copies all files from directory /tmp/images to directory images/ of the current directory, requesting confirmation if a file is going to be overwritten.
• cp -vR docs/ /shared/mp3s/* mystuff/: Copies the whole directory docs to the current directory plus all files in directory /shared/mp3s to directory mystuff located in the current directory.
• cp foo bar: Makes a copy of file foo under the name bar in the current directory.

Handling File Attributes

The series of commands shown here is used to change the owner or owner group of a file or its permissions. We looked at the different permissions in chapter 4.0.

chown, chgrp: change the owner and group of one or more files (CHange OWNer, CHange GRouP)

The syntax of the chown command is as follows:

chown [options] <user[.group]> <file|directory> [file|directory...]

The options include:

• -R: Recursive; to change the owner of all files and subdirectories in a given directory.
• -v: Verbose mode; describes all actions performed by chown; reports which files have changed owner as a result of the command and which files have not been changed.
• -c: Like -v, but only reports which files have been changed.

Some examples:

• chown nobody /shared/book.tex: changes the owner of file /shared/book.tex to nobody.
• chown -Rc john.music *.mid concerts/: attributes all files in the current directory ending with .mid and all files and subdirectories in directory concerts/ to user john and to group music, reporting only files affected by the command.

The chgrp command lets you change the group ownership of a file (or files); its syntax is very similar to that of chown:

chgrp [options] <group> <file|directory> [file|directory...]

The options for this command are the same as for chown, and it is used in a very similar way. Thus, the command:

chgrp disk /dev/hd*

attributes to the group disk all files in directory /dev/ which name begins with hd.

chmod: changing permissions on files and directories (CHange MODe)

The chmod command has a very distinct syntax. The general syntax is:

chmod [options] <change mode> <file|directory> [file|directory...]

but what distinguishes it is the different forms that the mode change can take. It can be specified in two ways:

• in octal; the owner user permissions then correspond to figures with the form <x>00, where <x> corresponds to the permission assigned: 4 for read permission, 2 for write permission and 1 for execution permission; similarly, the owner group permissions take the form <x>0 and permissions for "others" the form <x>. Then all you need to do is add together the assigned permissions to get the right figure. Thus, the permissions rwxr-xr-- correspond to 400+200+100 (owner permissions, rwx) +40+10 (group permissions, r-x) +4 (others' permissions, r--) = 754; in this way, the permissions are expressed in absolute terms: previous permissions are unconditionally replaced;
• with expressions: here permissions are expressed by a sequence of expressions separated by commas, with an expression taking the form [category]<+|-><permissions>. The category may be one or more u (User, permissions for owner), g (Group, permissions for owner group) or o (Others, permissions for "others"). If no category is specified the changes apply to all categories. A + sets a permission, a - removes the permission. Finally, the permission is one or more of r (Read), w (Write) or x (eXecute).

The main options are quite similar to those of chown or chgrp:

• -R: Change permissions recursively.
• -v: Verbose mode, describes actions carried out for each file.
• -c: Like -v but only shows files for which there has been a change of permissions.

Examples:

• chmod -R o-w /shared/docs: Recursively removes write permission for "others" on all files and subdirectories of /shared/docs/.
• chmod -R og-w,o-x private/: Recursively removes write permission for the group and others for the whole directory private/, and removes the execution permission for others.
• chmod -c 644 miscellaneous/file*: changes permissions of all files in directory miscellaneous/ with names beginning with file to rw-r--r-- (i.e. read permission for everyone and write permission only for the owner), and reports only files where a change has been made.

Shell globbing patterns and regular expressions

You probably already use globbing characters without knowing it. When you write a file in a Windows or when you look for a file, you use * to match a random string. For example, *.txt matches all files which name end with .txt. We also used it heavily in the last section. But there is more to globbing than *.

When you type a command like ls *.txt and press Return, the task of finding which files match the pattern *.txt is not done by the ls command, but by the shell itself. This requires a little explanation about how a command line is interpreted by the shell. When you type:

$ ls *.txt
readme.txt  recipes.txt

the command line is first split into words (ls and *.txt in this example) . When it sees a * in a word, it will interpret the whole word as a globbing pattern and will replace it with the names of all matching files. Therefore, the line just before the shell executes it has become ls readme.txt recipe.txt, which gives the expected result. Other characters make the shell react this way:

• ?: matches one and only one character, whatever that character;
• [...]: matches any character found into the brackets; characters can be referred to either as a range of characters (e.g, 1-9) or discrete values, or even both. Example: [a-zBE5-7] will match all characters a to z, a B, a E, a 5, a 6 or a 7;
• [!...]: matches any character not found in the brackets. [!a-z], for example, will match any character which is not a lowercase letter;
• {c1,c2}: matches c1 or c2, where c1 and c2 are also globbing patterns.

Here are some patterns and their meaning:

• /etc/*conf: All files in /etc which name end with conf. It can match /etc/inetd.conf, but it can also match /etc/conf.linuxconf, and also /etc/conf if such a file exists. Remember that * can match an empty string.
• image/cars,space[0-9]/*.jpg: All filenames ending with .jpg in directory image/cars, image/space0, ... , image/space9, if such directories exist.
• /usr/doc/*/README: All files named README in all immediate subdirectories of /usr/doc. This will make /usr/doc/mandrake/README match for example, but not /usr/doc/myprog/doc/README.
• *[!a-z]: All files which names do not end with a lowercase letter in the current directory.

Redirections and pipes

A little more about processes

To understand the principle of redirections and pipes, we need to explain a notion about processes which has not yet been introduced. Each Unix process (this also includes graphical applications) opens a minimum of three file descriptors: standard input, standard output and standard error. Their respective numbers are 0, 1 and 2. In general, these three descriptors are associated with the terminal from which the process was started, the input being the keyboard. The aim of redirections and pipes is to redirect these descriptors. The examples in this section will help you understand better.

Redirections

Imagine, for example, that you wanted a list of files ending with .gif[5] in directory images. This list is very long, so you want to store it in a file to look at it at leisure subsequently. You can enter the following command:

$ ls images/*.gif 1>file_list

This means that the standard output of this command (1) is redirected (>) to the file named file_list. The operator > is the output redirection operator. If the redirection file does not exist, it is created, but if it exists its previous contents are overwritten. However, the default descriptor redirected by this operator is the standard output and does not need to be specified on the command line. So you can write more simply:

$ ls images/*.gif >file_list

and the result will be exactly the same. Next, you can look at the file using a text file viewer such as less.

Now imagine that you want to know how many of these files there are. Instead of counting them by hand, you can use the utility called wc (Word Count) with the option -l, which writes on the standard output the number of lines in the file. One solution is as follows:

wc -l 0<file_list

and this gives the desired result. The operator < is the input redirection operator, and similarly the default redirected descriptor is the standard input one, i.e. 0, and you simply need to write the line:

wc -l <file_list

Now suppose that you want to look at this, remove all the file "extensions" and put the result in another file. One tool for doing this is sed, i.e. Stream EDitor. You simply redirect the standard input of sed to the file file_list and redirect its output to the result file, e.g. the_list:

sed -e 's/.gif$//g' <file_list >the_list

and there is your list created, ready for consultation at leisure with a viewer.

It can also be useful to redirect standard errors. For example, you want to know which directories in /shared you cannot access: one solution is to list this directory recursively and to redirect the errors to a file, while not displaying the standard output:

ls -R /shared >/dev/null 2>errors

which means that the standard output will be redirected (>) to /dev/null, a special file in which everything you write is lost (i.e. as a side effect the standard output is not displayed) and the standard error channel (2) is redirected (>) to the file errors.

Pipes

Pipes are in some way a combination of input and output redirections. The principle is that of a pipe, hence the name: one process sends data into one end of the pipe and another process reads the data at the other end. The pipe operator is |. Let us go back to the example of the file list above. Suppose you want to find out directly how many corresponding files there are without having to store the list in a temporary file, you then use the following command:

ls images/*.gif | wc -l

which means that the standard output of the ls command (i.e. the list of files) is redirected to the standard input of the wc command. This then gives you the desired result.

You can also directly put together a list of files "without extensions" using the following command:

ls images/*.gif | sed -e 's/.gif$//g' >the_list

or, if you want to consult the list directly without storing it in a file:

ls images/*.gif | sed -e 's/.gif$//g' | less

Pipes and redirections are not restricted solely to text that can be read by human beings. For example, the following command sent from a xterm:

xwd -root | convert -  /my_desktop.gif

will send a screenshot of your desktop to the my_desktop.gif[6] file in your personal directory.

Completion

Completion is a very handy functionality, and all modern shells (including Bash) have it. Its role is to give the user as little work to do as possible. The best way to illustrate completion is to give an example.

Example

Suppose your personal directory contains a file which name is file_with_very_long_name_impossible_to_type, and you want to look at it. Suppose you also have in the same directory another file called file_text. You are in your personal directory. So you type the following sequence:

$ less fi<TAB>

(i.e., type less fi and then press the TAB key). The shell will then extend the command line for you:)

$ less file_

and also give the list of possible choices (in its default configuration, which can be customised). Then type the following sequence of keys:

less file_w<TAB>

and the shell will extend the command line to give you the result you want:

less file_with_very_long_name_impossible_to_type

All you need to do then is press the Enter key to confirm and read the file.

More generally

The TAB key is not the only way to activate completion, although it is the most common. As a general rule, the word to be completed will be a command name for the first word of the command line (nsl<TAB> will give nslookup), and a file name for all the others, unless the word is preceded by a "magic" character from  , @ or $, in which case the shell will respectively try to complete a user name, a machine name or an environment variable name[7]. There is also a magic character for completing a command name (!) or a file name (/).

The other two ways to activate completion are the sequences Esc-<x> and C-x <x> (Esc being the Escape key, and C-x meaning Control+<x>), where <x> is one of the magic characters already mentioned. Esc-<x> will attempt to come up with a unique completion, and if it fails will complete the word with the largest possible substring in the list of choices. A beep means either that the choice is not unique, or quite simply that there is no corresponding choice. The sequence C-x <x> displays the list of possible choices without attempting any completion. Pressing the TAB key is the same as successively pressing Esc-<x> and C-x <x>, where the magic character depends on the context.

Thus, one way to see all the environment variables defined is to type the sequence C-x $ in a blank line. Another example: if you want to see the page of the manual for the command nslookup, you simply type man nsl then Esc-!, and the shell will automatically complete to man nslookup.

Starting and handling background processes: job control

You will have noticed that when you send an order from a terminal, you normally have to wait for the command to finish before the shell returns control to you: you sent the command in the foreground. However, there are occasions when this is not desirable.

Suppose, for example, that you have decided to copy a large directory recursively to another. You also decided to ignore errors, so you redirect the error channel to /dev/null:

cp -R images/ /shared/ 2>/dev/null

A command like this can take several minutes to finish. You then have two solutions: the first is violent, and means stopping (killing) the command and then doing it again when you have the time. To do this, type C-c (Control+'c'): this will take you back to the prompt.

But suppose you want the command to run while doing something else. The solution is then to shift the process to the background. To do this, type C-z to suspend the process:

$ cp -R images/ /shared/ 2>/dev/null
  # Type C-z here
[1]+  Stopped                 cp -R images/ /shared/ 2>/dev/null
$

and there you are again with the prompt. The process is then on standby, waiting for you to restart it (as shown by the keyword Stopped). That, of course, is what you want to do, but in the background. Type bg (for BackGround) to get the desired result:

$ bg
[1]+ cp -R images/ /shared/ 2>/dev/null &
$

The process will then start running again as a background task, as indicated by the & (ampersand) sign at the end of the line. You will then be back at the prompt and able to continue working. A process which runs as a background task, or in the background, is called a job.

Of course, you can start processes directly as background tasks, precisely by adding an '&' at the end of the command. So, for example, you can start copying the directory in the background by writing:

cp -R images/ /shared/ 2>/dev/null &

If you want, you can also restore this process to the foreground and wait for it to finish by typing fg (ForeGround). To put it into the background again, type the sequence C-z, bg.

You can start several jobs in this way: each command will then be given a job number. The shell command jobs lists all the jobs associated with the current shell. The job preceded by a + sign indicates the last process begun as a background task. To restore a particular job to the foreground, you can then type fg <n> where <n> is the job number, e.g. fg 5.

Note that you can also suspend or start full-screen applications (if they are properly programmed) in this way, such as less or a text editor like VI, and restore them to the foreground when you want.

A Final Word

As you can see, the shell is very complete and using it effectively is a matter of practice. In this relatively long chapter, we have only mentioned a few of the available commands: Linux-Mandrake has thousands of utilities, and even the most experienced users employ a hundred at most.

There are utilities for all tastes and purposes: you have utilities for handling images (like convert mentioned above, but also GIMP batch mode and all pixmap handling utilities), sounds (MP3 encoders, audio CD players), for CD writing, e-mail programs, FTP clients and even web browsers (lynx or w3m), not to mention all the administration tools.

Even if graphical applications with equivalent functions do exist, they are usually graphical interfaces built around these very same utilities; in addition, command line utilities have the advantage of being able to operate in non-interactive mode: you can start writing a CD and then log off the system in the confidence that the writing will take place (see the nohup(1) manual page).