2. The material below is intended for the in-class instruction. Others trying to learn this material offline can follow along on any Unix (like) machine, however, please be aware the file names and path names will be different on your machine.
Requirements
The module requires that you use a recent MacOS or Linux based system. Future versions of the module may accommodate Microsoft Windows users.
Using the UNIX shell
Your time in the UNIX environment will be spent in the shell, which runs in the Terminal window, if you're using Mac OS. The shell allows you to move and copy files, run programs, and more.
How do I open my terminal window?
In Mac OS, you can open a new terminal window by opening the Finder, selecting the Applications folder, then the Utilities sub-folder, and then double-clicking on the Terminal application. You may want to drag the Terminal icon to your Application Dock for ready access going forward.
Alternatively, you can just use Spotlight to search for "Terminal" each time, and open the application after it is listed in the Spotlight results list.
In Linux environments, your terminal program will depend on the version of Linux you have installed. If you are using Linux and you don't know how to open a terminal, raise your hand and let us know now.
Accessing your temporary CGRL account
The next step is to login to the CGRL system (poset.cgrl.berkeley.edu) with your user account and password. You can do this with the ssh command.
ls (list) When you first login, your current working directory is your home directory.
To find out what is in your home directory, type
[workshop@poset ~]$ ls
file1.txt
The ls command lists the contents of your current working directory. ls does not, in fact, cause all the files in your home directory to be listed, but only those ones whose name does not begin with a dot (.). Files beginning with a dot (.) are known as hidden files and usually contain important program configuration information. They are hidden because you should not change them unless you are very familiar with UNIX!!! To list all files in your home directory including those whose names begin with a dot, type
[workshop@poset ~]$ ls -a
. .. .bash_logout .bash_profile .bashrc .emacs file1.txt
ls is an example of a command which can take options: -a is an example of an option. The options change the behavior of the command. There are online manual pages that tell you which options a particular command can take, and how each option modifies the behavior of the command. (See later in this tutorial)
1.2 Making Directories mkdir (make directory)
We will now make a subdirectory in your home directory to hold the files you will be creating and using in the course of this tutorial. To make a subdirectory called unixstuff in your current working directory type
[workshop@poset ~]$ mkdir unixstuff
To see the directory you have just created, type
[workshop@poset ~]$ ls
file1.txt unixstuff
1.3 Changing to a different directory cd (change directory) The command cd directory means change the current working directory to 'directory'. The current working directory may be thought of as the directory you are in, i.e. your current position in the file-system tree.
To change to the directory you have just made, type
[workshop@poset ~]$ cd unixstuff
Type ls to see the contents (which should be empty)
[workshop@poset unixstuff]$ls
Now create a new directory backups under unixstuff
[workshop@poset unixstuff]$ mkdir backups
1.4 The directories . and .. Still in the unixstuff directory, type
[workshop@poset unixstuff]$ls -a
. .. backups
As you can see, in the unixstuff directory (and in all other directories), there are two special directories called (.) and (..) In UNIX, (.) means the current directory, so type
[workshop@poset unixstuff]$ cd .
means stay where you are (the unixstuff directory). This may not seem very useful at first, but using (.) as the name of the current directory will save a lot of typing, as we shall see later in the tutorial.
(..) means the parent of the current directory, so typing
[workshop@poset unixstuff]$ cd ..
will take you one directory up the hierarchy (back to your home directory). Note: typing cd with no argument always returns you to your home directory. This is very useful if you are lost in the file system. 1.5 Pathnames pwd (print working directory)
Pathnames enable you to work out where you are in relation to the whole file-system. For example, to find out the absolute pathname of your home directory, type cd to get back to your home directory
[workshop@poset unixstuff]$ cd ..
and then type
[workshop@poset ~]$ pwd
/global/home/workshop
The full pathname to the home directory will look something like this - /global/home/username
1.6 More about home directories and pathnames Understanding pathnames
First type cd to get back to your home directory
[workshop@poset unixstuff]$ cd
Then type ls to list the contents of your unixstuff directory
[workshop@poset ~]$ ls unixstuff
backups
Now type
[workshop@poset ~]$ ls backups
ls: backups: No such file or directory
The reason is, backups is not in your current working directory. To use a command on a file (or directory) not in the current working directory (the directory you are currently in), you must either cd to the correct directory, or specify its full pathname. To list the contents of your backups directory, you must type
[workshop@poset ~]$ ls unixstuff/backups
~ (your home directory) Home directories can also be referred to by the tilde ~ character. It can be used to specify paths starting at your home directory. So typing
[workshop@poset ~]$ ls ~/unixstuff
backups
will list the contents of your unixstuff directory, no matter where you currently are in the file system.
2.1 Copying Files cp (copy) cp file1 file2 is the command which makes a copy of file1 in the current working directory and calls it file2
First, cd to your home directory and type ls.
[workshop@poset ~]$ cd
[workshop@poset ~]$ ls
file1.txt unixstuff
now make a copy of file1.txt and call the copy file2.txt
[workshop@poset ~]$ cp file1.txt file2.txt
now list all the contents in your home directory
[workshop@poset ~]$ ls
file1.txt file2.txt unixstuff
Now try something more fancier: make a copy of file1.txt to the directory unixstuff and name this copy file3.txt
Now make a copy of file1.txt to the current directory without renaming it
[workshop@poset unixstuff]$ cp ../file1.txt .
Don't forget the dot (.) at the end. Remember, in UNIX, the dot means the current directory. 2.2 Moving files mv (move)
mv file1 file2 moves (or renames) file1 to file2 To move a file from one place to another, use the mv command. This has the effect of moving rather than copying the file, so you end up with only one file rather than two. It can also be used to rename a file, by moving the file to the same directory, but giving it a different name. We are now going to move file1.txt to your backup directory. This can be done in many different ways. Here I just list one example: cd to your home directory
[workshop@poset unixstuff]$ cd
display all the content in home directory
[workshop@poset ~]$ ls
file1.txt file2.txt unixstuff
To delete (remove) a file, use the rm command. Now cd to your unixstuff directory and list all the content, type
[workshop@poset ~]$cd unixstuff
[workshop@poset unixstuff]$ ls
backups file3.txt file5.txt
No remove (permanently!) file3.txt and file5.txt from unixstuff, and then list the content
[workshop@poset ~]$ rm file3.txt file5.txt
[workshop@poset unixstuff]$ ls
backups
Now make a new directory test under unixstuff and use rmdir to remove it. Make sure to check the content of unixstuff before and after running rmdir
[workshop@poset unixstuff]$ mkdir test
[workshop@poset unixstuff]$ ls
backups test
[workshop@poset unixstuff]$ rmdir test
[workshop@poset unixstuff]$ ls
backups
2.4 Displaying the contents of a file on the screen clear (clear screen) Before you start the next section, you may like to clear the terminal window of the previous commands so the output of the following commands can be clearly understood.
Now cd to your home directory
[workshop@poset unixstuff]$ cd
Type clear in the shell
[workshop@poset ~]$ clear
This will clear all text and leave you with the $ prompt at the top of the window.
Now list the content by using ls
[workshop@poset ~]$ ls
file2.txt unixstuff
file2.txt is a copy of file1.txt, but we have moved file1.txt to the directory backups. Now we can move file1.txt back. Then you can type ls to check the content
As you can see, the file is longer than the size of the window, so it scrolls past making it unreadable.
less The command less writes the contents of a file onto the screen a page at a time. Type
[workshop@poset ~]$ less file1.txt
Press the [space-bar] if you want to see another page, type [q] if you want to quit reading. As you can see, less is used in preference to cat for long files.
head The head command writes the first ten lines of a file to the screen.
[workshop@poset ~]$ head -5 file1.txt
>Transcript1
CAGGCTTAGCG...
>Transcript2
CAGAGCCTCCA…
>Transcript3
The above shows the first 5 lines in file1.txt
tail The tail command writes the last ten lines of a file to the screen. Its function is opposite to head. Please give it a try. In the shell, type clear, then tail file1.txt, then tail -5 file1.txt 2.5 Searching the contents of a file grep
grep is one of many standard UNIX utilities. It searches files for specified words or patterns. First clear the screen, then type
Other common and useful options for grep include: -v display those lines that do NOT match -n precede each matching line with the line number -c print only the total count of matched lines
Try some of them and see the different results. Don't forget, you can use more than one option at a time, for example, the number of lines without the words “Transcript” or “transcript” is
wc (word count) A handy little utility is the wc command, short for word count. To find out how many lines the file has, type
[workshop@poset ~]$ wc -l file1.txt
22
3.1 Redirecting the Output Most processes initiated by UNIX commands write to the standard output (that is, they write to the terminal screen), and many take their input from the standard input (that is, they read it from the keyboard). There is also the standard error, where processes write their error messages, by default, to the terminal screen
echo (writes its arguments to standard output)
In a terminal window type
[workshop@poset ~]$ echo hello!
hello!
We now direct the output hello! to a new file called test, check the content of test using cat
[workshop@poset ~]$ echo hello! > test
[workshop@poset ~]$ cat test
hello!
We use the > symbol to redirect the output of a command.
For example, to create a file called list1 containing a list of fruit, type
[workshop@poset ~]$ cat > list1
Then type in the names of some fruit. Press [return]after each one
pear
banana
apple
^d (Control + d to stop)
What happens is the cat command reads the standard input (what you typed via the keyboard) and the > redirects the output into a file called list1
To read the contents of the file, type cat file1 or less file1
[workshop@poset ~]$ cat list1
pear
banana
apple
The form >> appends standard output to a file. So to add more items to the file list1, type
[workshop@poset ~]$ cat >> list1
Then type in the names of more fruit
peach
grape
orange
^d (Control + d to stop)
To read the contents of the file, type
[workshop@poset ~]$ cat list1
pear
banana
apple
peach
grape
orange
Now we can write some vegetables to a new file called list2, then type in the names of some vegetables
[workshop@poset ~]$ cat > list2
lettuce
carrot
kale
^d (Control + d to stop)
You should now have two files. One contains six fruit, and the other contains three vegetables. We will now use the cat command to join (concatenate) list1 and list2 into a new file called biglist. Type
[workshop@poset ~]$ cat list1 list2 > biglist
What this is doing is reading the contents of list1 and list2 in turn, then outputing the text to the file biglist
The command sort alphabetically or numerically sorts a list. Type
[workshop@poset ~]$ sort biglist
apple
banana
carrot
grape
kale
lettuce
orange
peach
pear
To output the sorted list to a file called sortlist, type the following
[workshop@poset ~]$ sort biglist > sortlist
3.2 Pipes I (pipe) take the output of one command and use it as input for the command after the pipe. Useful for chaining commands which have one output and accept one input
An example: we want to sort the file biglist alphabetically, then display the first 5 lines in the terminal window, then from these five lines we want to grep any lines containing letter “r” or “R” and then count how many lines meet this category
4.1 Wildcards The characters * and ? The character * is called a wildcard, and will match against none or more character(s) in a file (or directory) name. For example, in your home directory, type
[workshop@poset ~]$ ls list*
list1 list2
This will list all files and subdirectories in the current directory starting with list (list…)
Try typing
[workshop@poset ~]$ ls *list
biglist sortlist
This will list all files and subdirectories in the current directory ending with list (…list)
Now typing
[workshop@poset ~]$ ls *list*
biglist list1 list2 sortlist
This will list all files and subdirectories in the current directory containing list (…list…)
The character ? will match exactly one character. So ls ?ouse will match files like house and mouse, but not grouse.
Try typing
[workshop@poset ~]$ ls ?list
ls: ?list: No such file or directory
Now try typing
[workshop@poset ~]$ ls ?ist1
list1
4.2 Filename conventions We should note here that a directory is merely a special type of file. So the rules and conventions for naming files apply also to directories. In naming files, characters with special meanings such as / * & % , should be avoided. Also, avoid using spaces within names. The safest way to name a file is to use only alphanumeric characters, that is, letters and numbers, together with _ (underscore) and . (dot). File names conventionally start with a lower-case letter, and may end with a dot followed by a group of letters indicating the contents of the file. For example, all files consisting of C code may be named with the ending .c, for example, prog1.c . Then in order to list all files containing C code in your home directory, you need only type ls *.c in that directory. 4.3 Getting Help man (on-line manuals) and whatis
There are on-line manuals which gives information about most commands. The manual pages tell you which options a particular command can take, and how each option modifies the behavior of the command. Type man command to read the manual page for a particular command. For example, to find out more about the wc (word count) command, type
[workshop@poset ~]$ man wc
Alternatively, type
[workshop@poset ~]$ whatis wc
whatis gives a one-line description of the command, but omits any information about options etc.
5.1 Other commonly used UNIX commands df
The df command reports on the space left on the file system. For example, to find out how much space is left on the fileserver, type
[workshop@poset ~]$ df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/sdb 15T 9.0T 5.6T 62% /global
-h means that it prints sizes in human readable format (e.g., 1K 234M 2G)
du
The du command outputs the size of a file or a directory in kilobytes. Below is an example for output size of a file
[workshop@poset ~]$ du -h file1.txt
12K
gzip
This command compresses a file. For example, to compress file1.txt, type
[workshop@poset ~]$ gzip file1.txt
Now list the content in home directory
[workshop@poset ~]$ ls
biglist file1.txt.gz list1 list2 sortlist test unixstuff
Now file1.txt is zipped and is ended with .gz
To unzip the file, use the gunzip command
[workshop@poset ~]$ gunzip file1.txt.gz
Now list the content in home directory
[workshop@poset ~]$ ls
biglist file1.txt list1 list2 sortlist test unixstuff
history
The shell keeps an ordered list of all the commands that you have entered. Each command is given a number according to the order it was entered.
[workshop@poset ~]$ history
…lots of commands…
Now we only want to repeat the last command
[workshop@poset ~]$ !!
Now we only want to repeat the second command in the history
[workshop@poset ~]$ !2
nano (a small and friendly text editor)
Create an empty file sequence.fasta
[workshop@poset ~]$ nano sequence.fasta
and copy the following fasta sequences to this file
uniq (when fed a text file, outputs the file with adjacent identical lines collapsed to one)
Firstcreates a new file uniq_test.txt using nano and copy the following content to this file gene1 A gene1 A gene6 F gene6 F gene6 F gene1 A gene2 B gene2 B gene4 D gene5 E gene6 F gene6 F gene6 F gene1 A gene1 A gene2 B gene2 B gene2 B gene3 C gene4 D gene4 D
Now we want to remove duplicated lines from the file
[workshop@poset ~]$ sort uniq_test.txt | uniq
gene1 A
gene2 B
gene3 C
gene4 D
gene5 E
gene6 F
Print only those lines which are not repeated (unique) in the input.
[workshop@poset ~]$ sort uniq_test.txt | uniq -u
gene3 C
gene5 E
Print only those lines which are repeated in the input.
[workshop@poset ~]$ sort uniq_test.txt | uniq -d
gene1 A
gene2 B
gene4 D
gene6 F
You can also get some statistics out of uniq with the -c option
[workshop@poset ~]$ sort uniq_test.txt | uniq -dc
5 gene1 A
5 gene2 B
3 gene4 D
6 gene6 F
{} (Brace expansion is a mechanism by which arbitrary strings may be generated) {A,B,C} expands all the elements within the braces {1..9} expands a range of elements within the braces
[workshop@poset ~]$ echo hello {A,B,C}
hello A B C
[workshop@poset ~]$ echo hello{A,B,C}
helloA helloB helloC
Create several folders with names starting with “folder” under the home directory
The above command grep each line starting with “C”, and ending with “T”, with any letters in between them, and then print out the line number of that line. . means any single character (except newline), * means that the preceding item will be matched zero or more times.
[workshop@poset ~]$ grep -A 1 ">Transcript10" file1.txt
-A N[int] means that it will print N lines of trailing context after matching lines.
-B N[int] means that it will print N lines of leading context before matching lines.
The above two command lines are useful for displaying sequences of a fasta file.
[workshop@poset ~]$ grep -A 1 ">Trascript1" file1.txt
What happened? It matches Trascript1, Trascript10 and Trascript11 since all there contain “Trascript1”. We just want the sequence after Transcript1 to be printed. We need to add more conditions. Several ways to do this:
[workshop@poset ~]$ grep -A 1 ">Transcript1$" file1.txt
[workshop@poset ~]$ grep -A 1 ">Transcript1\b" file1.txt
\b matches the empty string at the edge of a word. It sets a boundary to the matches.
sed (a stream editor) A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline) Here we use sed to find and replace complex string in a file. For example we want to replace seq1 with sequence 1 in the file sequence.fasta
[workshop@poset ~]$ sed 's/seq1/gene1/g' sequence.fasta
>gene1
AGTACGTAGTAGCTGCTGCTACGTGCGCTAGCTAGTACGTCAA
TCGTACGTCGACTGATCGTAGCTACGTCGTACGTAGGTACGTT
…
In the above example, s/A/B/g means that it will replace A with B in global search.
[workshop@poset ~]$ grep -A 1 ">Transcript1$" file1.txt | sed 's/A/a/g'
>Transcript1
CaGGCTTaGCGGTTTaTTGaTGTCTCCCaaGCCaGaGGaaGTCaCCaaaGGGaaGaGTCaCTTGCaGaaaaTGGTTTCCaTTTCaGaCaTTaCaTCCCTGTaCaGCaaCCTGCTCCTGTaCaCGTGTGGGTCCTCTGCCGaaGCCaCCaGGGCTGTCaTGaGaCaCCTTGCCGCTGTGTaTCaaCaTGGCaGCCTGCTaGGGCTTTCTGTTGCCaaGaGGCCTCTCTGGaGaCaGGCaTCTaTGCaaaGTGGGaaGGaCaCCaCTGaGCaaGaaaTTCTGaaaGCTaTCaaCaTCaaTTCCTTTGCaGaGTGTGGCaTCaaTTTaTTCCaTGaGaGTGTaTCTaaaTCaGCCCTGaGCCaaGaaTTCGaaGCTTTCTTTCGT
In the above example, we first find the sequence of Transcript1, transform A to a, and then print the output in the terminal.
Your instructor is Ke Bi.
MPORTANT NOTES
1.This workshop material is based on “UNIX Tutorial for Beginners” (http://www.ee.surrey.ac.uk/Teaching/Unix/) with modifications. “UNIX Tutorial for Beginners” is covered by a Creative Commons licence. If you need to download a copy of the original tutorial please go to http://www.ee.surrey.ac.uk/Teaching/Unix/download.html
2. The material below is intended for the in-class instruction. Others trying to learn this material offline can follow along on any Unix (like) machine, however, please be aware the file names and path names will be different on your machine.
Requirements
The module requires that you use a recent MacOS or Linux based system. Future versions of the module may accommodate Microsoft Windows users.
Using the UNIX shell
Your time in the UNIX environment will be spent in the shell, which runs in the Terminal window, if you're using Mac OS. The shell allows you to move and copy files, run programs, and more.
How do I open my terminal window?
In Mac OS, you can open a new terminal window by opening the Finder, selecting the Applications folder, then the Utilities sub-folder, and then double-clicking on the Terminal application. You may want to drag the Terminal icon to your Application Dock for ready access going forward.
Alternatively, you can just use Spotlight to search for "Terminal" each time, and open the application after it is listed in the Spotlight results list.
In Linux environments, your terminal program will depend on the version of Linux you have installed. If you are using Linux and you don't know how to open a terminal, raise your hand and let us know now.
Accessing your temporary CGRL account
The next step is to login to the CGRL system (poset.cgrl.berkeley.edu) with your user account and password. You can do this with the ssh command.
1.1 Listing files and directories
ls (list)
When you first login, your current working directory is your home directory.
To find out what is in your home directory, type
The ls command lists the contents of your current working directory. ls does not, in fact, cause all the files in your home directory to be listed, but only those ones whose name does not begin with a dot (.). Files beginning with a dot (.) are known as hidden files and usually contain important program configuration information. They are hidden because you should not change them unless you are very familiar with UNIX!!!
To list all files in your home directory including those whose names begin with a dot, type
ls is an example of a command which can take options: -a is an example of an option. The options change the behavior of the command. There are online manual pages that tell you which options a particular command can take, and how each option modifies the behavior of the command. (See later in this tutorial)
1.2 Making Directories
mkdir (make directory)
We will now make a subdirectory in your home directory to hold the files you will be creating and using in the course of this tutorial. To make a subdirectory called unixstuff in your current working directory type
To see the directory you have just created, type
1.3 Changing to a different directory
cd (change directory)
The command cd directory means change the current working directory to 'directory'. The current working directory may be thought of as the directory you are in, i.e. your current position in the file-system tree.
To change to the directory you have just made, type
Type ls to see the contents (which should be empty)
Now create a new directory backups under unixstuff
1.4 The directories . and ..
Still in the unixstuff directory, type
As you can see, in the unixstuff directory (and in all other directories), there are two special directories called (.) and (..)
In UNIX, (.) means the current directory, so type
means stay where you are (the unixstuff directory). This may not seem very useful at first, but using (.) as the name of the current directory will save a lot of typing, as we shall see later in the tutorial.
(..) means the parent of the current directory, so typing
will take you one directory up the hierarchy (back to your home directory).
Note: typing cd with no argument always returns you to your home directory. This is very useful if you are lost in the file system.
1.5 Pathnames
pwd (print working directory)
Pathnames enable you to work out where you are in relation to the whole file-system. For example, to find out the absolute pathname of your home directory, type cd to get back to your home directory
and then type
The full pathname to the home directory will look something like this -
/global/home/username
1.6 More about home directories and pathnames
Understanding pathnames
First type cd to get back to your home directory
Then type ls to list the contents of your unixstuff directory
Now type
The reason is, backups is not in your current working directory. To use a command on a file (or directory) not in the current working directory (the directory you are currently in), you must either cd to the correct directory, or specify its full pathname. To list the contents of your backups directory, you must type
~ (your home directory)
Home directories can also be referred to by the tilde ~ character. It can be used to specify paths starting at your home directory. So typing
will list the contents of your unixstuff directory, no matter where you currently are in the file system.
2.1 Copying Files
cp (copy)
cp file1 file2 is the command which makes a copy of file1 in the current working directory and calls it file2
First, cd to your home directory and type ls.
now make a copy of file1.txt and call the copy file2.txt
now list all the contents in your home directory
Now try something more fancier: make a copy of file1.txt to the directory unixstuff and name this copy file3.txt
Check the content in unixstuff
Now cd to unixstuff
Now make a copy of file1.txt to the current directory and name it file5.txt
Now make a copy of file1.txt to the current directory without renaming it
Don't forget the dot (.) at the end. Remember, in UNIX, the dot means the current directory.
2.2 Moving files
mv (move)
mv file1 file2 moves (or renames) file1 to file2
To move a file from one place to another, use the mv command. This has the effect of moving rather than copying the file, so you end up with only one file rather than two. It can also be used to rename a file, by moving the file to the same directory, but giving it a different name.
We are now going to move file1.txt to your backup directory. This can be done in many different ways. Here I just list one example:
cd to your home directory
display all the content in home directory
now move file1.txt to the directory backups
Now check the content in the home directory again by using ls
You can see file1.txt is gone. Now check the content in the directory backups
2.3 Removing files and directories
rm (remove), rmdir (remove empty directory)
To delete (remove) a file, use the rm command.
Now cd to your unixstuff directory and list all the content, type
No remove (permanently!) file3.txt and file5.txt from unixstuff, and then list the content
Now make a new directory test under unixstuff and use rmdir to remove it. Make sure to check the content of unixstuff before and after running rmdir
2.4 Displaying the contents of a file on the screen
clear (clear screen)
Before you start the next section, you may like to clear the terminal window of the previous commands so the output of the following commands can be clearly understood.
Now cd to your home directory
Type clear in the shell
This will clear all text and leave you with the $ prompt at the top of the window.
Now list the content by using ls
file2.txt is a copy of file1.txt, but we have moved file1.txt to the directory backups. Now we can move file1.txt back. Then you can type ls to check the content
cat
The command cat can be used to display the contents of a file on the screen. Type:
As you can see, the file is longer than the size of the window, so it scrolls past making it unreadable.
less
The command less writes the contents of a file onto the screen a page at a time. Type
Press the [space-bar] if you want to see another page, type [q] if you want to quit reading. As you can see, less is used in preference to cat for long files.
head
The head command writes the first ten lines of a file to the screen.
First clear the screen then type head file1.txt
Then type
The above shows the first 5 lines in file1.txt
tail
The tail command writes the last ten lines of a file to the screen. Its function is opposite to head. Please give it a try. In the shell, type clear, then tail file1.txt, then tail -5 file1.txt
2.5 Searching the contents of a file
grep
grep is one of many standard UNIX utilities. It searches files for specified words or patterns. First clear the screen, then type
As you can see, grep has printed out each line containing the word “Transcript”.
Now try typing
The grep command is case sensitive; it distinguishes between “Transcript” and “transcript”.
To ignore upper/lower case distinctions, use the -i option, i.e. type
Other common and useful options for grep include:
-v display those lines that do NOT match
-n precede each matching line with the line number
-c print only the total count of matched lines
Try some of them and see the different results. Don't forget, you can use more than one option at a time, for example, the number of lines without the words “Transcript” or “transcript” is
wc (word count)
A handy little utility is the wc command, short for word count. To find out how many lines the file has, type
3.1 Redirecting the Output
Most processes initiated by UNIX commands write to the standard output (that is, they write to the terminal screen), and many take their input from the standard input (that is, they read it from the keyboard). There is also the standard error, where processes write their error messages, by default, to the terminal screen
echo (writes its arguments to standard output)
In a terminal window type
We now direct the output hello! to a new file called test, check the content of test using cat
We use the > symbol to redirect the output of a command.
For example, to create a file called list1 containing a list of fruit, type
Then type in the names of some fruit. Press [return] after each one
^d (Control + d to stop)
What happens is the cat command reads the standard input (what you typed via the keyboard) and the > redirects the output into a file called list1
To read the contents of the file, type cat file1 or less file1
The form >> appends standard output to a file. So to add more items to the file list1, type
Then type in the names of more fruit
^d (Control + d to stop)
To read the contents of the file, type
Now we can write some vegetables to a new file called list2, then type in the names of some vegetables
^d (Control + d to stop)
You should now have two files. One contains six fruit, and the other contains three vegetables. We will now use the cat command to join (concatenate) list1 and list2 into a new file called biglist. Type
What this is doing is reading the contents of list1 and list2 in turn, then outputing the text to the file biglist
To read the contents of the new file, type
The command sort alphabetically or numerically sorts a list. Type
To output the sorted list to a file called sortlist, type the following
3.2 Pipes
I (pipe) take the output of one command and use it as input for the command after the pipe. Useful for chaining commands which have one output and accept one input
An example: we want to sort the file biglist alphabetically, then display the first 5 lines in the terminal window, then from these five lines we want to grep any lines containing letter “r” or “R” and then count how many lines meet this category
4.1 Wildcards
The characters * and ?
The character * is called a wildcard, and will match against none or more character(s) in a file (or directory) name. For example, in your home directory, type
This will list all files and subdirectories in the current directory starting with list (list…)
Try typing
This will list all files and subdirectories in the current directory ending with list (…list)
Now typing
This will list all files and subdirectories in the current directory containing list (…list…)
The character ? will match exactly one character. So ls ?ouse will match files like house and mouse, but not grouse.
Try typing
Now try typing
4.2 Filename conventions
We should note here that a directory is merely a special type of file. So the rules and conventions for naming files apply also to directories.
In naming files, characters with special meanings such as / * & % , should be avoided. Also, avoid using spaces within names. The safest way to name a file is to use only alphanumeric characters, that is, letters and numbers, together with _ (underscore) and . (dot).
File names conventionally start with a lower-case letter, and may end with a dot followed by a group of letters indicating the contents of the file. For example, all files consisting of C code may be named with the ending .c, for example, prog1.c . Then in order to list all files containing C code in your home directory, you need only type ls *.c in that directory.
4.3 Getting Help
man (on-line manuals) and whatis
There are on-line manuals which gives information about most commands. The manual pages tell you which options a particular command can take, and how each option modifies the behavior of the command. Type man command to read the manual page for a particular command.
For example, to find out more about the wc (word count) command, type
Alternatively, type
whatis gives a one-line description of the command, but omits any information about options etc.
5.1 Other commonly used UNIX commands
df
The df command reports on the space left on the file system. For example, to find out how much space is left on the fileserver, type
-h means that it prints sizes in human readable format (e.g., 1K 234M 2G)
du
The du command outputs the size of a file or a directory in kilobytes. Below is an example for output size of a file
gzipThis command compresses a file. For example, to compress file1.txt, type
Now list the content in home directory
Now file1.txt is zipped and is ended with .gz
To unzip the file, use the gunzip command
Now list the content in home directory
historyThe shell keeps an ordered list of all the commands that you have entered. Each command is given a number according to the order it was entered.
Now we only want to repeat the last command
Now we only want to repeat the second command in the history
nano (a small and friendly text editor)
Create an empty file sequence.fasta
and copy the following fasta sequences to this file
>seq1
AGTACGTAGTAGCTGCTGCTACGTGCGCTAGCTAGTACGTCAA
TCGTACGTCGACTGATCGTAGCTACGTCGTACGTAGGTACGTT
CGACGTAGATGCTAGCTGACTCGATGC
>seq2
CGATCGATCGTACGTCGACTGATCGTAGCTACGTCGTACGTAG
GTACGTGATCGTACGTCGACTGATCGTAGCTATCGTAGCTACC
CATCGTCAGTTACTGCATGCTCG
>seq3
GATCGTACGTGTAGCTGCTGTAGCTGATCGTACGGTAGCAGCT
CGTCGACTGATCGTAGCTGATCGTACGTCGACTGATCGTAGCT
AGTACGTAGTAGCTGTAGCTGATCGTACGTAGCTGATCGTACG
>seq4
CGATCGATCGTACGTCGACTGATCGTAGCTACGTCGTACGTAG
GTACGTGATCGTACGTCGACTGATCGTAGCTATCGTAGCTACC
GTAGATGCTAGCTGACTCGAT
Type ctrl+X, Y, and then return to quit.
Now peek inside this file by using cat or less
wget (retrieve files from web servers)
cut (extract sections from each line of input)
Read the contents of gene_exp.diff, type
-S causes lines longer than the screen width to be chopped rather than folded
Extract the third column from gene_exp.diff and display it using less
Extract the third column from gene_exp.diff, exclude lines containing “-“, sort the names alphabetically and then display it using less
uniq (when fed a text file, outputs the file with adjacent identical lines collapsed to one)
Firstcreates a new file uniq_test.txt using nano and copy the following content to this file
gene1 A
gene1 A
gene6 F
gene6 F
gene6 F
gene1 A
gene2 B
gene2 B
gene4 D
gene5 E
gene6 F
gene6 F
gene6 F
gene1 A
gene1 A
gene2 B
gene2 B
gene2 B
gene3 C
gene4 D
gene4 D
Now we want to remove duplicated lines from the file
Print only those lines which are not repeated (unique) in the input.
Print only those lines which are repeated in the input.
You can also get some statistics out of uniq with the -c option
{} (Brace expansion is a mechanism by which arbitrary strings may be generated)
{A,B,C} expands all the elements within the braces
{1..9} expands a range of elements within the braces
[workshop@poset ~]$ echo hello {A,B,C} hello A B C [workshop@poset ~]$ echo hello{A,B,C} helloA helloB helloCCreate several folders with names starting with “folder” under the home directory[workshop@poset ~]$ mkdir folder_{A,B,C} [workshop@poset ~]$ ls folder* folder_A: folder_B: folder_C:Create new several empty files with names beginning with “txt” using touch under the home directory[workshop@poset ~]$ touch txt{1..5}.txt [workshop@poset ~]$ ls txt* txt1.txt txt2.txt txt3.txt txt4.txt txt5.txtfind
find is useful for finding file or directory names that match certain criteria.
Search the home directory and print any files or directories beginning with list
Search the home directory to print all subdirectories
More about grep
The above command grep each line ending with a “T” from sequence.fasta and print out the line number of that line. $ matches the end of a line
The above command grep each line starting with a “C” and print out the line number of that line. ^ matches the beginning of a line
The above command grep each line starting with “C”, and ending with “T”, with any letters in between them, and then print out the line number of that line. . means any single character (except newline), * means that the preceding item will be matched zero or more times.
-A N[int] means that it will print N lines of trailing context after matching lines.
-B N[int] means that it will print N lines of leading context before matching lines.
The above two command lines are useful for displaying sequences of a fasta file.
What happened? It matches Trascript1, Trascript10 and Trascript11 since all there contain “Trascript1”. We just want the sequence after Transcript1 to be printed. We need to add more conditions. Several ways to do this:
\b matches the empty string at the edge of a word. It sets a boundary to the matches.
sed (a stream editor)
A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline)
Here we use sed to find and replace complex string in a file. For example we want to replace seq1 with sequence 1 in the file sequence.fasta
In the above example, s/A/B/g means that it will replace A with B in global search.
In the above example, we first find the sequence of Transcript1, transform A to a, and then print the output in the terminal.