Mastering the Command-Line Interface (CLI)

Table of Contents

• Module 1: Creating, Deleting, and Moving Files
  • Creating and Updating Files and Directories
  • Copying and Moving Files and Directories
  • Deleting Files and Directories
  • Linking Files: Hard Links and Symbolic Links
• Module 2: Filtering Data Streams
  • Controlling Output with Wildcards and Globbing
  • Filtering Text with grep
  • Searching the File System with locate and find
  • Refining Searches by Object Attributes
• Module 3: Editing and Manipulating Data Streams
  • Ordering Data with sort
  • Parsing Structured Data with sort, head, and tail
  • Extracting Fields with cut
  • Quantifying Data with wc
• Module 4: Redirecting Data Streams
  • Appending and Overwriting Output
  • Understanding Standard Input, Output, and Error
  • Chaining Commands and Building Robust Scripts
• Summary

The command line shell is where nearly everything on a Linux system happens, and nearly everything that happens at the command line involves manipulating text in one way or another. This course focuses on using command-line tools to work with plain-text data — whether that data lives in files or is moving between processes and destinations. All demonstrations run against a Linux system operating inside a container, which behaves exactly like a physical machine for the purposes of these commands.

mindmap
  root((Mastering the CLI))
    Files and Directories
      Create / Update
      Copy / Move
      Delete
      Hard and Soft Links
    Filtering Data Streams
      Wildcards and Globbing
      grep
      locate and find
      Attribute-based Search
    Editing and Manipulating Streams
      sort
      cut
      wc
    Redirecting Data Streams
      Append / Overwrite
      stdin / stdout / stderr
      Chaining and Scripts

Module 1: Creating, Deleting, and Moving Files

Creating and Updating Files and Directories

Using ls to list the contents of an empty directory named project shows nothing, since there is nothing there to display yet. The -l flag would normally provide a long listing with ownership, permissions, size, and timestamp information, but with an empty directory there is nothing extra to show.

ls
ls -l

A new file can be created directly with the nano text editor. Passing a filename that does not already exist causes nano to create it in the current directory:

nano newfile

After typing some simple content, Ctrl+X saves the file and exits nano. Listing the directory again shows newfile present, owned by the current user and group, along with the date and time it was last updated.

The touch command provides a second way to create files. It does one of two things: it creates a brand-new, empty file if none exists, or it updates the modification timestamp of a file that already exists.

touch newerfile
ls -l

Listing the directory shows both files, but newfile is 7 bytes (it has real content typed into nano) while newerfile is 0 bytes (created empty by touch).

touch can also be used purely to refresh an existing file’s timestamp without changing its contents:

touch newfile
ls -l

Running this against the already-existing newfile updates its modification time to the current moment. This capability has practical uses:

• Protecting an important file from being swept up by automated deletion or archival processes that operate on file age.
• Using timestamps as manual markers for when a particular event occurred.

One thing touch cannot do is create directories. That is the job of mkdir:

mkdir dir1
ls

The new dir1 entry appears in a distinct color (typically blue) to indicate that it is a directory rather than a regular file.

So far, every object has been created in the current working directory because no other location was specified. The following sequence shows what happens when you try to be more specific about file system locations, and why several intuitive attempts to do so can fail.

pwd
# /home/ubuntu/project

mkdir etc/dir2
# mkdir: cannot create directory 'etc/dir2': No such file or directory

This attempt fails because there is no leading forward slash in front of etc. Without the slash, the shell assumes etc is meant to be a subdirectory relative to the current location, and no such subdirectory exists there.

mkdir /etc/dir2
# mkdir: cannot create directory '/etc/dir2': Permission denied

Adding the leading slash correctly points at the system-wide /etc directory, but this time the command fails because of permissions — an ordinary user account cannot create objects in a protected system directory without elevated (sudo) privileges. Only users who belong to the sudo group (or, on some systems, the wheel group) can invoke sudo successfully; the primary account created during Linux installation is normally granted this membership by default.

sudo mkdir /etc/dir2
ls /etc

With sudo, the directory is created successfully (most setups will also prompt for a password before executing the elevated command). Listing /etc confirms dir2 now appears near the top.

A further attempt to create a chain of nested subdirectories in one command illustrates another common pitfall:

sudo mkdir /etc/dir2/place1/place2
# mkdir: cannot create directory '/etc/dir2/place1/place2': No such file or directory

This fails because mkdir, by default, expects every parent directory in the path to already exist — it will not create place2 if place1 does not yet exist. The command could be split into two separate mkdir invocations, but there is a faster, single-command solution: the -p (path) flag, which instructs mkdir to create every directory necessary along the given path.

sudo mkdir -p /etc/dir2/place1/place2
tree /etc/dir2

The -p flag is frequently used inside software installation scripts to guarantee that configuration or logging directory structures exist before the software tries to use them. The tree command confirms the complete nested directory structure was created as expected.

flowchart TD
    A["mkdir etc/dir2"] -->|No leading slash, treated as relative path| B[Fails: No such file or directory]
    C["mkdir /etc/dir2"] -->|No sudo, protected system path| D[Fails: Permission denied]
    E["sudo mkdir /etc/dir2"] --> F[Succeeds]
    G["sudo mkdir /etc/dir2/place1/place2"] -->|Parent path1 does not exist yet| H[Fails: No such file or directory]
    I["sudo mkdir -p /etc/dir2/place1/place2"] -->|-p creates every missing parent| J[Succeeds: full nested path created]

Copying and Moving Files and Directories

The tree command provides a quick visual of everything living inside the project directory — an empty dir1 directory plus a couple of files (the host project directory itself is also counted, which is why the directory count appears one higher than expected).

tree

The cp command copies a file to a new location without removing the original:

cp newfile dir1/
tree

While typing the file name, only the first two or three characters need to be typed before pressing Tab to invoke tab-completion — the shell inspects the current context, finds an object whose name matches those characters, and fills in the rest automatically. Running tree again shows a new copy of newfile inside dir1, alongside the original still present in project.

The mv command works the same way syntactically, but it moves the object rather than copying it, leaving nothing behind at the source location:

mv newerfile dir1/
tree

tree confirms newerfile is now only present inside dir1, and it no longer appears in project.

Both tools work equally well when targeting specific file system locations elsewhere on disk:

sudo cp newfile /etc/dir2/
ls

The original copy of newfile remains in the local directory, confirming that cp (unlike mv) does not remove the source.

mv can also be used purely to rename a file or directory in place:

mv newfile newestfile
ls

The file that was previously newfile is now listed as newestfile.

There is one more important variant of the copy command worth knowing: scp (secure copy), the SSH-powered equivalent of cp used for transferring files between machines over a network. SSH itself is a long-established Linux tool that is now also available on macOS and Windows.

To identify the network address needed for the transfer, ip addr is used to inspect the local network devices:

ip addr

This reveals the machine’s IP address on the local network (for example, 10.0.3.227) — an address usable only from the local network, not from the open internet.

From a second terminal running on a separate workstation on the same network, a single local file named transfer is copied over to the container using scp:

scp transfer ubuntu@10.0.3.227:/home/ubuntu/project/dir1

The syntax specifies: the local file to send (transfer), the remote account identity to assume (ubuntu), the remote machine’s address (10.0.3.227), a colon, and finally the destination path on the remote system. Because authentication keys had previously been exchanged between the two machines, no password prompt appeared — otherwise scp would normally request the remote account’s password. Listing dir1 back on the container confirms transfer arrived at its expected destination.

sequenceDiagram
    participant WS as Local Workstation
    participant VM as Container / Remote Host (10.0.3.227)
    WS->>WS: ls (confirm 'transfer' file exists locally)
    WS->>VM: scp transfer ubuntu@10.0.3.227:/home/ubuntu/project/dir1
    VM-->>WS: Authenticated via pre-exchanged SSH keys (no password prompt)
    VM->>VM: ls dir1 (confirm 'transfer' arrived)

Deleting Files and Directories

Removing a file with rm is straightforward:

rm newestfile
ls

newestfile is now gone. Removing directories is handled by a different command, rmdir:

mkdir dir3
rmdir dir3
ls

dir3 no longer appears after the rmdir call. However, rmdir has an important limitation. To demonstrate it, a new directory with a file inside it is created:

mkdir dir4
touch dir4/stuff
tree

Attempting to remove dir4 with rmdir now fails:

rmdir dir4
# rmdir: failed to remove 'dir4': Directory not empty

rmdir is only designed to remove empty directories. This is a safety mechanism that helps prevent accidentally deleting files nested inside subdirectories that you may not realize are there.

It is worth emphasizing that rm and rmdir behave nothing like a visual desktop’s trash can. A trash can keeps deleted items recoverable until it is emptied. Running rm or rmdir on an object removes it from the system immediately, with no built-in mechanism to recover it afterward. Always be completely certain before running these commands.

To remove a non-empty directory and everything inside it, rm supports a recursive flag:

rm -r dir4
tree

The -r (recursive) flag tells rm to operate on every layer beneath the target, however deep the nesting goes. This is powerful, but also genuinely dangerous — with no built-in recovery, it becomes trivially easy to accidentally destroy critical resources. At the same time, without a tool like this, everyday administrative work would be far less efficient.

For an added layer of safety, rm supports an interactive confirmation flag:

touch throwaway.txt
rm -i throwaway.txt
# rm: remove regular file 'throwaway.txt'? y

With -i (interactive), rm prompts for confirmation (y to proceed) before deleting anything. It is a small extra step, but building the habit of adding -i can prevent serious future mistakes.

Combining recursion with the wildcard character shows just how far-reaching a single rm invocation can be:

tree
rm -r *
tree

Here, -r makes the operation recursive (applying to nested subdirectories too), while the asterisk (*) makes the target global, matching every object in the current location. Together, this combination destroys everything within reach of the current directory.

Extreme caution: adding just one more character to that command — a leading forward slash together with sudo — would target the very top of the root partition, deleting every single file and directory on the entire file system:

# DO NOT RUN — DESTROYS THE ENTIRE FILE SYSTEM
sudo rm -r /*

This command is shown here purely as a warning, not as something to ever actually type on a real system.

Deleting File System Objects: Safety Checklist

• Prefer rm -i for anything destructive and irreversible.
• Never combine rm -r with a wildcard unless you have verified exactly what will be matched.
• Treat any command involving sudo rm -r / as an absolute last resort that should never be run casually.

Linking Files: Hard Links and Symbolic Links

Moving a file to a new location is one way to improve accessibility, but Linux also supports linking — allowing a single file to effectively exist in two file system locations simultaneously.

Consider a scenario where an important application configuration file, myconfig.text, needs to live in /etc so that all users can access it:

sudo nano /etc/myconfig.text

A realistic-looking configuration is added and saved. Next, a hard link to this file is created, living in the local project directory. Because the source file in /etc is owned by root, sudo is required:

sudo ln /etc/myconfig.text ~/project/hardlink_myconfig.text
cat ~/project/hardlink_myconfig.text

The order of arguments to ln matters: the source file’s full path comes first, followed by the name and location of the new link. Printing the contents of the new hard link with cat shows content identical to the source file, even though it lives in a completely different directory.

To prove the two are truly connected, new content is appended to the source file in /etc, and the local hard link immediately reflects the change (the reverse — editing the local link and seeing the change in /etc — is equally true, since both names point to the very same underlying data):

sudo bash -c 'echo "timeout=30" >> /etc/myconfig.text'
cat ~/project/hardlink_myconfig.text

Examining the link’s metadata with ls -l shows the file is owned by root and by the root group, and — critically — a link count of 2, indicating that two directory entries point to the same underlying inode (the data structure that actually represents the file’s contents on disk):

ls -l ~/project/hardlink_myconfig.text

A symbolic link (symlink) can also be created to reference the same file. In many ways hard and symbolic links behave the same — either version can be read or edited, with changes affecting the other. The syntax is nearly identical, but adds the -s flag, and no sudo is required to create a symlink:

ln -s /etc/myconfig.text ~/project/config_sym.text
ls -l ~/project

Listing the directory shows both link files, but ownership differs: the hard link is owned by root, while the symlink is owned by the local (non-root) user who created it. The symlink’s own inode is only connected to a single file — it does not share an inode with the source the way a hard link does. In effect, the symlink file is not a “real” file at all, but a lightweight, virtual placeholder that points at the actual file elsewhere. cat on the symlink still prints content identical to the source.

The critical difference becomes clear once the source file is deleted:

sudo rm /etc/myconfig.text
cat ~/project/config_sym.text
# cat: config_sym.text: No such file or directory

cat ~/project/hardlink_myconfig.text
# ... content still intact ...

ls -l ~/project

The symlink is now broken (dangling) — it silently stops working even though nothing was done to it directly, because it was only ever a pointer to a path that no longer resolves. The hard link, on the other hand, survives the complete loss of the original source file, because it is a genuine second directory entry pointing at the same inode; that inode’s link count simply drops from 2 down to 1.

graph TD
    subgraph Hard Link
        Inode1[("inode #4821 (file contents)")]
        A["/etc/myconfig.text"] --> Inode1
        B["~/project/hardlink_myconfig.text"] --> Inode1
    end
    subgraph Symbolic Link
        Inode2[("inode #4822 (file contents)")]
        C["/etc/myconfig.text"] --> Inode2
        D["~/project/config_sym.text"] -.pointer/path reference.-> C
    end

Choosing between a hard link and a symlink depends entirely on the relationship the use case requires between the two file references.

Module 2: Filtering Data Streams

This module focuses on manipulating data to make it more useful. The biggest challenge with data today is that there is simply too much of it — being effective as an administrator requires filtering text down to exactly what matters.

Controlling Output with Wildcards and Globbing

Listing a directory shows 11 files and a single directory named old. In a real environment this could easily be hundreds of files, many of them irrelevant to the task at hand.

ls

Most of the file names use a .csv extension. CSV (comma-separated values) is a common formatting standard for spreadsheet-style data. To narrow the view down to just the CSV files:

ls *.csv

The asterisk (*) is a wildcard that matches any combination of characters, so this pattern matches any name ending in .csv. Almost every spreadsheet file is listed — except one: a backup file whose name technically contains .csv, but not at the very end (for example, report-01.csv.bak). Because the .csv substring is not the final part of the name, ls *.csv misses it. Adding a second wildcard after csv catches it too:

ls *.csv*

To narrow down further — say, only report files with version numbers below 10 (versions 1–3, excluding 10–12) — a more specific pattern targeting the string report-0 works:

ls *report-0*

Wildcards are not limited to viewing files; they can be woven directly into commands that modify objects. For example, moving only the higher-numbered report versions (10 and up) into the old directory:

mv *report-1* old/
ls
ls old

This pattern-based approach — often called globbing — is the key to controlling operations across large collections of objects without manually specifying every file name.

A second common scenario involves rotated log files in a typical /var/log-style directory. Active log files are shown in one color, while older, rotated versions — assigned incrementing version numbers and compressed with gzip — appear in another color. By default, Linux automatically deletes sufficiently old rotated logs to keep storage from being overwhelmed, but the naming pattern can still be exploited for filtering:

ls alternatives*.*.gz

Inserting an asterisk between the final two dots matches every gzip-compressed file from the alternatives logs, regardless of version number.

The question mark (?) wildcard is a useful alternative to the asterisk in cases where you need to match an exact number of characters — each ? represents exactly one character, whereas * represents any number of characters:

ls alternatives*.??.gz

This matches only files with exactly two characters between the two dots, excluding files with single-digit version numbers (1 through 9).

Command substitution is a related technique for inserting the output of one command directly into another. For example, copying a report file and appending the current date to the new file’s name:

ls
cp report-01.csv report-01-$(date +%F).csv
ls

The $( ... ) syntax tells the shell to execute the enclosed command (date +%F, which prints the current date in YYYY-MM-DD format) and substitute its output directly into the surrounding command line — in this case, into the new file’s name.

A second example uses which to locate the file system path of an installed binary, then feeds that result straight into ls:

which python3
# /usr/bin/python3

ls -l $(which python3)

The output reveals that python3 is really just a symlink pointing to the actual python3.12 binary in the same directory — a common technique for keeping application environments stable across version updates.

The older, equivalent syntax for command substitution uses backtick characters instead of $( ... ):

ls -l `which python3`

The results are identical, though backticks become progressively harder to read and nest for more complex use cases, which is why $( ... ) is the modern, preferred syntax.

flowchart LR
    A["Wildcard pattern"] --> B{"Character: * or ?"}
    B -->|"*"| C["Matches ANY number of characters<br/>(including zero)"]
    B -->|"?"| D["Matches EXACTLY ONE character"]
    C --> E["ls *.csv → matches report.csv, sales-data.csv, etc."]
    D --> F["ls report-0?.csv → matches report-01.csv, report-02.csv<br/>but NOT report-100.csv"]

Filtering Text with grep

grep is the single most versatile tool for filtering text streams on the Linux command line — it is rare to work at a terminal for long without reaching for it.

To put grep through its paces, a large data source is needed: the system log file, /var/log/syslog, which can easily contain thousands of lines. Suppose the goal is to find system events reported using the word “listening”:

grep Listening /var/log/syslog

By default, grep is case-sensitive, so the exact case of the search term matters. The results highlight every match of the search string, but there are still far too many lines to be immediately useful.

wc -l /var/log/syslog
# 4987 /var/log/syslog

grep Listening /var/log/syslog | wc -l
# 32

wc (word count) reports that the full log file contains nearly 5,000 lines, and the grep-filtered output is still 32 lines — too much to narrow down by eye. Further refinement requires chaining multiple filters. Rather than invoking grep as a standalone command, it can also be composed with cat:

cat /var/log/syslog | grep Listening

This produces identical output but with additional flexibility for chaining. A second grep in the pipeline can exclude lines that are not relevant:

cat /var/log/syslog | grep Listening | grep -v agent

The -v flag inverts the match, returning only lines that do not contain the given string — in this case, excluding the frequently occurring but irrelevant word “agent.” This narrows dozens of lines down to about a screen’s worth (roughly 20 lines).

Rather than excluding a string, a second filter can also be used to further include only lines matching an additional term:

cat /var/log/syslog | grep Listening | grep dbus

There is no practical limit on how many grep stages can be chained in a single pipeline. This particular chain narrows the result down to just three lines — few enough to inspect directly.

To preserve that output for later reference (by yourself or colleagues), redirect it to a file:

cat /var/log/syslog | grep Listening | grep dbus >> /home/ubuntu/project/dbus_logs.txt

It is good practice to use the full file system path in redirection targets, since a script containing this command might one day run from an unexpected working directory. Note also the use of two greater-than characters (>>) rather than one: a single > would overwrite any existing content in the destination file, while >> appends to whatever is already there.

cat dbus_logs.txt

Finally, when the exact search term is already known — particularly one containing spaces, dots, or other characters with special meaning as regular expressions — the string should be enclosed in quotation marks (either single or double, both work) to force grep to treat it literally rather than interpreting embedded regular-expression metacharacters:

grep "session opened for user" /var/log/syslog

flowchart LR
    A["/var/log/syslog<br/>(~5000 lines)"] --> B["grep Listening<br/>(32 lines)"]
    B --> C["grep -v agent<br/>(~20 lines, excludes 'agent')"]
    C --> D["grep dbus<br/>(3 lines, includes only 'dbus')"]
    D --> E[">> dbus_logs.txt<br/>(appended, saved for later)"]

Searching the File System with locate and find

When a file system contains hundreds of thousands of files, it is only a matter of time before something important becomes hard to find. Two complementary tools address this: locate and find.

locate relies on a pre-built index containing information about every directory and file on the system. Because the index can fall out of sync every time files are created, moved, or deleted, it is typically refreshed automatically once per day — but it can also be updated manually:

sudo updatedb

Manually refreshing is useful when you know the file you are looking for was created very recently. The tool itself is simple: follow locate with a search string that should match at least part of a real file name.

locate "apache2.conf"

Quotation marks are used here to escape the dot, which would otherwise be treated as a regular-expression wildcard. This returns two hits nearly instantly: the actual configuration file in /etc/apache2, and a metadata file that is part of the APT package repository information for the Apache2 package.

locate output can also be piped to grep to narrow down results that are otherwise too broad:

locate cron | grep sysstat

A typical /etc directory contains several cron-related subdirectories (cron.daily, cron.hourly, cron.monthly, cron.weekly) holding configuration for scheduled tasks — cron being the traditional Linux system for automated task scheduling. Searching for a remembered but vaguely-located tool called sysstat by running locate cron alone returns too many results, but piping to grep sysstat immediately narrows it down to the two files and one directory using that name.

locate is also excellent for tracking down old or half-remembered files spread across many directories accumulated over years of work — a quick locate <keyword> | grep <refinement> search can save significant time compared with manually browsing a large file tree.

locate’s close cousin, find, shares the same overall goal but works very differently — rather than consulting a pre-built index, it performs a live search of the file system, which makes it slower but always fully up to date and far more capable of filtering by attributes beyond just the name.

sudo find / -iname "*.csv"

This begins the search at the root directory (/) and, via -iname, performs a case-insensitive, loosely matching search for names ending in .csv. (The stricter -name argument performs a case-sensitive match instead.) sudo is required because find will attempt to look inside protected system directories; expect to see a number of “permission denied” errors for directories containing dynamic resources the current user cannot read — this is normal and unavoidable.

sudo find / -iname "*.csv" | wc -l
# ~600

Piping the results to wc confirms there are close to 600 matching CSV files across the system, and the output can naturally be piped further to grep to zero in on specific files of interest.

flowchart TD
    A[Need to find a file] --> B{Know it existed recently<br/>and want speed?}
    B -->|Yes| C["locate <keyword><br/>(fast, index-based, may be stale)"]
    B -->|No — need live,<br/>attribute-based search| D["find <path> -iname/-name/-type/-size/...<br/>(slower, always current, highly flexible)"]
    C --> E[Refine with grep if too many hits]
    D --> E

Refining Searches by Object Attributes

find becomes especially powerful once its attribute-filtering options come into play.

find ~/demos -type d

The -type d argument restricts results to directories only (excluding files and symlinks). The tilde (~) is shorthand for the current user’s home directory (equivalent to /home/ubuntu), so this lists just the directories nested within ~/demos — in this example, demos and old.

find ~/demos -type f

Replacing d with f returns files instead, no matter how deeply nested, each printed with its complete file system path.

A more advanced example combines find with -exec to run another command against each result:

find ~/demos -type f -exec grep -hi "31:40" {} \;

Rather than piping the simple list of file names to grep afterward, -exec executes a grep command within the framework of find itself, searching inside each returned file for the string 31:40 (a timestamp fragment known in advance to exist in one of the files). The -h flag forces grep to print each match’s file name, and -i makes the search case-insensitive (though it has no effect here since the search string contains no letters). The curly braces ({}) act as a placeholder that gets replaced with each matched file name in turn, and the trailing backslash before the semicolon (\;) escapes the semicolon so the shell does not interpret it prematurely, marking the end of the -exec command.

Running this returns the directory location and name of the file with a positive match, along with the matching timestamped log line itself.

find can also filter by modification time:

find /var/log -mtime -7

Given a directory containing many files (69, in this example), -mtime -7 returns only files modified within the past 7 days (more precisely, within the last seven 24-hour periods before the start of today). Using a plus sign instead of a minus sign (-mtime +7) would return files whose last modification is older than 7 days. Changing the value narrows or widens the window accordingly:

find /var/log -mtime -2

Finding empty files is another common use case:

find . -empty

This can help clear out unused objects in busy directories, though a large number of empty files can also be a symptom of failing processes worth diagnosing.

Full-system searches can produce overwhelming numbers of results:

find / -type f | wc -l
# 473000

One tool for narrowing that down without needing to change the starting directory is -maxdepth:

find / -maxdepth 2 | wc -l
# 161

-maxdepth 2 limits results to two levels of directories — the root directory itself and the immediate contents of the dozen or so subdirectories directly beneath it — dramatically narrowing the search.

Searching by file size helps identify what is consuming storage capacity:

find / -size +1G

This returns only files larger than 1 GB — useful for tracking down runaway log files or forgotten large downloads whenever a storage drive is nearing capacity. In one example, the single file matching this filter turned out to be a large CSV dataset downloaded for analytics purposes.

Combining -size with -exec provides sizes alongside the matched files:

find / -size +500M -exec ls -sh {} \;

The -s flag on ls prints each file’s size, and -h renders the size in human-readable units rather than raw byte counts — an excellent starting point for freeing up disk space when several oversized files are identified this way.

flowchart LR
    A["find ~/demos -type f"] --> B["-exec grep -hi 'pattern' {} \;"]
    B --> C["grep runs once per matched file<br/>{} substituted with each file path"]
    C --> D["Matching lines printed,<br/>prefixed with source file name (-h)"]

Module 3: Editing and Manipulating Data Streams

Modern data parsing is usually done in spreadsheets or with Python analytics tools, but the Linux command line offers surprisingly powerful native parsing tools that are often faster and more convenient for quick, targeted tasks.

Ordering Data with sort

Given a plain-text file listing programming languages, sort reorders the list alphabetically without modifying the original file (the new order could optionally be redirected to overwrite the file or written to a new one):

sort languages.txt

Given a file of unordered numbers, applying sort by default treats the values as text rather than numbers:

sort numbers.txt
# 1
# 10
# 150
# 2
# 23
# 3

Notice that 23 appears after 150 — because, by default, sort ignores numeric value and orders lexicographically (character by character). Adding the -n flag restores expected numeric ordering:

sort -n numbers.txt
# 1
# 2
# 3
# 10
# 23
# 150

sort also handles complex, structured text such as log timestamps. Extracting the final 10 lines of the system log with tail and saving them to a file:

tail /var/log/syslog > log.txt

Feeding that file to sort shows it can correctly parse complex timestamps, but by default the output order is unchanged from tail’s output — because tail already prints entries in ascending (oldest-to-newest) order, and sort’s default behavior is also ascending.

sort log.txt

Adding the -r (reverse) flag shows the most recent events first:

sort -r log.txt

Other useful sort flags include -f, which ignores the difference between uppercase and lowercase characters when comparing values.

Parsing Structured Data with sort, head, and tail

Consider a CSV file, sales.csv, containing fictional sales transaction data with ten comma-separated columns: transaction ID, transaction date, client name/region, product category, units sold, unit price, gross revenue, and profit margin.

cat sales.csv

To reorder rows by profit — highest to lowest — a more advanced sort invocation is required:

sort -t, -k10 -r sales.csv

• -t, sets the field delimiter to a comma (the natural choice for a CSV file, though embedded commas within individual fields can complicate this).
• -k10 tells sort to compare based specifically on the 10th field — the profit column in this data set.
• -r sorts in descending order, from highest to lowest.

The output correctly ranks the profit column from a high of $830 down to a low of $80 at the bottom — however, the header row ends up sorted to the bottom of the output as well, which hurts readability. A future version of sort is expected to support a --header flag to solve this directly:

# Not yet available in all sort versions — future syntax:
sort -t, -k10 -r --header sales.csv

Until that feature is broadly available, a simple workaround uses head and tail together:

head -n 1 sales.csv
tail -n +2 sales.csv | sort -t, -k10 -r

head -n 1 prints just the header row on its own. tail -n +2 instructs tail to start printing from the second line of the file onward (rather than its usual behavior of printing only the final 10 lines), effectively skipping the header before piping the remaining data rows into sort.

flowchart LR
    A["sales.csv"] --> B["head -n 1 sales.csv<br/>(prints header alone)"]
    A --> C["tail -n +2 sales.csv<br/>(skips header, keeps data rows)"]
    C --> D["sort -t, -k10 -r<br/>(sort data rows by profit, descending)"]
    B --> E[Combined: readable header + sorted data]
    D --> E

Basic analytics are also possible directly at the command line. The cut command (explored in depth in the next section) extracts a single field/column from a stream — useful for parsing high-volume log data to identify patterns quickly:

cut -d, -f10 sales.csv

-d specifies the comma as the field delimiter, and -f10 selects the 10th field from the left — the profit column. Piping that isolated column to sort -n and then to uniq -d reveals any repeated profit values:

cut -d, -f10 sales.csv | sort -n | uniq -d
# 260

uniq -d reports only values that occur more than once — in this data, at least two transactions shared an identical $260 profit value. While not hugely meaningful for this small illustrative example, this kind of duplicate-detection technique can expose valuable, hard-to-spot patterns in much larger datasets.

Storage usage can be quickly summarized using du (disk usage), which estimates how much space directories consume:

du -h /var/log | sort -h

The -h flag (on both du and sort) renders sizes in human-readable units (K, M, G) rather than raw byte counts, and piping to sort -h reorders the output from smallest to largest. One caveat: this output does not visually distinguish files from directories — an entry might represent a directory whose combined contents total, say, 62 MB, or it might represent a single large file like syslog.

Extracting Fields with cut

Field-based extraction with cut depends on a consistent delimiter character. sales.csv uses commas, but sometimes another delimiter — such as a tab — is needed. The sed stream editor can transform delimiters across an entire file:

sed 's/,/\t/g' sales.csv > sales.tsv
cat sales.tsv

The sed expression is enclosed in single quotes. It begins with s (substitute), targets the comma, replaces it with a tab character (\t), and the trailing g applies the substitution globally — to every comma in the document, not just the first one per line. The result is written to a new file, sales.tsv (the accepted convention for tab-separated-value documents), leaving the original sales.csv untouched. (A GUI text editor’s find-and-replace dialog — using \t as the replacement text — can achieve the same transformation interactively, but sed allows the exact same operation to be scripted and repeated at the command line.)

With tab-separated data available, cut can extract the date column and further split it into year, month, and day sub-fields:

cut -d$'\t' -f2 sales.tsv | cut -d- -f2

The first cut sets the tab character combination as the delimiter (-d$'\t') and selects field 2 — the date-stamp column. That result is piped into a second cut, this time using a dash (-) as the delimiter and selecting the second of the three date sub-fields (the month). The output reveals all transaction months at a glance — in this example, exclusively January or February — a pattern that becomes far more valuable to visualize as data volume grows.

cut’s usefulness has practical limits when a data source lacks a single consistent delimiter. An Apache web server access log illustrates this:

cat access.log

No single delimiter character cleanly separates every field — quotation marks work for some segments, spaces for others, but even spaces are unreliable because some fields (like user-agent strings) contain embedded spaces of their own. Even so, using a space as the delimiter for the first field reliably isolates origin IP addresses, and the ninth field reliably captures the HTTP response code:

cut -d' ' -f1 access.log
cut -d' ' -f9 access.log

This works acceptably well for this particular log format, even if it is not perfectly robust for every field.

An authentication log, /var/log/auth.log, offers more consistent structure. A sample row contains six colons (:) which, when highlighted, create a total of seven fields — not all of which are meant as delimiters, but all of which are usable for that purpose:

cut -d: -f1 /var/log/auth.log

Requesting field 1 with a colon delimiter returns the full date stamp, though the hour/minute (t14/t15-style) values at the end of the timestamp actually continue into the next two colon-delimited fields, since colons are also used inside the time portion of the stamp. Extracting just the month requires a second nested cut:

cut -d: -f1 /var/log/auth.log | cut -d- -f2 | uniq -d
# 12

Piping the first field into a second cut (using a dash delimiter this time) isolates the month sub-field, and piping that to uniq (without arguments, reporting values that appear more than once) shows that the number 12 — December — is the only repeating month value across the file.

Finally, combining the first three colon-delimited fields reconstructs the complete date-and-time stamp in one operation:

cut -d: -f1-3 /var/log/auth.log

flowchart TD
    A["sales.csv (comma-delimited)"] -->|"sed 's/,/\t/g'"| B["sales.tsv (tab-delimited)"]
    B -->|"cut -d$'\t' -f2 (date column)"| C["2024-01-15"]
    C -->|"cut -d- -f2 (month sub-field)"| D["01"]

Quantifying Data with wc

wc (word count) reports the number of lines, words, and bytes in a stream — its basic form works against an entire file such as /var/log/syslog:

wc /var/log/syslog

Individual metrics can be isolated with dedicated flags:

wc -l /var/log/syslog   # lines only
wc -w /var/log/syslog   # words only
wc -c /var/log/syslog   # bytes only

Using a wildcard produces sizes for every matching file plus a running total:

wc -l *.log

A particularly useful pattern combines wc with watch, which executes a specified command repeatedly at a fixed interval:

watch -n 1 "wc -l /var/log/apache2/access.log"

-n 1 sets a one-second execution interval. access.log records every remote request against the local Apache web server. Watching the line count update in real time (for example, while repeatedly refreshing the server’s home page in a browser) provides a simple but effective live-monitoring technique — useful for spotting anomalies such as a possible denial-of-service (DDoS) attack.

wc handles very large files efficiently:

wc -l large-dataset.csv
# 7823451 large-dataset.csv

Counting nearly 7.8 million rows this way takes only a couple of seconds — far faster than loading the same file into a GUI spreadsheet application (assuming there is even enough RAM available to do so) and scrolling to the bottom to check the row count.

A more advanced example filters out empty and comment lines before counting:

grep -vE '^#|^\s*$' large-dataset.csv | wc -l

The -vE combination tells grep to return only lines that do not match either of two alternate regular expressions: lines beginning with a # (comments) or lines consisting only of whitespace (empty lines). Piping the filtered result to wc -l produces a count of genuinely populated data rows.

sequenceDiagram
    participant User
    participant Watch as watch -n 1
    participant WC as wc -l access.log
    User->>Watch: Start monitoring
    loop Every 1 second
        Watch->>WC: Execute
        WC-->>Watch: Current line count
        Watch-->>User: Refreshed display
    end
    Note over User,WC: Sudden spikes in line count<br/>can indicate abnormal traffic (e.g. DDoS)

Module 4: Redirecting Data Streams

Redirection — rather than GUI tools or manually editing configuration files — is how Linux administrators accomplish a large share of their day-to-day work, enabling far more automation than manual point-and-click alternatives. Getting comfortable with redirection requires some initial effort, but the underlying concepts pay off immediately once learned. This module covers appending and overwriting file content, the three standard data streams, and chaining commands together — including inside scripts.

Appending and Overwriting Output

Consider a task: assemble a set of commands that generate a daily automated system-health report. The report should be available for administrators to consult throughout the day, then rewritten from scratch each morning. Useful building blocks include:

• date — prints the current date and time.
• uptime — shows time elapsed since the last reboot, number of logged-in users, and recent CPU load averages.
• df -h — prints used and available space per partition, in human-readable units.
• free -h — prints used and available system RAM (physical and swap), in human-readable units.

date > system_report.txt

A single > either creates the file if it does not exist, or completely overwrites its existing contents if it does — appropriate here since the goal is a fresh report each run.

echo "Uptime:" >> system_report.txt
uptime >> system_report.txt

>> appends this section without disturbing what came before, producing an uptime section containing the header text followed by the actual output of uptime (hours since boot, logged-in user count, and recent load averages).

echo "Disk Usage:" >> system_report.txt
df -h >> system_report.txt
cat system_report.txt

Printing the report so far confirms the date, uptime, and disk-usage sections are all present in the expected order. In a production setting, all of these commands would normally be assembled into a single script and automated — a technique demonstrated later in this module.

One more section is still needed: system memory and network status.

echo "Memory Usage:" >> system_report.txt
free -h >> system_report.txt
cat system_report.txt

The memory section correctly lists both physical and swap capacity. However, adding the final network section illustrates exactly how easy it is to make a costly redirection mistake:

echo "Network Status:" > system_report.txt

Printing the report immediately afterward shows it is nearly empty — because a single > was used instead of >>. Instead of appending the new “Network Status:” heading to the end of the file, it silently overwrote all of the previously collected date, uptime, disk, and memory sections with just that one line.

flowchart TD
    A["date > system_report.txt"] --> B["echo/uptime >> system_report.txt"]
    B --> C["echo/df -h >> system_report.txt"]
    C --> D["echo/free -h >> system_report.txt"]
    D --> E{"Next append uses > instead of >>?"}
    E -->|"Yes (mistake)"| F["Entire report wiped — only the last line survives"]
    E -->|"No (correct: >>)"| G["Report retains all prior sections plus the new one"]

Understanding Standard Input, Output, and Error

Every piece of data moving through a Linux system falls into one of three categories:

• Standard input (stdin) — data coming in from somewhere else, such as a keyboard. Short form stdin, numbered 0.
• Standard output (stdout) — the normal, expected data a program generates. Short form stdout, numbered 1.
• Standard error (stderr) — error and diagnostic messages produced when something goes wrong. Numbered 2.

A simple mnemonic: “0 reads, 1 talks, 2 complains.”

The pipe (|) is the mechanism used throughout this course to connect one program’s stdout to another program’s stdin, forming chains of commands. The keyboard itself is also a valid source of stdin. For example, invoking grep with a search term but no target file causes it to wait for typed input:

grep stream

With no file specified, grep assumes text should be typed directly into the shell. Any pasted or typed paragraph containing the word “stream” will be matched and printed back, with the match highlighted, once Enter is pressed. Ctrl+D signals end-of-input and exits this mode.

stdout is the everyday, expected result of a command:

cat access.log | wc -c
echo "Backup successful" > myfile
cat myfile

The content cat sends to wc, and the text echo writes to myfile, are both considered stdout because they represent the desired, successful output of each command. Running the same echo command without the redirect prints “Backup successful” to the terminal instead:

echo "Backup successful"

This happens because stdout defaults to the terminal screen unless explicitly redirected elsewhere.

stderr is easy to demonstrate — it only requires a mistake:

ls /nodir
# ls: cannot access '/nodir': No such file or directory

Since /nodir does not exist, this produces an error message. Even though it might seem like ordinary command output, it is in fact stderr — a message specifically defined by the developers of ls to help users understand what went wrong, deliberately separated from normal stdout.

To highlight this distinction further:

ls . ~

This requests a double listing: . (the current directory) and ~ (the current user’s home directory) — both printed as ordinary stdout, current directory contents first.

ls /etc /nodir > output.txt
cat output.txt

If both directories existed, both listings would be written into output.txt. Since /nodir does not exist, the error message for /nodir prints immediately to the terminal (because stderr was never redirected), while the successful /etc listing is still correctly written into output.txt. Printing output.txt confirms the error text never made it into the file — Linux correctly routed stdout and stderr to their separate destinations, even though both streams originated from a single command invocation.

flowchart LR
    K["Keyboard (stdin, fd 0)"] --> P["Running Command"]
    P -->|"Normal results (stdout, fd 1)"| O1["Terminal screen (default)<br/>or redirected file"]
    P -->|"Errors / diagnostics (stderr, fd 2)"| O2["Terminal screen (default)<br/>or redirected file"]

Chaining Commands and Building Robust Scripts

Many Linux commands include built-in handling for their own error messages, but it is also possible to manually redirect stderr to a custom destination:

ls /etc /nodir 2> errors.txt
cat errors.txt

The 2> syntax specifically targets file descriptor 2 (stderr). Here, the contents of /etc still print normally to the screen (as stdout), while any error generated by the failed /nodir lookup is instead written into errors.txt rather than appearing on-screen. Printing errors.txt confirms the error message landed there as intended, instead of at the terminal.

Redirecting error messages this way is genuinely useful in production scripts — it forms the foundation of proper error handling. It is also possible to combine both streams into a single destination:

ls ~ /nodir &> output.txt
cat output.txt

&> tells Linux to redirect both stdout and stderr to the very same destination. Because a single > is used, any existing content in output.txt is also overwritten first. Running this shows no visible error on-screen at all — instead, the error message appears at the very start of output.txt, followed by the successful home-directory listing.

This capability becomes especially valuable inside a Bash script. A script is simply a plain-text file, typically using a .sh extension, that begins with a shebang line — a hash character followed by an exclamation mark and the file system path of the shell that should interpret the script:

#!/bin/bash

echo "Starting backup..."

cp important.txt /backup/

if [ $? -ne 0 ]; then
    echo "Backup failed" >&2
else
    echo "Backup succeeded"
fi

Walking through this script:

• The shebang (#!/bin/bash) specifies that Bash should interpret the script. Different shells can have subtle syntax differences, so this line matters.
• echo "Starting backup..." writes a status update to stdout, letting a user know a longer-running operation is in progress rather than appearing to hang.
• cp important.txt /backup/ attempts to copy a file into a backup directory. If either important.txt or /backup/ does not actually exist, this command fails.
• The if/else block inspects $?, a special shell variable holding the exit status (an integer) of the most recently executed foreground command. An exit status of 0 exclusively means success; any non-zero value indicates failure.
• If the exit status is not 0 (-ne 0), the script prints “Backup failed” to stderr (>&2 redirects this specific echo’s output to file descriptor 2). Otherwise, it prints “Backup succeeded” to standard output.
• fi closes the if block (read as “if” spelled backwards, and also the start of the word “finish”).

Before a script can be executed directly, it needs the executable permission bit set, using chmod (change mode):

chmod +x backup.sh
ls -l backup.sh

The +x flag adds the executable bit, and the directory listing now highlights the script in a different color to indicate its executable status.

./backup.sh

The leading ./ tells the shell to look for the script in the current directory rather than searching the system PATH. Since important.txt and /backup/ do not exist in this example, the “Starting backup…”, the underlying cp error message, and finally “Backup failed” all print to the terminal in sequence, essentially instantly.

Redirecting stdout and stderr to separate log files demonstrates the payoff of proper stream separation within a script:

./backup.sh > success.log 2> error.log
cat success.log
cat error.log

• success.log contains only “Starting backup…” — there is no “Backup succeeded” message, because the backup did not, in fact, succeed.
• error.log contains both the internal error message generated by cp itself and the script’s own “Backup failed” message — because both were explicitly directed to stderr.

In other words, this script correctly and cleanly handles its own errors, in large part through disciplined, correct use of stream redirection.

flowchart TD
    A["#!/bin/bash"] --> B["echo 'Starting backup...' (stdout)"]
    B --> C["cp important.txt /backup/"]
    C --> D{"$? -ne 0 ?"}
    D -->|"Yes: cp failed"| E["echo 'Backup failed' >&2 (stderr)"]
    D -->|"No: cp succeeded"| F["echo 'Backup succeeded' (stdout)"]
    E --> G["fi"]
    F --> G

Summary

This course walked through the practical foundations of working with plain-text data at the Linux command line, across four connected areas:

• File and directory management — creating files and directories with nano, touch, and mkdir (including the recursive -p path-creation flag); copying and moving objects locally and across the network with cp, mv, and scp; safely deleting files and directories with rm, rmdir, and the recursive/interactive -r/-i flags; and understanding the crucial distinction between hard links (which share an inode and survive deletion of the original path) and symbolic links (lightweight path pointers that break when their target disappears).
• Filtering data streams — using wildcards (* and ?) and globbing to operate on precisely the right set of files; mastering grep for line-based text filtering, including inversion (-v), chaining multiple filters, and quoting literal search strings; and using locate (fast, index-based) and find (slower, live, and far more attribute-aware — by type, modification time, size, and emptiness) to track down files across large file systems.
• Editing and manipulating data streams — reordering data with sort (including numeric, reverse, and field-based sorting for structured CSV data), combining head/tail to work around header rows, extracting columns with cut across varying delimiter schemes, and quantifying data volume and patterns with wc, including real-time monitoring via watch.
• Redirecting data streams — the critical difference between overwriting (>) and appending (>>); the three standard streams stdin/stdout/stderr and their default behaviors; redirecting stdout and stderr independently or together; and applying all of this inside a real Bash script that inspects $? to make robust, automatable decisions about success and failure.

The unifying theme across all four modules is that Linux treats nearly everything as a stream of text, and the shell provides a rich, composable toolkit — pipes, redirection operators, and small single-purpose utilities — for shaping, filtering, and routing that text exactly where it needs to go. Mastery of these tools is what allows administrators to automate work that would otherwise require slow, manual, GUI-based intervention.

Command-Line Quick Reference Cheat Sheet

Search Terms

mastering · command-line · interface · cli · linux · systems · administration · networking · security · data · deleting · directories · output · streams · filtering · moving · sort · system