Terminal GuideTerminal Guide

awk Command Guide

awk is a powerful text processing language. Learn how to extract fields, filter data, and perform calculations from the command line.

10 min read•Last updated: January 19, 2025
Dai Aoki

Dai Aoki

CEO at init, Inc. / CTO at US & JP startups / Creator of WebTerm

Quick Reference

Basic

awk '{print}' filePrint all lines
awk '{print $1}' filePrint first field
awk '{print $NF}' filePrint last field

Fields

-F:Set field separator
$1, $2, $3Field 1, 2, 3
NFNumber of fields

Pattern

/pattern/ {action}Pattern matching
$1 > 10 {print}Conditional
NR==1 {print}First line only

Variables

NRLine number
NFNumber of fields
FSField separator

Common

'{sum+=$1} END{print sum}'Sum column
'{print NR, $0}'Add line numbers
'!seen[$0]++'Remove duplicates

Downloadable Image Preview

Failed to generate preview

Basic Syntax

awk processes text line by line, splitting each line into fields.

bash
awk 'pattern { action }' filename

You can also pipe input to awk:

bash
echo "hello world" | awk '{ print $2 }'
# Output: world

Field Variables

awk automatically splits each line into fields (by whitespace by default).

Built-in Variables

$0Entire line
$1, $2, ...Individual fields
NFNumber of fields in current line
NRCurrent line number
FSField separator (default: whitespace)
OFSOutput field separator
RSRecord separator (default: newline)
bash
# Print first field
awk '{ print $1 }' file.txt

# Print multiple fields
awk '{ print $1, $3 }' file.txt

# Print last field
awk '{ print $NF }' file.txt

# Print second-to-last field
awk '{ print $(NF-1) }' file.txt

Field Separator

Change the field separator with -F.

bash
# CSV files (comma-separated)
awk -F',' '{ print $1, $2 }' data.csv

# Colon-separated (like /etc/passwd)
awk -F':' '{ print $1 }' /etc/passwd

# Multiple separators
awk -F'[,;:]' '{ print $1 }' file.txt

Pattern Matching

Process only lines that match a pattern.

bash
# Lines containing "error"
awk '/error/ { print }' log.txt

# Lines NOT containing "debug"
awk '!/debug/ { print }' log.txt

# Lines starting with #
awk '/^#/ { print }' config.txt

# Lines where field 3 > 100
awk '$3 > 100 { print }' data.txt

# Lines where field 1 equals "admin"
awk '$1 == "admin" { print }' users.txt

BEGIN and END Blocks

Execute code before or after processing lines.

bash
# Print header and footer
awk 'BEGIN { print "=== START ===" }
     { print }
     END { print "=== END ===" }' file.txt

# Set field separator in BEGIN
awk 'BEGIN { FS=":" } { print $1 }' /etc/passwd

# Calculate sum and average
awk 'BEGIN { sum=0 }
     { sum += $1 }
     END { print "Sum:", sum, "Avg:", sum/NR }' numbers.txt

Arithmetic Operations

bash
# Sum of a column
awk '{ sum += $1 } END { print sum }' numbers.txt

# Calculate percentage
awk '{ print $1, $2, ($2/$1)*100 "%" }' data.txt

# Running total
awk '{ total += $1; print $0, total }' numbers.txt

String Functions

String Functions

length(s)Length of string
substr(s, start, len)Substring extraction
index(s, target)Find position of substring
split(s, arr, sep)Split string into array
tolower(s)Convert to lowercase
toupper(s)Convert to uppercase
gsub(regex, repl, s)Global substitution
bash
# String length
awk '{ print length($0) }' file.txt

# Substring (first 10 characters)
awk '{ print substr($0, 1, 10) }' file.txt

# Convert to uppercase
awk '{ print toupper($0) }' file.txt

# Replace text
awk '{ gsub(/old/, "new"); print }' file.txt

Formatting Output

bash
# Printf for formatted output
awk '{ printf "%-10s %5d\n", $1, $2 }' data.txt

# Format with headers
awk 'BEGIN { printf "%-15s %10s\n", "Name", "Score" }
     { printf "%-15s %10d\n", $1, $2 }' scores.txt

# Set output field separator
awk 'BEGIN { OFS="," } { print $1, $2, $3 }' data.txt
Tip
Use printf for precise formatting. The format specifiers work like C's printf.

Practical Examples

Extract usernames from /etc/passwd

bash
awk -F':' '{ print $1 }' /etc/passwd

Sum file sizes from ls

bash
ls -l | awk '{ sum += $5 } END { print sum " bytes" }'

Count occurrences

bash
awk '{ count[$1]++ } END { for (word in count) print word, count[word] }' file.txt

Filter log by date

bash
awk '/2024-01-15/ { print }' access.log

Print lines longer than 80 characters

bash
awk 'length > 80 { print NR": "$0 }' file.txt

Remove duplicate lines

bash
awk '!seen[$0]++' file.txt

Parse Apache access log

bash
awk '{ print $1 }' access.log | sort | uniq -c | sort -rn | head -10

Conditional Statements

bash
# If-else
awk '{ if ($3 > 50) print "PASS:", $0; else print "FAIL:", $0 }' scores.txt

# Ternary operator
awk '{ print ($3 > 50 ? "PASS" : "FAIL"), $1 }' scores.txt

Summary

awk is invaluable for text processing and data extraction. Key takeaways:

  • Use $1, $2, ... to access fields
  • Use -F to change field separator
  • Use patterns to filter lines before actions
  • Use BEGIN and END for initialization and summary
  • Use printf for formatted output

Official Documentation

For authoritative information, refer to the official documentation:

Related Articles

sed Command Guide

Stream editor for text transformation

linux8 min read

grep Command: Practical Examples & Usage Guide

Master grep with practical examples for developers. Learn pattern matching, regex, and real-world use cases.

linux12 min read

xargs Command Guide

Build commands from standard input

linux6 min read