Shell Scripting File Operations: Read, Write & Manage Files
Files are the foundation of Unix-based systems. Everything is a file — configurations, logs, devices, and even processes. If you want to automate system administration, data processing, or deployment pipelines, you need to be fluent in file operations.
In this guide, we'll cover reading files efficiently, writing and appending data, manipulating files and directories, searching with grep and sed, and handling structured formats like CSV and JSON.
Reading Files
Shell scripts read files constantly — parsing logs, loading configurations, processing data. The method you choose depends on whether you need the entire file at once or line by line.
Read Entire File into a Variable
For small files, you can load everything into memory at once:
1#!/bin/bash 2 3content=$(cat file.txt) 4echo "$content"
Caution: Only use this for files you know are small. Loading a multi-gigabyte log file into a variable will exhaust system memory.
Read Line by Line (The Right Way)
For most file processing tasks, read line by line using a while loop:
1#!/bin/bash 2 3while IFS= read -r line; do 4 echo "Line: $line" 5done < file.txt
Why this pattern is best:
IFS=prevents leading/trailing whitespace from being trimmedread -rprevents backslash escapes from being interpreted< file.txtredirects the file into the loop (more efficient thancat file.txt | while)
Read with Line Numbers
When you need to reference line numbers — for error reporting or data processing:
1#!/bin/bash 2 3line_num=1 4while IFS= read -r line; do 5 echo "$line_num: $line" 6 ((line_num++)) 7done < file.txt
Read a Specific Line
For targeted extraction without loading the entire file:
1#!/bin/bash 2 3# Read line 5 only 4sed -n '5p' file.txt 5 6# Read first line 7head -n 1 file.txt 8 9# Read last line 10tail -n 1 file.txt 11 12# Read last 10 lines 13tail -n 10 file.txt 14 15# Read lines 10-20 16sed -n '10,20p' file.txt
Read First N Lines with a Limit
1#!/bin/bash 2 3# Preview first 5 lines of a large log 4head -n 5 /var/log/syslog 5 6# Preview last 5 lines 7tail -n 5 /var/log/syslog 8 9# Follow a log in real-time (press Ctrl+C to stop) 10tail -f /var/log/nginx/access.log
Read File into an Array
1#!/bin/bash 2 3# Load all lines into an array 4mapfile -t lines < file.txt 5 6echo "Total lines: ${#lines[@]}" 7echo "First line: ${lines[0]}" 8echo "Last line: ${lines[-1]}"
Writing Files
Creating and modifying files is just as important as reading them. Bash provides several ways to write data, each suited to different scenarios.
Overwrite a File (>)
1#!/bin/bash 2 3echo "This is a new file." > file.txt
The > operator creates the file if it doesn't exist, or truncates it to zero length if it does. Use with caution.
Append to a File (>>)
1#!/bin/bash 2 3echo "Log entry at $(date)" >> file.txt
The >> operator adds content to the end without touching existing data. This is the standard way to build log files.
Write Multi-Line Content with Here Documents
For configuration files, scripts, or any multi-line text:
1#!/bin/bash 2 3cat > config.ini << EOF 4[database] 5host=localhost 6port=5432 7name=myapp 8 9[server] 10bind=0.0.0.0 11port=8080 12EOF
Tip: Use
<< 'EOF'(with quotes) to prevent variable expansion if you're writing literal text that contains$symbols.
Formatted Output with printf
When you need precise formatting — columns, padding, or specific number formats:
1#!/bin/bash 2 3printf "%-15s %5d\n" "Alice" 25 > output.txt 4printf "%-15s %5d\n" "Bob" 30 >> output.txt 5printf "%-15s %5d\n" "Charlie" 35 >> output.txt 6 7cat output.txt
Output:
Alice 25
Bob 30
Charlie 35
Write Only If File Doesn't Exist
Prevent accidental overwrites:
1#!/bin/bash 2 3if [ -e "important.conf" ]; then 4 echo "File already exists. Not overwriting." 5else 6 cat > important.conf << EOF 7key=value 8EOF 9fi
File Manipulation
Beyond reading and writing, scripts constantly create, copy, move, and delete files and directories.
Check if File Exists
Always verify before operating:
1#!/bin/bash 2 3if [ -f "file.txt" ]; then 4 echo "File exists." 5 rm file.txt 6else 7 echo "File not found." 8fi
Create Directories
1#!/bin/bash 2 3# Create directory and all parent directories if they don't exist 4mkdir -p /path/to/new/dir 5 6# Create with specific permissions 7mkdir -p -m 755 /var/app/data
Copy Files
1#!/bin/bash 2 3# Simple copy 4cp source.txt dest.txt 5 6# Copy and preserve metadata (timestamps, permissions) 7cp -p source.txt dest.txt 8 9# Recursive copy 10cp -r source_dir/ dest_dir/ 11 12# Copy only newer files 13cp -u source.txt dest.txt
Move and Rename
1#!/bin/bash 2 3# Rename a file 4mv old.txt new.txt 5 6# Move to another directory 7mv file.txt /backup/ 8 9# Move multiple files 10mv *.log /var/archive/
Delete Files and Directories
1#!/bin/bash 2 3# Delete a single file 4rm file.txt 5 6# Delete a directory and all contents 7rm -r directory/ 8 9# Delete only empty directories 10rmdir empty_dir/ 11 12# Force delete without prompts 13rm -rf old_build/
Warning:
rm -rfis permanent and powerful. Always double-check your path before executing.
Find Files
The find command is indispensable for locating files based on name, size, age, or type:
1#!/bin/bash 2 3# Find all .log files in /var/log 4find /var/log -name "*.log" 5 6# Find files modified more than 7 days ago 7find /var/log -name "*.log" -mtime +7 8 9# Find files larger than 100MB 10find /var/www -size +100M 11 12# Find and delete old logs 13find /var/log -name "*.tmp" -mtime +7 -delete 14 15# Find and execute a command on each file 16find . -name "*.sh" -exec chmod +x {} \;
Search File Contents with grep
1#!/bin/bash 2 3# Search for a pattern in a file 4grep "ERROR" file.txt 5 6# Recursive search in a directory 7grep -r "ERROR" /var/log/ 8 9# Case-insensitive search 10grep -i "error" file.txt 11 12# Show line numbers 13grep -n "pattern" file.txt 14 15# Invert match (lines NOT containing pattern) 16grep -v "DEBUG" app.log 17 18# Count matches 19grep -c "ERROR" app.log 20 21# Show context (2 lines before and after) 22grep -C 2 "CRITICAL" app.log 23 24# Search with regular expressions 25grep -E "^[0-9]{4}-[0-9]{2}" dates.txt
Search and Replace with sed
1#!/bin/bash 2 3# Replace first occurrence on each line 4sed -i 's/old/new/' file.txt 5 6# Replace all occurrences 7sed -i 's/old/new/g' file.txt 8 9# Replace on specific line 10sed -i '5s/old/new/' file.txt 11 12# Delete lines matching pattern 13sed -i '/^#/d' file.txt 14 15# Delete empty lines 16sed -i '/^$/d' file.txt 17 18# Insert text before line 1 19sed -i '1i # Header comment' file.txt
Note: On macOS,
sed -irequires a backup extension:sed -i '' 's/old/new/' file.txt
Count Words, Lines, and Characters
1#!/bin/bash 2 3# Count lines 4wc -l file.txt 5 6# Count words 7wc -w file.txt 8 9# Count bytes 10wc -c file.txt 11 12# All three at once 13wc file.txt
Working with CSV and JSON
Structured data formats are common in modern scripting. Here's how to handle them in Bash.
Processing CSV Files
The standard approach uses IFS=',' with read:
1#!/bin/bash 2 3# data.csv: 4# Alice,30,New York 5# Bob,25,Los Angeles 6# Carol,35,Chicago 7 8while IFS=',' read -r name age city; do 9 echo "Name: $name, Age: $age, City: $city" 10done < data.csv
Output:
Name: Alice, Age: 30, City: New York
Name: Bob, Age: 25, City: Los Angeles
Name: Carol, Age: 35, City: Chicago
CSV with Headers
Skip the header line before processing:
1#!/bin/bash 2 3# Skip header and process data 4tail -n +2 data.csv | while IFS=',' read -r name age city; do 5 echo "$name is $age years old and lives in $city." 6done
CSV with Quoted Fields
For complex CSVs with quoted fields containing commas, use awk or Python:
1#!/bin/bash 2 3# Simple awk approach for basic quoted CSVs 4awk -F'"?,"?' '{print "Name: " $1 ", Role: " $2}' employees.csv
Best Practice: For production CSV processing, use
python3 -cwith thecsvmodule or tools likecsvkit. Pure Bash CSV parsing breaks on edge cases.
Processing JSON
Bash has no native JSON parser. For simple extractions, use jq — the command-line JSON processor.
1#!/bin/bash 2 3# Pretty-print JSON 4cat data.json | jq '.' 5 6# Extract a specific field 7cat data.json | jq '.name' 8 9# Extract nested field 10cat data.json | jq '.user.email' 11 12# Extract array elements 13cat data.json | jq '.users[].name' 14 15# Filter array 16cat data.json | jq '.users[] | select(.age > 25)' 17 18# Write JSON output 19jq -n '{name: "Alice", age: 30, active: true}'
JSON Without jq
If jq isn't available, use Python as a fallback:
1#!/bin/bash 2 3# Extract value using Python 4name=$(python3 -c "import json,sys; print(json.load(sys.stdin)['name'])" < data.json) 5echo "Name: $name" 6 7# Iterate over array 8python3 -c " 9import json, sys 10data = json.load(sys.stdin) 11for user in data['users']: 12 print(f\"{user['name']}: {user['email']}\") 13" < data.json
Real-World Example: Log Analyzer Script
Let's combine everything into a production-ready log analysis tool:
1#!/bin/bash 2 3set -euo pipefail 4 5LOG_FILE="${1:-/var/log/syslog}" 6REPORT_FILE="log_report_$(date +%Y%m%d).txt" 7 8if [ ! -f "$LOG_FILE" ]; then 9 echo "Error: Log file not found: $LOG_FILE" >&2 10 exit 1 11fi 12 13echo "=== Log Analysis Report ===" > "$REPORT_FILE" 14echo "Source: $LOG_FILE" >> "$REPORT_FILE" 15echo "Generated: $(date)" >> "$REPORT_FILE" 16echo "" >> "$REPORT_FILE" 17 18# Count total lines 19total_lines=$(wc -l < "$LOG_FILE") 20echo "Total Lines: $total_lines" >> "$REPORT_FILE" 21 22# Count errors 23error_count=$(grep -ci "error" "$LOG_FILE" || true) 24echo "Error Mentions: $error_count" >> "$REPORT_FILE" 25 26# Count warnings 27warn_count=$(grep -ci "warn" "$LOG_FILE" || true) 28echo "Warning Mentions: $warn_count" >> "$REPORT_FILE" 29 30# Extract top 5 error messages 31echo "" >> "$REPORT_FILE" 32echo "Top Error Patterns:" >> "$REPORT_FILE" 33grep -i "error" "$LOG_FILE" | sed 's/^.*ERROR: //i' | sort | uniq -c | sort -rn | head -n 5 >> "$REPORT_FILE" 34 35# Find entries from last hour 36echo "" >> "$REPORT_FILE" 37echo "Recent Entries (last hour):" >> "$REPORT_FILE" 38current_hour=$(date +%H) 39grep "^$(date +%b' '%e) $current_hour:" "$LOG_FILE" | tail -n 10 >> "$REPORT_FILE" 40 41echo "Report saved to: $REPORT_FILE"
Best Practices for File Operations
- Always check if files exist before reading or deleting
- Quote filenames — spaces and special characters will break unquoted paths
- Use
set -euo pipefailso your script exits on errors instead of continuing with bad data - Prefer
>>over>for logs unless you explicitly want truncation - Use
mkdir -pto avoid errors when directories already exist - Never parse
ls— use glob patterns orfindinstead - Redirect errors with
2>/dev/nullwhen checking for file existence to keep output clean - Use
mktempfor temporary files to avoid collisions - Backup before in-place edits —
sed -i.bak 's/old/new/' file - Validate JSON/CSV with schema checks when data integrity matters
Quick Reference
1# Reading 2content=$(cat file.txt) 3while IFS= read -r line; do ...; done < file.txt 4mapfile -t lines < file.txt 5head -n 5 file.txt 6tail -n 5 file.txt 7sed -n '10p' file.txt 8 9# Writing 10echo "text" > file.txt 11echo "text" >> file.txt 12cat > file.txt << EOF 13printf "%s\n" "text" > file.txt 14 15# Manipulation 16cp src dest 17mv old new 18rm file 19rm -r dir 20mkdir -p dir 21find /path -name "*.log" -mtime +7 22 23# Searching 24grep "pattern" file 25grep -r "pattern" dir/ 26grep -i "pattern" file 27grep -n "pattern" file 28sed -i 's/old/new/g' file 29wc -l file 30 31# CSV 32while IFS=',' read -r c1 c2 c3; do ...; done < file.csv 33 34# JSON 35cat file.json | jq '.field' 36python3 -c "import json; ..." < file.json
File operations are the bread and butter of shell scripting. Whether you're parsing gigabytes of logs, generating configuration files, or cleaning up old backups, the techniques in this chapter give you the control and efficiency to handle files at any scale.
Next up: Error handling and debugging — because even the best scripts fail sometimes, and when they do, you need to know why.