32blogby Studio Mitsu

jq Complete Guide: Master JSON Processing in the Terminal

Learn jq from basic filters to advanced JSON transformation: API responses, config files, and scripting integration with practical examples.

by omitsu15 min read
jqJSONCLILinuxbash
On this page

You need to extract specific fields from an API response. Parse a package.json to list dependencies. Filter error entries from JSON-formatted logs. All from the terminal, without writing a script.

That's what jq does. Often called the Swiss Army knife of JSON processing, it handles formatting, filtering, transforming, and aggregating JSON data — all in a single command.

This guide walks through everything from basic filters to real-world scripting examples, with ready-to-run commands throughout.

What is jq?

jq is a lightweight, command-line JSON processor written in C. It ships as a single binary with zero external dependencies. The source code is available on GitHub under the MIT license.

Key features:

  • Pretty-printing — format minified JSON into readable output
  • Field extraction — pull out exactly the values you need
  • Filtering — select data matching specific conditions
  • Transformation — reshape data structures and compute aggregates
  • Pipelining — chain multiple operations into a single expression

Just as sed and awk are the go-to tools for text processing, jq is the standard for JSON. Combined with curl, it's an essential part of any modern CLI toolkit.

Installation

powershell
# winget (recommended)
winget install jqlang.jq

# If using WSL, install via Linux package manager instead
# sudo apt install jq

Verify the installation:

bash
jq --version
text
jq-1.8.1

Basic usage

Pretty-print JSON

The simplest filter is . — it takes the input and pretty-prints it.

bash
echo '{"name":"test","value":42,"active":true}' | jq '.'
json
{
  "name": "test",
  "value": 42,
  "active": true
}

To read from a file, pass the filename as an argument:

bash
jq '.' data.json

Extract fields

Use dot notation to access specific fields:

bash
echo '{"name":"jq","version":"1.8.1"}' | jq '.name'
text
"jq"

For raw string output without quotes, use -r (raw output):

bash
echo '{"name":"jq","version":"1.8.1"}' | jq -r '.name'
text
jq

Nested fields

Chain dots to access deeply nested values:

bash
echo '{"data":{"users":[{"name":"Alice","age":30}]}}' | jq '.data.users[0].name'
text
"Alice"

Array operations

bash
# First element
echo '[10,20,30]' | jq '.[0]'

# Last element
echo '[10,20,30]' | jq '.[-1]'

# Array length
echo '[10,20,30]' | jq 'length'

# Slice (index 1 to 2)
echo '[10,20,30,40]' | jq '.[1:3]'

Output formatting

bash
# Compact output (single line)
echo '{"name":"test","value":42}' | jq -c '.'

# Tab indentation
echo '{"name":"test","value":42}' | jq --tab '.'

# Sorted keys
echo '{"b":2,"a":1}' | jq -S '.'

Filters and pipes

The real power of jq lies in combining filters. Use the pipe operator | to chain operations together.

Array iterator

[] expands each element of an array:

bash
echo '{"users":[{"name":"Alice"},{"name":"Bob"},{"name":"Charlie"}]}' \
  | jq '.users[].name'
text
"Alice"
"Bob"
"Charlie"

select — conditional filtering

select() keeps only elements matching a condition:

bash
echo '{"users":[{"name":"Alice","age":30},{"name":"Bob","age":25},{"name":"Charlie","age":35}]}' \
  | jq '.users[] | select(.age > 28)'
json
{
  "name": "Alice",
  "age": 30
}
{
  "name": "Charlie",
  "age": 35
}

map — transform arrays

map() applies a filter to each element and returns a new array:

bash
echo '{"users":[{"name":"Alice","age":30},{"name":"Bob","age":25}]}' \
  | jq '.users | map(.name)'
json
[
  "Alice",
  "Bob"
]

Object construction

Build new objects with only the fields you need:

bash
echo '{"users":[{"name":"Alice","age":30,"email":"alice@example.com","role":"admin"}]}' \
  | jq '.users[] | {name: .name, email: .email}'
json
{
  "name": "Alice",
  "email": "alice@example.com"
}

String interpolation

Use \() to embed values in strings. Combine with -r for clean output:

bash
echo '{"users":[{"name":"Alice","email":"alice@example.com"},{"name":"Bob","email":"bob@example.com"}]}' \
  | jq -r '.users[] | "\(.name): \(.email)"'
text
Alice: alice@example.com
Bob: bob@example.com

Conditional expressions

bash
echo '{"status":"ok","data":"hello"}' \
  | jq 'if .status == "ok" then .data else "error: \(.status)" end'
text
"hello"

Sorting, grouping, and aggregation

bash
# Sort by field
echo '[{"name":"Charlie","age":35},{"name":"Alice","age":30},{"name":"Bob","age":25}]' \
  | jq 'sort_by(.age)'

# Unique values
echo '["apple","banana","apple","cherry","banana"]' | jq 'unique'

# Sum
echo '[10,20,30,40]' | jq 'add'

# Average
echo '[10,20,30,40]' | jq 'add / length'

keys and has

bash
# List all keys
echo '{"name":"test","version":"1.0","license":"MIT"}' | jq 'keys'

# Check if a key exists
echo '{"name":"test"}' | jq 'has("name")'

Real-world use cases

Processing API responses

Combining jq with curl to extract data from API responses is one of the most common use cases.

bash
curl -s https://api.github.com/repos/jqlang/jq \
  | jq '{name: .name, stars: .stargazers_count, forks: .forks_count, license: .license.spdx_id}'
json
{
  "name": "jq",
  "stars": 31000,
  "forks": 1600,
  "license": "MIT"
}

Combining results across paginated API responses:

bash
for page in 1 2 3; do
  curl -s "https://api.github.com/users/octocat/repos?per_page=100&page=${page}"
done | jq -s 'flatten | map({name: .name, stars: .stargazers_count}) | sort_by(.stars) | reverse'

Parsing package.json

Quickly inspect project dependencies without opening the file:

bash
# List dependency names
jq '.dependencies | keys' package.json
json
[
  "next",
  "react",
  "react-dom"
]
bash
# Show packages with versions
jq -r '.dependencies | to_entries[] | "\(.key)@\(.value)"' package.json
text
next@^16.0.0
react@^19.0.0
react-dom@^19.0.0
bash
# Count dependencies vs devDependencies
jq '{deps: (.dependencies | length), devDeps: (.devDependencies | length)}' package.json

JSON log analysis

Process JSON Lines format (one JSON object per line) log files:

bash
# Extract error logs with timestamps
cat app.log \
  | jq -r 'select(.level == "error") | "\(.timestamp) [\(.level)] \(.message)"'
text
2026-03-08T10:15:30Z [error] Database connection timeout
2026-03-08T10:18:45Z [error] Failed to process request
bash
# Count entries by log level
cat app.log \
  | jq -s 'group_by(.level) | map({level: .[0].level, count: length})'
json
[
  { "level": "error", "count": 5 },
  { "level": "info", "count": 142 },
  { "level": "warn", "count": 23 }
]

Editing config files

Modify JSON configuration files from the command line:

bash
# Update a value
jq '.database.port = 5433' config.json > tmp.json && mv tmp.json config.json
bash
# Add a field
jq '.database.ssl = true' config.json > tmp.json && mv tmp.json config.json
bash
# Delete a field
jq 'del(.debug)' config.json > tmp.json && mv tmp.json config.json
bash
# Merge two JSON files (override takes precedence)
jq -s '.[0] * .[1]' base.json override.json > merged.json

Advanced techniques

Passing shell variables safely (--arg / --argjson)

You'll frequently need to use shell variables inside jq filters. Direct string interpolation is fragile — use --arg (strings) and --argjson (numbers, arrays, objects) instead.

bash
# --arg: pass as a string
username="Alice"
echo '{"users":[{"name":"Alice","age":30},{"name":"Bob","age":25}]}' \
  | jq --arg name "${username}" '.users[] | select(.name == $name)'
json
{
  "name": "Alice",
  "age": 30
}
bash
# --argjson: pass as a number or JSON value (not quoted)
min_age=28
echo '{"users":[{"name":"Alice","age":30},{"name":"Bob","age":25}]}' \
  | jq --argjson min "${min_age}" '.users[] | select(.age >= $min)'
bash
# Dynamically build a JSON object
key="version"
value="2.0"
jq -n --arg k "${key}" --arg v "${value}" '{($k): $v}'
json
{
  "version": "2.0"
}

--arg always passes the value as a string. If you need numeric comparison, use --argjson. Passing "28" via --arg won't match .age (number 28) in a comparison like select(.age >= $min) because the types differ.

Error handling (try-catch / ? operator)

Essential when processing incomplete JSON or fields that may not exist.

bash
# ? operator: suppress errors and continue (shorthand for try)
echo 'null' | jq '.foo?'

.foo? won't raise a field access error on null — it returns null silently. Useful when data structures vary in a pipeline.

bash
# // operator: return a fallback when the value is null or false
echo '{"a":1} {"b":2} {"a":3}' | jq '.a // empty'
text
1
3
bash
# try-catch: return a fallback value on error
echo '{"data":"not-a-number"}' \
  | jq 'try (.data | tonumber) catch "parse error"'
text
"parse error"
bash
# Handle arrays where some elements are missing fields
echo '[{"name":"Alice","email":"a@example.com"},{"name":"Bob"}]' \
  | jq '.[] | {name, email: (.email // "N/A")}'
json
{
  "name": "Alice",
  "email": "a@example.com"
}
{
  "name": "Bob",
  "email": "N/A"
}

Format strings (@base64 / @uri / @html / @tsv)

jq has built-in formatters for converting output into specific formats.

bash
# @base64: Base64 encode
echo '{"token":"hello:world"}' | jq -r '.token | @base64'
text
aGVsbG86d29ybGQ=
bash
# @base64d: Base64 decode
echo '"aGVsbG86d29ybGQ="' | jq -r '@base64d'
text
hello:world
bash
# @uri: URL encode (useful for building query parameters)
echo '{"q":"jq filter examples","lang":"ja"}' \
  | jq -r '"https://example.com/search?q=\(.q | @uri)&lang=\(.lang)"'
text
https://example.com/search?q=jq%20filter%20examples&lang=ja
bash
# @tsv: convert arrays to TSV (tab-separated values)
echo '[{"name":"Alice","age":30},{"name":"Bob","age":25}]' \
  | jq -r '.[] | [.name, .age] | @tsv'
text
Alice	30
Bob	25

Streaming large JSON files (--stream)

Processing a JSON file of hundreds of megabytes with jq '.' loads everything into memory. --stream processes it incrementally as path-value pairs.

bash
# --stream: incremental processing (saves memory)
echo '{"users":[{"name":"Alice"},{"name":"Bob"}]}' \
  | jq --stream 'select(.[0][-1] == "name") | .[1]'
text
"Alice"
"Bob"
bash
# Extract a specific key from a huge file without loading it all
jq --stream 'select(.[0][0] == "error_count") | .[1]' huge_report.json

Stream mode uses a different filter syntax than normal mode, so there's a learning curve. But when regular mode runs out of memory, this is your fallback.

Accessing environment variables ($ENV / env)

Useful in CI/CD pipelines and scripts where you need to embed environment variables into JSON.

bash
export APP_VERSION="1.5.0"
export APP_ENV="production"

# $ENV: access specific environment variables
jq -n '{version: $ENV.APP_VERSION, env: $ENV.APP_ENV}'
json
{
  "version": "1.5.0",
  "env": "production"
}
bash
# env: access all environment variables as an object
jq -n 'env | keys | map(select(startswith("APP_")))'
json
[
  "APP_ENV",
  "APP_VERSION"
]

$ENV vs env: $ENV.KEY accesses a specific key directly. env returns all environment variables as an object, so you can filter with keys or select.

Scripting examples

Integrating jq into shell scripts makes automated JSON processing straightforward. Combine it with xargs to pipe extracted values into other commands in parallel.

Export GitHub repos to CSV

bash
#!/bin/bash
# github-repos-to-csv.sh
# Export a user's public repositories to CSV format

USERNAME="${1:?Usage: $0 <github-username>}"
OUTPUT="repos.csv"

echo "name,stars,forks,language,updated" > "${OUTPUT}"

page=1
while true; do
  response=$(curl -s "https://api.github.com/users/${USERNAME}/repos?per_page=100&page=${page}")

  count=$(echo "${response}" | jq 'length')
  if [ "${count}" -eq 0 ]; then
    break
  fi

  echo "${response}" \
    | jq -r '.[] | [.name, .stargazers_count, .forks_count, (.language // "N/A"), .updated_at[:10]] | @csv' \
    >> "${OUTPUT}"

  page=$((page + 1))
done

total=$(tail -n +2 "${OUTPUT}" | wc -l)
echo "Exported ${total} repositories to ${OUTPUT}"

Key points:

  • The @csv filter handles CSV formatting automatically, including quoting and escaping
  • // "N/A" is the alternative operator — it returns a default when a value is null or false
  • Pagination is handled with a simple loop until an empty response

Merge and aggregate JSON reports

bash
#!/bin/bash
# merge-json-reports.sh
# Combine JSON report files and generate summary statistics

REPORT_DIR="${1:?Usage: $0 <report-directory>}"

if [ ! -d "${REPORT_DIR}" ]; then
  echo "Error: Directory '${REPORT_DIR}' not found" >&2
  exit 1
fi

file_count=$(find "${REPORT_DIR}" -name "*.json" -type f | wc -l)
if [ "${file_count}" -eq 0 ]; then
  echo "Error: No JSON files found in '${REPORT_DIR}'" >&2
  exit 1
fi

# Merge all JSON files and compute summary
find "${REPORT_DIR}" -name "*.json" -type f -exec cat {} + \
  | jq -s '{
    total_files: length,
    total_records: (map(.records // 0) | add),
    total_errors: (map(.errors // 0) | add),
    avg_duration_ms: (map(.duration_ms // 0) | add / length | floor),
    statuses: (group_by(.status) | map({status: .[0].status, count: length})),
    date_range: {
      earliest: (map(.timestamp) | sort | first),
      latest: (map(.timestamp) | sort | last)
    }
  }'

echo ""
echo "Processed ${file_count} files from ${REPORT_DIR}"

Key points:

  • -s (slurp) combines multiple JSON objects into a single array
  • group_by and map together produce per-status counts
  • // 0 provides fallback values for missing fields
  • floor truncates decimal places in the average

Security note

When processing untrusted JSON, keep these points in mind:

  • Keep jq updated — parser vulnerabilities are discovered periodically. If jq --version shows anything below 1.8.1, update immediately
  • Limit input size — processing very large JSON files can consume significant memory. Use head -c upstream in the pipeline to cap input size, or consider the streaming parser (--stream) for large datasets
  • Guard against shell injection — never pipe jq output directly into eval or bash -c. Always capture values in quoted variables
bash
# Dangerous (never do this)
eval $(curl -s https://example.com/config.json | jq -r '.command')

# Safe (capture in a variable)
value=$(curl -s https://example.com/config.json | jq -r '.setting')
echo "Setting: ${value}"

FAQ

What's the difference between jq and yq?

jq is a JSON-only processor. yq handles YAML, XML, and TOML as well, using a jq-like syntax internally. For JSON-only work, jq is faster and lighter. If you need to process YAML or multiple formats, yq is worth considering.

Can I use jq without installing it?

Yes — jq play is an online playground where you can test filters in your browser. It's great for learning before installing or debugging complex expressions. Just don't paste sensitive data into it.

jq runs out of memory on large JSON files. What can I do?

Use the --stream option to process JSON incrementally as path-value pairs, which significantly reduces memory usage. The filter syntax differs from normal mode, but it's the go-to solution for files of hundreds of megabytes or more.

How do I convert jq output to CSV?

Use the @csv formatter. Build an array and pipe it to @csv: jq -r '.[] | [.name, .age] | @csv'. It handles quoting and escaping automatically. For TSV output, use @tsv instead.

How do I use shell variables inside jq filters?

Use --arg (for strings) or --argjson (for numbers, arrays, objects). Direct string interpolation is fragile and error-prone. For numeric comparisons, you must use --argjson — otherwise the type mismatch between a string and a number will cause unexpected results.

How do I handle null or missing fields safely in jq?

The // operator (alternative operator) provides a fallback value: .email // "N/A". The ? operator suppresses errors: .foo?. For more control, use try-catch: try (.data | tonumber) catch "parse error".

Should I use jq or Python's json module?

For one-liners and shell script processing, jq is significantly faster. For complex logic (heavy branching, database integration, etc.), Python offers better readability. In CI/CD pipelines, jq is preferred for its minimal dependencies.

Wrapping Up

jq is the essential tool for JSON processing on the command line.

  • Basics — . for pretty-printing, .field for extraction, -r for raw string output
  • Filters — select for conditional filtering, map for transformation, pipes for chaining
  • Real-world — API response processing, config file editing, log analysis
  • Scripting — @csv conversion, -s for merging, shell script integration

Combine curl for data retrieval, jq for JSON processing, and sed & awk for text formatting — and you'll have a powerful data processing pipeline right in your terminal. Start with curl ... | jq '.' and build from there.

Related articles: