💻 dawid.dev

Exporting and Merging Text Files for AI Processing

2 min read

📌 Purpose

This script recursively scans all files in a specified directory and saves their contents to an output file. Binary files are skipped, and a note [SKIP] Binary file: <path> is added instead. This allows for easy merging of multiple text files into a single document that can be uploaded to AI tools like ChatGPT, Claude, DeepSeek, etc.

🖥️ Example Script

#!/bin/bash
 
echo "Exporting text file contents..."
 
$output_file="merged_output.txt"
find . -type f -print0 | while IFS= read -r -d '' file; do
    echo -e "\n========== START: $file ==========" >> "$output_file"
    if file "$file" | grep -q text; then
        cat "$file" >> "$output_file"
    else
        echo "[SKIP] Binary file: $file" >> "$output_file"
    fi
    echo -e "\n========== END: $file ==========" >> "$output_file"
done
 
echo "Export completed: $output_file"

📜 Script Functionality

  1. Recursive Directory Search

    • The find command locates all files and outputs a null-separated list (-print0 ensures compatibility with filenames containing special characters).

  2. File Type Check

    • Each file is analyzed with file "$file" to determine whether it is a text file.

  3. Saving Content to Output File

    • If a file is text-based, its content is appended to the output file.
    • If a file is binary, a placeholder note is added instead.

  4. Formatting the Output

    • Each file is wrapped between ========== START: <file path> ========== and ========== END: <file path> ==========, ensuring clear sectioning.

📂 Example Output Format

========== START: path/to/file1.txt ==========
Sample text file content.
========== END: path/to/file1.txt ==========
 
========== START: path/to/file2.log ==========
Log file content here.
========== END: path/to/file2.log ==========
 
========== START: path/to/image.png ==========
[SKIP] Binary file: path/to/image.png
========== END: path/to/image.png ==========

📊 Use Cases

  • Merging Files for AI Processing – Preparing large text datasets to be uploaded into AI tools such as ChatGPT, Claude, DeepSeek, and others.
  • Project Archiving – Capturing the full content of text files in a single file.
  • Source Code Analysis – Reviewing and searching through all files at once.
  • Identifying Binary Files – Detecting non-text files for alternative processing methods.

Graph View