Bash script to iterate files in directory and pattern match filenames

Posted on Feb 13, 2022

Question

I need to process a large number of files in a directory. The files can be partitioned into several groups, based upon the file names. That is to say, the file names can be pattern matchedne which 'group' they belong to. For instance, the names are like this:

  • YYYYMMDD_*_bulk_import.csv
  • YYYYMMDD_*_genstats_import.csv
  • YYYYMMDD_*allstats.csv

etc ...

Each 'group' has a different processing methodology (i.e. a different command is called for processing).

I want to write a bash script to:

  1. Iterate through all CSV files in the directory
  2. Determine which 'group' a file belongs to by pattern matching its name to known patterns (like the examples I gave above)
  3. Call a command based on the determined grouping.

I am running on Ubuntu 10.0.4. I am new to bash, and would appreciate skeleton code snippet that will help me get started in writing this script.

Answer

The easiest way is probably just to iterate each group separately. This side-steps the parsing issue entirely.

DIRECTORY=.

for i in $DIRECTORY/YYYYMMDD_*_bulk_import.csv; do # Process $i done

for i in $DIRECTORY/YYYYMMDD_*_genstats_import.csv; do # Process $i done

for i in $DIRECTORY/YYYYMMDD_*allstats.csv; do # Process $i done

Set DIRECTORY to whatever directory you want to search. The default . will search the current working directory.