File Management Scripts

The two primary scripts used to automate tasks are:

1. import-wsprspots.sh
2. convert-to-7z.sh

As the project grows, these may be added to a larger instabble collection of utilities. But for now, they will remain in the files section under utilities.

Usage

Each Bash script has two arrays, a Year and Month. To limit the amount of data that is processed at any given time, you should remove the elements you do not want processed. For example, to process all months in 2017, your arrays look like the following

# Year Array: edit as needed to process files in smaller chunks
# '2008' '2009' '2010' '2011' '2012' '2013' '2014' '2015' '2016' '2017' '2018' '2019'
declare -a yarray=( '2017' )

# Month Array: edit as needed to process files in smaller chunks
# Note: year 2008 does not have month 01, or 02 and will be skipped.
declare -a marray=( "01" "02" "03" "04" "05" "06" "07" "08" "09" "10" "11" "12")

The same is true for using the convert-to-7z.sh script also. Just remove the years and months that you do not want processed.

Re-Compressing Files

After you've imported the raw CSV data into MongoDB, you have an option to leave the file as is, or recompress it to 7z for mat and add the stated to a seperate table in the database (filestats).

It's totally up to the user if they wish to recompress or not. However, doing so would render the import-wsprspots.sh inoperative. As time allows, we'll add either another script, or modify the existing one to allow of 7z compressed source files for import.

So, the current sequence would be to run all of the files you want in your database, them compress them; leaving only those that have yet to be added in their original *.gz form.