csv-file-downloader

README

csv-file-downloader

csv-file-downloader download files from url in CSV. And create CSV file with label and path for Google Cloud AutoML.

Installation

Install csv-file-downloader by command below,

$ go get github.com/evalphobia/csv-file-downloader

Usage

root command

$ csv-file-downloader
Commands:

  help       show help
  download   Download files from --file csv
  list       Create csv list file from --output dir

download command

$ csv-file-downloader help download
Download files from --file csv

Options:

  -h, --help           display help information
  -f, --file          *download list file --file='/path/to/dir/input.csv'
  -n, --name          *column name for filename --name='name'
  -l, --label         *column name for label --label='group'
  -u, --url           *column name for URL --url='path'
  -p, --parallel[=2]   parallel number --parallel=2
  -o, --out            outout dir --out='/path/to/dir/out'

# Save CSV file with name, label and URL.
$ cat my_file_list.csv

id,label,image_url
1,cat,http://example.com/foo.jpg
2,dog,http://example.com/bar.jpg
3,cat,https://example.com/foo2.JPG
4,human,https://example.com/baz.png?q=1
5,human,https://example.com/baz2.png


# Download files from URL in CSV.
$ csv-file-downloader download -f ./my_file_list.csv -o ./save -n "id" -l "label" -u "image_url"


# Chech downloaded files.
$ tree ./save

./save
├── cat
│   ├── 1.jpg
│   ├── 3.JPG
├── dog
│   ├── 2.jpg
└── human
    ├── 4.png
    └── 5.png

3 directories, 5 files

list command

$ csv-file-downloader help list
Create csv list file from --output dir

Options:

  -h, --help                      display help information
  -i, --input                    *image dir path --input='/path/to/image_dir'
  -o, --output[=./output.csv]    *output TSV file path --output='./output.csv'
  -t, --type[=jpg,jpeg,png,gif]   comma separate file extensions --type='jpg,jpeg,png,gif'
  -p, --prefix                    prefix for file path --prefix='gs://<your-bucket-name>'

# Create file list from given dir and save it to output CSV file.
$ csv-file-downloader list -i ./save -o result.csv -p "gs://my-bucket/test-project"


# Check saved CSV file.
$ cat result.csv

gs://my-bucket/test-project/cat/1.jpg,cat
gs://my-bucket/test-project/cat/3.JPG,cat
gs://my-bucket/test-project/dog/2.jpg,dog
gs://my-bucket/test-project/human/4.png,human
gs://my-bucket/test-project/human/5.png,human