Roland Ayala

Roland Ayala

My head's in the cloud

Introduction

Importpix is an open-source, cross-platform command-line utility that imports photos into user-defined libraries, with the following features and benefits:

  • Organizes photos and images using Exif metadata
  • Handles duplicate detection
  • Handles XMP sidecar files
  • Supports unlimited, user-defined libraries
  • Stateless design with no database or service required
  • Photos and images are not modified
  • Works on Linux, MacOS, and Windows

For system requirements and installation instructions see the Installation section.

Getting Started

Follow the steps below to get you up and running with importpix. However, before committing yourself to importing all of your photos it’s recommended you take some time to more fully familiarize yourself with importpix.

Install Importpix

Follow the instructions for your OS at the bottom of this page to install importpix.

Make Library

Create an empty directory anywhere on your system (or accessible to your system if a network share). This will be the location of your first library. For example:

md /users/murphy/pictures/library

Substitute actual path for the one shown in the example. Next, make the new library by entering the following command:

importpix --make library /users/murphy/pictures/library

Accept the default values for layout and filename template, and confirm setting at the confirmation prompt by entering y. If all went well, you should see confirmation that your library has been successfully created. If you inspect the directory contents it may appear empty, but there is a hidden file present at the root that contains the library configuration information.

Note: The target directory must be empty prior to attempting make library operation else the operation will fail.

Import Photos

Enter the following command to import photos into the library created:

importpix --library /users/murphy/pictures/library /Volumes/DCIM

In this example, /Volumes/DCIM is the source directory containing the photos to be imported. To include photos included in subdirectories under the source directory, use the -r option as follows:

importpix -r --library /users/murphy/pictures/library /Volumes/DCIM

Optimize Workflow

A configuration file can be used to simplify user input, especially you intend to import into the same library most or all of the time. Instead of needing to specify the library path for each import operation, this and other command-line arguments can be read from a configuration file. If you’re working with multiple libraries, you might want to have a config files for each library. The --config tag specifies the config file to use, but this is just trading off one command-line arg for another — albeit a little more user friendly and less error prone than having to enter a library path.

A default config file is a specially named config file located in the user’s home directory or current directory. If present, the default config file is automatically loaded without needing to use the --config option tag. This can go along way towards simplifying user input, especially if only working with a single library. For example, instead of this:

importpix -r --library /users/murphy/pictures/library /Volumes/DCIM

A default config file can be use to simplify input to this:

importpix /Volumes/DCIM

Assuming you want a default config file, it’s recommended you create it your home directory so you can run importpix from any directory for the settings to be applied.

Make Default Config — Linux / MacOS

cd ~
importpix --library /users/murphy/pictures/library --make conifg

This command creates a hidden file named .importpix in your home directory, which is the default config file.

Make Default Config — Windows

cd /D %HOMEDRIVE%%HOMEPATH%
importpix --library c:\users\murphy\pictures --make conifg

This command creates a hidden file named importpix.cfg in your home directory, which is the default config file.

Usage

Importpix utilizes a command-line interface and is executed from the the terminal or command prompt window. The command operations are categorized as follows:

  • Import command
  • Library commands
  • Other commands

Note: A library must be available and accessible to the system before attempting an import operation. See library commands for details on how to create a library.

Import Command

importpix [-OPTIONS] [--TAGS] <source>

<Required arguments>:
   source - Source path containing photos to import

[OPTIONS]:
   e - Keywords exclusive
   l - Enable file logging
   r - Import files in source subdirectories
   x - Copy sidecar files
   y - Suppress confirmation prompt

[TAGS]:
  config   - Path to configuration file
  keywords - Comma separated list of keywords to write to image (†)
  library  - Path to library
  minsize  - Exclude images below minsize (bytes) from being imported

IMPORTANT: TAGS ending with the dagger symbol (†) require that the image(s) be modified. All images included in the import collection are affected. It is highly recommended that you keep a backup of original files when used.

Source

Specifies the directory path containing photos to import. The path must be enclosed in quotes if it contains spaces.

MacOS examples:

   importpix /Volumes/DCIM
   importpix "/Volumes/Nikon D500"
   importpix /Users/roland/Pictures

Windows examples:

  importpix E:\DCIM
  importpix C:\Users\Roland\Pictures

Note: These examples assume a library target has been specified in default config file. If not, the library target must be specified using the --library tag or in an alternate config file specified using the --config tag.

Import Options

Options can be set individually or bundled together. For example, the following lines are equivalent:

importpix -r -l -x /Volumes/DCIM
importpix -rlx /Volumes/DCIM

[-e] Option

Exclusive keywords.

This option only applies if the keywords tag is used to set keywords on the files being imported. By default, any previously set keywords are preserved for each image.

By using this option, any previously set keywords are excluded and the resultant property includes only the keyword or keywords specified by the keywords tag.

[-l] Option

Log output to file.

All output is logged to a file under the _logs directory in the library targeted by the import operation. Log file are organized by month and year, and named using a Julian timestamp coinciding with the start of import operation.

[-r] Option

Include all subdirectories under source in search for file assets to import.

[-x] Option

Include sidecar files (.XMP) in payload, if present. In the following example, IMG_0001.XMP is imported if the -x option is set.

IMG_0001.NEF
IMG_0001.XMP

Internal filename references to the photo or image in the XMP file are updated to reflect the imported filename.

[-y] Option

Disable import confirmation prompt.

Import Tags

[–config <cfgfile>] Tag

The config tag specifies the name of the config file to use, as illustrated in the following example:

importpix --config landscapes.cfg /Volumes/D500/DCIM

To further simplify user input, Importpix looks for a default configuration file named .importpix in the following search order:

  • Bin directory
  • Current directory
  • Home folder

Note: On Windows systems, the name of the default configuration file is importpix.cfg.

[–keywords <keywords>] Tag

The keywords tag specifies the keyword(s) to write to all files included in the import collection. The value can be a single keyword or a list. To write multiple keywords use a comma separated list. If the value contains spaces then it must be contained within quotes.

importpix --keywords=mammal
importpix --keywords=mammal,canine,dog
importpix --keywords="mamal, canine, dog"
importpix --keywords="mamal, canine, dog, rex"
importpix --keywords="mamal, canine, dog, rex, memorial day"

In the examples above, the 2nd and 3rd values are equivalent. Importpix will trim any leading or trailing whitespace from the keywords. Also note that the last three examples have the keywords encapsulated between quotation marks, as require because the values contain whitespace between words.

Any previously set keywords for each image file are preserved unless the keywords exclusive option (-e) is used, in which case any previously set values are lost. Importpix treats keywords as case-insensitive when evaluating if a keyword is previously existing or not. If previously set, the keyword will be update to match the case specified using the --keywords tag.

IMPORTANT: Using the keyword tag modifies all images in the import collection. It is highly recommended that a backup of original files be kept when used.

[–library <libpath>] Tag

The library tag specifies the name of the library to receive files during an import operation, as illustrated in the following example:

importpix --library /users/murphy/photos /Volume/D500/DCIM

In this example the photos in /Volume/D500/DCIM are imported into the library located at/users/murphy/photos.

[–minsize <bytes>] Tag

The minsize tag establishes the smallest file size (bytes) for photos and images to be imported.

Setting this value to 0 disables the feature, and has the same effect as not including the tag.

Library Commands

Display library information:

Prints library properties to screen.

importpix --info library [libpath]

[Optional argument]:
   libpath - Library path.

Note: If library path is not specified then Importpix uses the library from the default config file, if available.

Make a new library:

Makes a new library from an empty directory.

importpix [-OPTIONS] [--TAGS] --make library <libpath>

<Required arguments>:
   libpath - Path to directory to make into a library

[OPTIONS]:
  y - Suppress confirmation prompt
  o - Disable online check for newer version  

[TAGS]:
  ltempl - Layout template for new library
  ftempl - Filename template for new library
  title  - Library title
  author - Author name

[–ltempl <template>] Tag

The ltempl tag sets the layout template for the library. If not set then the user is prompted to select from list of standard template values, or a custom value set using a config file.

[–ftempl <template>] Tag

The ftempl tag sets the filename template for the library. If not set then the user is prompted to select from list of standard template values, or a custom value set using a config file.

[–title <string>] Tag

The title tag sets the optional library title property. If the string value contains spaces then it needs to be enclosed in quotes.

[–author <string>] Tag

The author tag sets the optional library author property. If the string value contains spaces then it needs to be enclosed in quotes.

Note: If argument is not provided then importpix defaults to current user login name.

Other Commands

Make a config file:

Makes a new config file, which simplifies user input when executing import and make library commands; see the config files for details.

importpix [-OPTIONS] [--TAGS] --make config [path]

[OPTIONS]:
   l - Enable file logging.
   r - Import files in source subdirectories.
   o - Disable online check for newer version
   x - Copy sidecar files.

[TAGS]:
  author -  Author name
  ftempl -  Filename template for new library
  library - Library path
  ltempl -  Layout template for new library
  minsize - Exclude images below minsize (bytes) from being imported

Makes a new config file at the path specified. If path is not specified then a default named file is created in current directory.

Platform Default Config
Linux .importpix
MacOS .importpix
Windows importpix.cfg

The [-OPTIONS] and [--TAGS] map to those used during import and make library operations and, if set, values are saved to the config file and applied during the operation without the user needing to re-enter them from the command line. For additional information about each option and tag, refer to the respective sections for each operation.

Note: For config file values to be applied, the user must specify the file path the to config file using the --config tag, or have a default config present in the search order.

Config Files

Config files offer a convenient method of specifying command-line arguments without needing to re-enter them each time an operation is executed. If a config file used, any command-line arguments supplied take precedence over their respective config file values. Config files can also be used simplify workflow across multiple libraries, where each config file maps to a specific library.

The following three (3) examples illustrate how config files can be used to simplify user input:

Import operation without config file:

importpix -rlx --library /users/murphy/photos /Volumes/DCIM

Import operation with config file:

importpix --conifg myconfig.cfg /Volumes/DCIM

Import operation with default config file:

importpix /Volumes/DCIM

In these example, the config file properties can be set such that all three commands produce the same output.

For operations having a --config tag option, importpix searches for a default config file in the following search path order:

  • Bin directory (same directory as importpix)
  • Current directory
  • Current user’s home directory

The following codeblock illustrates a sample config file, which is intended to be end-user editable.

#                           IMPORTPIX CONFIG FILE
#
# This is an end-user editable config file for the importpix image importer
# utility. The values contained herein can be used to override default
# configuration values and specify command line option defaults.
#
# A hashtag (#) character at beginning of a line denotes a comment, which
# results in the line being ignored.
#
# Replace tilde (~) characters with property values to override defaults. A
# A property value can also be unset by replacing it with tilde, or by
# commenting out the line.
#
# See importpix documentation for the complete list of settable values.
#

# Schema version. !! IMPORTANT: Do not change version value !!
version: 1.00

# Author name to use when making new libraries
author: ~

# Custom filename template to include in --make library options
ftempl: ~

# Enable logging output to file
l: ~

# Path to target library to receive imported files
library: "/users/murphy/pictures/library"

# Custom layout template to include in --make library options
ltempl: ~

# Minimum size (in bytes) required for images to be imported
minsize: ~

# Search source subdirectories for file assets to import
r: 1

# Include XMP sidecar files
x: 1

In this config file example, the library path is set so that it does not need to entered from the command-line using the --library option each time importpix is run. Similarly, the recursive search and import sidecar files options are set.

Note: When specifying library value on Windows systems, use double-backslashes in path string. For example: "D:\\\\Pictures\\Library".

Logs

If the logging -l option is enabled, all output written to a log file under the -logs directory located at the root of the library targeted by the import operation.

Concepts

Importpix organizes photos by copying them from one location to another. In doing so, it uses the Exif metadata embedded in each photo to determine the destination path and filename. User defined templates establish which Exif properties are used to generate the path and filename values.

Libraries

A library is simply a designated directory accessible to the system Importpix is running – it can be local storage or accessible via a network share. Each library has at its root a file named .pixlib, which contains the configuration properties for the library as illustrated here:

# !!!!!!!!!!!!!!!!!!!!!!! DO NOT EDIT THIS FILE !!!!!!!!!!!!!!!!!!!!!!!!
# Changing filename and layout templates values after a library has been
# created can result in unpredictable behavior. Instead, create a new
# library with desired template values and re-import from this folder
# if desire is to change how imported images are named and organized.
{
   "formatting" : {
      "path_case" : 2,
      "type_case" : 2,
      "base_case" : 2
   },
   "templates" : {
      "filename" : "P{[DATE1]}-{[HHMMSS]}-{[INDEX]}",
      "layout" : "{[YYYY]}#{[MM]}"
   },
   "created_on" : "2018-03-29T23:16:13",
   "package_info" : {
      "author" : "Roland Ayala",
      "description" : "Image Importer Library",
      "version" : "1.00",
      "license" : "Artistic License 2.0"
   }
}

The layout and filename template property values respectively determine how photos are organized and named within the library. These values are customizable, but should not be changed after the first import has occurred. See the templates section for details.

The path_case, base_case, and type_case formatting property values determine the capitalization rules applied to the layout and filename templates, as indicated in the following table:

Value Format
0 Unmodified
1 Lower case
2 Upper case

The *_case property meanings are as follows:

Property Meaning
path_case The collection path expanded from layout template.
base_case The base filename expanded from filename template.
type_case The extension of the file asset being imported, i.e., file type.

Collections

A collection is the relative path within a library where photos are stored. The path is created by expanding the library layout template using Exif metadata properties.

Payloads

A payload is a collection of image assets capturing the same moment, where each asset is a different format or edited version of the original. For example, the following images belong to the same payload:

IMG_0001.NEF
IMG_0001.XMP
IMG_0001.JPG
IMG_0001.TIF
IMG_0001-Edit.TIF

Payloads ensure assets that belong together stay together after being imported into a library. If imported separately, these same files could resolve to different collections or filenames because not all metadata properties are guaranteed to be uniformly shared across formats.

As each asset is added to a payload, select metadata properties are reflected onto the payload. The payload remains in a consistent state and is importable so long as the reflected properties are consistent with the properties of previously added assets. If a metadata conflict occurs then the payload is flagged accordingly and it gets copied into the library’s indeterminate folder during the import operation.

Templates

Templates expand Exif metadata into the collection path and filename values for each payload. They are string values containing tokens, where each token maps to an Exif metadata property. A token is a bracketed string value enclosed in curly braces, as illustrated here:

{[TOKEN]}

Template tokens are replaced with or derived from Exif property values for each payload processed during the import process. The following lists tokens available for use when defining layout and filename templates.

Token Type † Example ‡
DATE1 Date 20180212
DATE2 Date 180212
DATE3 Date Feb 12, 2018
DATE4 Date February 12, 2018
JDN Date 2458162
YYYY Date Part 2018
YY Date Part 18
MM Date Part 02
MONTH Date Part February
MON Date Part Feb
DD Date Part 12
DAY Date Part Monday
DA Date Part Mon
HHMMSS Time 143205
HOUR Time Part 14
MINUTE Time Part 32
SECOND Time Part 05
MAKE TString Make
MODEL TString Model
SERNUM TString Serial number
INDEX TIndex 7801
FILENAME TIndex IMG_0001

† See metadata types for details behind each token is derived from Exif metadata.

‡ The example values provided are for the following Exif properties in file named IMG_0001.NEF:

Exif Property Value
DateTimeOriginal 2018:02:12 14:32:05
Make NIKON CORPORATION
Model NIKON D500
SerialNumber 3XXXXXXX
ShutterCount 7801

Layout Templates

A layout template establishes the organizational structure (or layout) of a library. As each payload is processed, Exif metadata is used to expand the template into a collection path. The layout template must begin and end with a token, and a hashtag (#) serves as a path separator.

The following table illustrates layout template examples and expanded values (collection paths) using above sample metadata:

Layout Template Collection Path
{[YYYY]}#{[MM]} 2018/02
{[YYYY]}#{[MON]} 2018/Feb
{[YYYY]}#{[MM]}#{[DD]} 2018/02/12

Note: For Windows systems, the path separator is expanded to a backslash instead for forward slash as shown.

Filename Templates

A filename template establishes the naming rules for files imported into a library. As each payload is processed, Exif metadata is used to expand the template into a filename. The filename template must end with a TIndex type token.

The following table illustrates filename template examples and expanded values (filenames) using above sample metadata:

Filename Template Filename
P{[DATE1]}-{[HHMMSS]}-{[INDEX]} P20180212-143205-007801
{[FILENAME]} IMG_0001

Metadata Types

Date

Date tokens (and their parts) are expanded using date portion the following Exif properties, in order of precedence listed:

  • DateTimeOriginal
  • CreateDate

If one of these properties is not set on the payload then it is indeterminate and gets copied to the indeterminate folder on import.

Time

Time tokens (and their parts) are expanded using the time portion of same Exif properties used for Date.

TString

TString tokens are string values are expanded from Exif metadata that have been processed to remove all non-alpha numeric characters.

TIndex

TIndex tokens are expanded using the following Exif properties and filename parsed data, in order of precedence listed:

  • ImageCount
  • ShutterCount
  • Filename

If ImageCount and ShutterCount properties are not available, an attempt is made to extract an index value from the source filename. If this image originated from a camera and filename was not renamed then the likelihood is high that an index can be parsed from the filename. Alternatively, a TIndex token can also be the source file basename.

All filename temlates must end with a TIndex token. This is to help avoid library collection filename conflicts. Time values are insufficient because timestamp values are typically seconds resolution and burst shots can yield several frames per second.

Indeterminates

If payload metadata conflicts are present or if Exif timestamp info is not set on the image, then the import payload is flagged indeterminate and its assets are copied under the _indeterminate directory located at the root of the library targeted by the import operation. The original source file names are retained, and they are copied into a subdirectories named using hash value of source directory from where they came. A file named `_source.txt contains the original unhashed value.

The intent of indeterminates is to avoid having importpix make guesses about how to import payloads when the information needed is lacking, or when payload metadata conflicts exist.

When dealing with large volumes of photos, guessing results in hiding misfiled photos amongst the others. Instead, Importpix provides a well-known, special library location that the user can visit to take corrective action. They typical scenario would involve adding required metadata (using a utility like ExifTool) or resolving the conflict, and then reattempting import.

Installation

ALPHA RELEASE

Importpix is currently alpha quality and is only available via GitHub distribution:

https://github.com/rolanday/Image-Importer

Once ready, it will also be published to cpanm and be made available as standalone binary for ease of distribution and installation.

System Requirements

Importpix is supported on the following OS platforms:

  • Linux
  • MacOS
  • Windows

Importpix is a Perl module and installation instructions vary by platform as described below for each OS. The advanced user with knowledge of Perl can alternatively choose to perform a manual install as described below.

Linux

Prerequisites

Install Perl package manager by entering the following command from a terminal window:

sudo apt install cpanminus

Installation

Open a terminal window.

Download the latest version of Importpix as a tarball into the current directory using the following command:

wget --no-check-certificate https://github.com/rolanday/Image-Importer/archive/master.tar.gz -O importpix.tar.gz

Install import pix using the following command:

sudo cpanm local importpix.tar.gz

Finally, use the following command to complete validation that the correct version is installed correctly:

importpix --version

After installation is complete importpix.tar.gz can be safely deleted using the following command:

rm importpix.tar.gz

MacOS

Prerequisites

Install homebrew package manager using the instruction provided on the homebrew site:

https://brew.sh

After homebrew installed, execute the following commands from the terminal window:

brew install cpanminus
brew install wget

Installation

Open a terminal window.

Download the latest version of Importpix as a tarball into the current directory using the following command:

wget --no-check-certificate https://github.com/rolanday/Image-Importer/archive/master.tar.gz -O importpix.tar.gz

Install import pix using the following command:

sudo cpanm local importpix.tar.gz

Finally, use the following command to complete validation that the correct version is installed correctly:

importpix --version

After installation is complete importpix.tar.gz can be safely deleted using the following command:

rm importpix.tar.gz

Windows

Prerequisites

For Windows systems, Perl is not included and ActiveState Perl version 5 or higher must be installed separately. The freely available ActivePerl Community Edition is all that’s required.

Once ActivePerl installation is complete, run the following commands:

ppm install dmake
ppm install Carp::Assert
ppm install Image::Exiftool
ppm install File::Find::Rex
ppm install Log::Log4perl
ppm install JSON::Validator
ppm install YAML

Installation

Follow the manual instructions provided in the next section.

Manual Installation

The following platform neutral instructions are for the advanced user familiar with how to install Perl modules.

Download the following file to get the latest version of Importpix:

https://github.com/rolanday/Image-Importer/archive/master.zip

Once downloaded, you’ll have a file name Image-Importer-master.zip in your download location. Unzip this file into a folder named Image-Importer-master and go to it from the terminal or command prompt window.

Enter the following commands:

perl Make.PL
make
make test
sudo make install

Note: For Windows systems, use dmake instead of make, and see Windows installation instructions for prerequisites to executing these commands.

Any missing Perl dependencies are reported and must be installed manually or by using cpanm package manager.

The make test command is optional, but helps validate that the required modules were installed correctly. Enter the following command to complete validation that the correct version is installed correctly:

importpix --version

Appendix A - Known File Extensions

Importpix imports photos and images with the following extensions:

EXT Description
3FR Hasselblad
ARW Sony Digital Camera Image
CR2 Canon RAW
CRW Canon RAW
DNG Adobe Digital Negative Image File
EIP Enhanced Image Package File
ERF Epson RAW File
FFF Hasselblad RAW
HEIC Apple HEIF Image (iOS 11 & higher)
IIQ Phase One RAW
J6I Ricoh Camera Image File
JPG Standards format
JPEG Standards format
K25 Kodak K25 Image
KDC Kodak Photo-Enhancer File
MEF Mamiya RAW Image
MFW Mamiya Camera RAW File
MOS Leaf Camera RAW File
MRW Minolta RAW Image File
NEF Nikon Electronic Format RAW Image
NRW Nikon Raw Image File
ORF Olympus RAW File
PEF Pentax Electronic File
PNG Standards format
PSD Photoshop File
RAF Fuji RAW Image File
RAW Raw Image Data File
RW2 Panasonic RAW Image
RWL Leica RAW Image
RWZ Rawzor Compressed Image
SR2 Sony RAW
SRF Sony RAW
SRW Samsung RAW
TIF Standards format
TIFF Standards format
X3F SIGMA X3F Camera RAW

Updated:

Comments

You May Also Enjoy