wuttasync.importing.csv

Importing from CSV

class wuttasync.importing.csv.FromCsv(config, **kwargs)

Base class for importer/exporter using CSV file as data source.

Note that this assumes a particular “format” for the CSV files. If your needs deviate you should override more methods, e.g. open_input_file().

The default logic assumes CSV file is mostly “standard” - e.g. comma-delimited, UTF-8-encoded etc. But it also assumes the first line/row in the file contains column headers, and all subsequent lines are data rows.

input_reader

While the input file is open, this will reference a csv.DictReader instance.

csv_encoding = 'utf_8'

Encoding used by the CSV input file.

You can specify an override if needed when calling process_data().

get_input_file_name()

By default this returns the importer/exporter model name plus CSV file extension, e.g. Widget.csv

It calls get_model_title() to obtain the model name.

get_source_objects()

This returns a list of data records “as-is” from the CSV source file (via input_reader).

Since this uses csv.DictReader by default, each record will be a dict with key/value for each column in the file.

open_input_file()

Open the input file for reading, using a CSV parser.

This tracks the file handle via input_file and the CSV reader via input_reader.

It also updates the effective fields list per the following logic:

First get the current effective field list, e.g. as defined by the class and/or from caller params. Then read the column header list from CSV file, and discard any which are not found in the first list. The result becomes the new effective field list.

class wuttasync.importing.csv.FromCsvToSqlalchemyHandlerMixin

Mixin class for CSV → SQLAlchemy ORM import handlers.

This knows how to dynamically generate importer classes to target the particular ORM involved. Such classes will inherit from FromCsvToSqlalchemyMixin, in addition to whatever FromImporterBase and ToImporterBase reference.

This all happens within define_importers().

FromImporterBase

This must be set to a valid base class for the CSV source side. Default is FromCsv which should typically be fine; you can change if needed.

alias of FromCsv

ToImporterBase = None

For a handler to use this mixin, this must be set to a valid base class for the ORM target side. The define_importers() logic will use this as base class when dynamically generating new importer/exporter classes.

define_importers()

This mixin overrides typical (manual) importer definition, and instead dynamically generates a set of importers, e.g. one per table in the target DB.

It does this based on the target model, as returned by get_target_model(). It calls make_importer_factory() for each model class found.

get_target_model()

This should return the app model or a similar module containing data model classes for the target side.

The target model is used to dynamically generate a set of importers (e.g. one per table in the target DB) which can use CSV file as data source. See also define_importers().

Subclass must override this if needed; default behavior is not implemented.

make_importer_factory(model_class, name)

Generate and return a new importer class, targeting the given data model class.

The newly-created class will inherit from:

• FromCsvToSqlalchemyMixin

• FromImporterBase

• ToImporterBase

And coercers will be set on the class, to the result of make_coercers().

Parameters:

• model_class – A data model class.

• name – The “model name” for the importer/exporter. New class name will be based on this, so e.g. Widget model name becomes WidgetImporter class name.

Returns:

The new class, meant to process import/export targeting the given data model.

class wuttasync.importing.csv.FromCsvToSqlalchemyMixin

Mixin class for CSV → SQLAlchemy ORM importers.

Such importers are generated automatically by FromCsvToSqlalchemyHandlerMixin, so you won’t typically reference this mixin class directly.

This mixin adds data type coercion for each field value read from the CSV file; see normalize_source_object().

coercers

Dict of coercer functions, keyed by field name. This is an empty dict by default; however typical usage does not require you to set it, as it’s auto-provided from make_coercers().

Each coercer function should accept a single value, and return the coerced value, e.g.:

def coerce_int(val):
    return int(val)

normalize_source_object(obj)

Normalize a source record from CSV input file. See also the parent docs for wuttasync.importing.base.Importer.normalize_source_object().

This will invoke the appropriate coercer function for each field, according to coercers.

Parameters:

obj – Raw data record (dict) from CSV reader.

Returns:

Final data dict for the record.

class wuttasync.importing.csv.FromCsvToWutta(config, **kwargs)

Handler for CSV → Wutta app database import.

This uses FromCsvToSqlalchemyHandlerMixin for most of the heavy lifting.

ToImporterBase

alias of ToWutta

wuttasync.importing.csv.make_coercer(attr)

Returns a coercer function suitable for use by normalize_source_object().

This is typically called from make_coercers(). The resulting function will coerce values to the data type defined by the given attribute, e.g.:

def coerce_int(val):
    return int(val)

Parameters:

attr – SQLAlchemy mapped attribute, e.g. wuttjamaican.db.model.upgrades.Upgrade.exit_code.

Returns:

Coercer function based on mapped attribute data type.

wuttasync.importing.csv.make_coercers(model_class)

Returns a dict of coercer functions for use by normalize_source_object().

This is called automatically by make_importer_factory(), in which case the result is assigned to coercers on the importer class.

It will iterate over all mapped fields, and call make_coercer() for each.

Parameters:

model_class – SQLAlchemy mapped class, e.g. wuttjamaican.db.model.base.Person.

Returns:

Dict of coercer functions, keyed by field name.