wuttasync.importing.handlers

Data Import / Export Handlers

class wuttasync.importing.handlers.FromFileHandler(config, **kwargs)

Handler for import/export which uses input file(s) as data source.

This handler assumes its importer/exporter classes inherit from FromFile for source parent logic.

class wuttasync.importing.handlers.FromSqlalchemyHandler(config, **kwargs)

Base class for import/export handlers using SQLAlchemy ORM (DB) as data source.

This is meant to be used with importers/exporters which inherit from FromSqlalchemy. It will set the source_session attribute when making them; cf. get_importer_kwargs().

This is the base class for FromWuttaHandler, but can be used with any database.

See also ToSqlalchemyHandler.

begin_source_transaction()

This calls make_source_session() and assigns the result to source_session.

commit_source_transaction()

This commits and closes source_session.

get_importer_kwargs(key, **kwargs)

This modifies the new importer kwargs to add:

• source_session - reference to source_session

See also docs for parent method, get_importer_kwargs().

make_source_session()

Make and return a new db session for the data source.

Default logic is not implemented; subclass must override.

Returns:

Session instance

rollback_source_transaction()

This rolls back, then closes source_session.

source_session = None

Reference to the db session for data source.

This will be None unless a transaction is running.

class wuttasync.importing.handlers.FromWuttaHandler(config, **kwargs)

Handler for import/export which uses Wutta ORM (app database) as data source.

This inherits from FromSqlalchemyHandler.

See also ToWuttaHandler.

get_source_title()

This overrides default logic to use get_title() as the default value.

Subclass can still define source_title (or generic_source_title) to customize.

See also docs for parent method: get_source_title()

make_source_session()

This calls make_session() and returns it.

source_key = 'wutta'

class wuttasync.importing.handlers.ImportHandler(config, **kwargs)

Base class for all import/export handlers.

Despite the name ImportHandler this can be used for export as well. The logic is no different on a technical level and the “export” concept is mostly only helpful to the user. The latter is important of course and to help with that we track the orientation to distinguish.

The role of the “import/export handler” (instance of this class) is to orchestrate the overall DB connections, transactions and then invoke the importer/exporter instance(s) to do the actual data assessment/transfer. Each of the latter will be an instance of (a subclass of) Importer.

property actioner

Convenience property which effectively returns the orientation as a noun - i.e. one of:

• 'importer'

• 'exporter'

See also actioning.

property actioning

Convenience property which effectively returns the orientation in progressive verb tense - i.e. one of:

• 'importing'

• 'exporting'

See also actioner.

begin_source_transaction()

Begin a transaction on the source side, if applicable.

This is normally called from begin_transaction().

begin_target_transaction()

Begin a transaction on the target side, if applicable.

This is normally called from begin_transaction().

begin_transaction()

Begin an import/export transaction, on source and/or target side as needed.

This is normally called from process_data().

Default logic will call both:

• begin_source_transaction()

• begin_target_transaction()

commit_source_transaction()

Commit the transaction on the source side, if applicable.

This is normally called from commit_transaction().

commit_target_transaction()

Commit the transaction on the target side, if applicable.

This is normally called from commit_transaction().

commit_transaction()

Commit the current import/export transaction, on source and/or target side as needed.

This is normally called from process_data().

Default logic will call both:

• commit_target_transaction()

• commit_source_transaction()

Note

By default the target transaction is committed first; this is to avoid edge case errors when the source connection times out. In such cases we want to properly cleanup the target and then if an error happens when trying to cleanup the source, it is less disruptive.

consume_kwargs(kwargs)

This method is called by process_data().

Its purpose is to give handlers a hook by which they can update internal handler state from the given kwargs, prior to running the import/export task(s).

Any kwargs which pertain only to the handler, should be removed before they are returned. But any kwargs which (also) may pertain to the importer/exporter instance, should not be removed, so they are passed along via get_importer().

Parameters:

kwargs – Dict of kwargs, “pre-consumption.” This is the same kwargs dict originally received by process_data().

Returns:

Dict of kwargs, “post-consumption.”

define_importers()

This method must “define” all importer/exporter classes available to the handler. It is called from the constructor.

This should return a dict keyed by “model name” and each value is an importer/exporter class. The end result is then assigned as importers (in the constoructor).

For instance:

return {
    'Widget': WidgetImporter,
}

Note that the model name will be displayed in various places and the caller may invoke a specific importer/exporter by this name etc. See also get_importer().

dry_run = False

Flag indicating whether data import/export should truly happen vs. dry-run only.

If true, the data transaction will be rolled back at the end; if false then it will be committed.

See also rollback_transaction() and commit_transaction().

get_importer(key, **kwargs)

Returns an importer/exporter instance corresponding to the given key.

Note that this will always create a new instance; they are not cached.

The key will be the “model name” mapped to a particular importer/exporter class and thus must be present in importers.

This method is called from process_data() but may also be used by ad-hoc callers elsewhere.

It will call get_importer_kwargs() and then construct the importer/exporter instance using those kwargs.

Parameters:

• key – Model key for desired importer/exporter.

• **kwargs – Extra/override kwargs for the importer.

Returns:

Instance of (subclass of) Importer.

get_importer_kwargs(key, **kwargs)

Returns a dict of kwargs to be used when construcing an importer/exporter with the given key. This is normally called from get_importer().

Parameters:

• key – Model key for the desired importer/exporter, e.g. 'Widget'

• **kwargs – Any kwargs we have so collected far.

Returns:

Final kwargs dict for new importer/exporter.

classmethod get_key()

Returns the import/export key for the handler. This is a combination of source_key and target_key and orientation.

For instance in the case of Wutta → CSV export, the key is: export.to_csv.from_wutta

Note that more than one handler may use the same key; but only one will be configured as the “designated” handler for that key, a la get_import_handler().

See also get_spec().

get_source_title()

Returns the display title for the data source.

By default this returns source_key, but this can be overriden by class attribute.

Base class can define generic_source_title to provide a new default:

class FromExcelHandler(ImportHandler):
    generic_source_title = "Excel File"

Subclass can define source_title to be explicit:

class FromExcelToWutta(FromExcelHandler, ToWuttaHandler):
    source_title = "My Spreadsheet"

See also get_title() and get_target_title().

classmethod get_spec()

Returns the “class spec” for the handler. This value is the same as what might be used to configure the default handler for a given key.

For instance in the case of CSV → Wutta, the default handler spec is wuttasync.importing.csv:FromCsvToWutta.

See also get_key().

get_target_title()

Returns the display title for the data target.

By default this returns target_key, but this can be overriden by class attribute.

Base class can define generic_target_title to provide a new default:

class ToExcelHandler(ImportHandler):
    generic_target_title = "Excel File"

Subclass can define target_title to be explicit:

class FromWuttaToExcel(FromWuttaHandler, ToExcelHandler):
    target_title = "My Spreadsheet"

See also get_title() and get_source_title().

get_title()

Returns the full display title for the handler, e.g. "CSV → Wutta".

Note that the orientation is not included in this title.

It calls get_source_title() and get_target_title() to construct the full title.

get_warnings_email_key()

Returns the email key to be used for sending the diff warning email.

The email key should be unique to this import/export type (really, the import/export key) but not necessarily unique to one handler.

If warnings_email_key is set, it will be used as-is.

Otherwise one is generated from get_key().

Returns:

Email key for diff warnings

importers = None

This should be a dict of all importer/exporter classes available to the handler. Keys are “model names” and each value is an importer/exporter class. For instance:

{
    'Widget': WidgetImporter,
}

This dict is defined during the handler constructor; see also define_importers().

Note that in practice, this is usually an OrderedDict so that the “sorting” of importer/exporters can be curated.

If you want an importer/exporter instance you should not use this directly but instead call get_importer().

orientation = 'import'

Orientation for the data flow. Must be a value from Orientation:

• Orientation.IMPORT (aka. 'import')

• Orientation.EXPORT (aka. 'export')

Note that the value may be displayed to the user where helpful:

print(handler.orientation.value)

See also actioning.

It’s important to understand the difference between import/export and source/target; they are independent concepts. Source and target indicate where data comes from and where it’s going, whereas import vs. export is mostly cosmetic.

How a given data flow’s orientation is determined, is basically up to the developer. Most of the time it is straightforward, e.g. CSV → Wutta would be import, and Wutta → CSV would be export. But confusing edge cases certainly exist, you’ll know them when you see them. In those cases the developer should try to choose whichever the end user is likely to find less confusing.

process_changes(changes)

Run post-processing operations on the given changes, if applicable.

This method is called by process_data(), if any changes were made.

Default logic will send a “diff warning” email to the configured recipient(s), if warnings mode is enabled. If it is not enabled, nothing happens.

Parameters:

changes – OrderedDict of changes from the overall import/export job. The structure is described below.

Keys for the changes dict will be model/importer names, for instance:

{
    "Sprocket": {...},
    "User": {...},
}

Value for each model key is a 3-tuple of (created, updated, deleted). Each of those elements is a list:

{
    "Sprocket": (
        [...], # created
        [...], # updated
        [...], # deleted
    ),
}

The list elements are always tuples, but the structure varies:

{
    "Sprocket": (
        [ # created, 2-tuples
            (obj, source_data),
        ],
        [ # updated, 3-tuples
            (obj, source_data, target_data),
        ],
        [ # deleted, 2-tuples
            (obj, target_data),
        ],
    ),
}

process_data(*keys, **kwargs)

Run import/export operations for the specified models.

Parameters:

*keys – One or more importer/exporter (model) keys, as defined by the handler.

Each key specified must be present in importers and thus will correspond to an importer/exporter class.

A transaction is begun on the source and/or target side as needed, then for each model key requested, the corresponding importer/exporter is created and invoked. And finally the transaction is committed (assuming normal operation).

See also these methods which may be called from this one:

• consume_kwargs()

• begin_transaction()

• get_importer()

• process_data() (on the importer/exporter)

• process_changes()

• rollback_transaction()

• commit_transaction()

rollback_source_transaction()

Rollback the transaction on the source side, if applicable.

This is normally called from rollback_transaction().

rollback_target_transaction()

Rollback the transaction on the target side, if applicable.

This is normally called from rollback_transaction().

rollback_transaction()

Rollback the current import/export transaction, on source and/or target side as needed.

This is normally called from process_data(). It is “always” called when dry_run is true, but also may be called if errors are encountered.

Default logic will call both:

• rollback_target_transaction()

• rollback_source_transaction()

Note

By default the target transaction is rolled back first; this is to avoid edge case errors when the source connection times out. In such cases we want to properly cleanup the target and then if an error happens when trying to cleanup the source, it is less disruptive.

runas_username = None

Username responsible for running the import/export job. This is mostly used for Continuum versioning.

source_key = None

Key identifier for the data source.

This should “uniquely” identify the data source, within the context of the data target. For instance in the case of CSV → Wutta, csv is the source key.

Among other things, this value is used in get_key().

target_key = None

Key identifier for the data target.

This should “uniquely” identify the data target. For instance in the case of CSV → Wutta, wutta is the target key.

Among other things, this value is used in get_key().

transaction_comment = None

Optional comment to apply to the transaction, where applicable. This is mostly used for Continuum versioning.

warnings = False

Boolean indicating the import/export should run in “warnings” mode.

If set, this declares that no changes are expected for the import/export job. If any changes do occur with this flag set, a diff warning email is sent within process_changes().

See also warnings_recipients, warnings_max_diffs and warnings_email_key.

warnings_email_key = None

Explicit email key for sending the diff warning email, unique to this import/export type.

Handlers do not normally set this, so the email key is determined automatically within get_warnings_email_key().

See also warnings.

warnings_max_diffs = 15

Max number of record diffs (per model) to show in the warning email.

See also warnings.

warnings_recipients = None

Explicit recipient list for the warning email. If not set, the recipients are determined automatically via config.

See also warnings.

enum wuttasync.importing.handlers.Orientation(value)

Enum values for ImportHandler.orientation.

Valid values are as follows:

IMPORT = <Orientation.IMPORT: 'import'>

EXPORT = <Orientation.EXPORT: 'export'>

class wuttasync.importing.handlers.ToSqlalchemyHandler(config, **kwargs)

Base class for import/export handlers which target a SQLAlchemy ORM (DB).

This is the base class for ToWuttaHandler, but can be used with any database.

See also FromSqlalchemyHandler.

begin_target_transaction()

Establish a new db session via make_target_session() and assign the result to target_session.

commit_target_transaction()

Commit the target_session.

make_target_session()

Make and return a new db session for the import/export.

Subclass must override this; default logic is not implemented.

rollback_target_transaction()

Rollback the target_session.

target_session = None

Reference to the SQLAlchemy db session for the target side.

This may often be a session for the app database (i.e. for importing to Wutta DB) but it could also be any other.

This will be None unless an import/export transaction is underway. See also begin_target_transaction().

class wuttasync.importing.handlers.ToWuttaHandler(config, **kwargs)

Handler for import/export which targets Wutta ORM (app database).

This inherits from ToSqlalchemyHandler.

See also FromWuttaHandler.

make_target_session()

This creates a typical db session for the app by calling make_session().

It then may “customize” the session slightly. These customizations only are relevant if Wutta-Continuum versioning is enabled:

If runas_username is set, the responsible user (continuum_user_id) will be set for the new session as well.

Similarly, if transaction_comment is set, it (continuum_comment) will also be set for the new session.

Returns:

Session instance.

target_key = 'wutta'