Programmatic Usage

You can get a lot of mileage out of the CLI (see Command Line Interface) but sometimes you may need to do something e.g. from within the web app, by invoking the import/export logic without using CLI. Here are some basic examples which show how to go about it.

As a general rule the first step is to instantiate the import/export handler. This requires a special kind of spec string indicating the “type” of import/export handler you need. For instance this gets the CSV → Poser (import) handler:

app = config.get_app()
handler = app.get_import_handler("import.to_poser.from_csv", require=True)

See also get_import_handler().

Full Import/Export (aka. mimic CLI)

You just have to call process_data() on the handler, specifying the desired model name(s) and any other params.

Some params (e.g. dry_run) really apply to the handler, but others will be passed along to the Importer constructor(s):

changes = handler.process_data("Widget", "Sprocket",
                               create=True, update=False, dry_run=True)

Note that it returns a structure indicating which data was “changed” (if any). See process_changes() for more info on that structure.

Import/Export just 1 record

This can be a fair bit more complicated, depending on the scenario. Most of the real-world use cases will be necessarily somewhat custom. So we’ll start simple.

Exporting a record from in-app table

This would not be like exporting to CSV, but rather exporting to another system which was being kept in sync.

For example let’s say you have an external CRM to manage customers, but you have those same customers in your app database. And then let’s say you have reason to sync just one of those, from your app to the CRM.

The first thing is to collect all params you will need to give it, e.g.:

params = {
   # nb. this may 'create' new record in CRM if not already present
   "create": True,
   "update": True,
   "delete": False,
   "dry_run": False,
}

Then you must tell the handler to “consume” some of these params, via consume_kwargs(). It will return the “final” set of params which you will use later. Note that we’re getting the Poser → CRM (export) handler here:

handler = app.get_import_handler("export.to_crm.from_poser", require=True)
params = handler.consume_kwargs(params)

The other thing which usually happens here, but can depend on the situation, is to officially begin a transaction in the handler:

handler.begin_transaction()

You will need to commit or rollback this transaction at the end.

Now with the “filtered” params you create the desired importer/exporter. And again, we just use “importer” as a stand-in for both orientations here:

importer = handler.get_importer("Customer", **params)

You should be sure to disable caches_target, otherwise the importer may try to read the full target data set into memory:

importer.caches_target = False

Now back to the specific customer account, here we have a reference to the local in-app customer, that we intend to export to CRM. We’ll ask the importer to normalize_source_data():

data = importer.normalize_source_data(source_objects=[customer])

And finally, we ask the importer to “import” that data, via process_data():

changes = importer.process_data(source_data=data)

Oh yeah and don’t forget to commit or rollback:

if handler.dry_run:
    handler.rollback_transaction()
else:
    handler.commit_transaction()

Importing a record from external system

The reason the example above is “simple” is that we start out with a single source record, and the importer already knows how to (try to) fetch the corresponding target record (which may or may not exist yet).

So for this example we’ll go “complicated” and say that we want to import a single customer account from the CRM, but again will start out with only the local account “known” at this point in the code.

The task changes slightly then, because the importer needs to be given the source record(s) for this approach to work. So we must first track down the CRM record through other means:

crm_customer = fetch_crm_customer_for_local_customer(customer)

But once you have that, the rest of the process is actually the same. Here we’ll use the CRM → Poser (import) handler:

params = {
   # nb. we know 'create' does not apply since we already have both records
   "create": False,
   "update": True,
   "delete": False,
   "dry_run": False,
}

handler = app.get_import_handler("import.to_poser.from_crm", require=True)
params = handler.consume_kwargs(params)
handler.begin_transaction()

importer = handler.get_importer("Customer", **params)
importer.caches_target = False

# nb. make sure to give the *source* record, from CRM in this case
data = importer.normalize_source_data(source_objects=[crm_customer])
changes = importer.process_data(source_data=data)

if handler.dry_run:
    handler.rollback_transaction()
else:
    handler.commit_transaction()