Multithreaded database client

New in version 7.0.

The database client (emtellipro-db-client) has been rebuilt to process documents using multiple worker threads. Each worker repeatedly takes a batch of documents, submits it to the NLP API, retrieves the results, and stores them to the output database. This lets a single run submit, process, and save in parallel rather than one stage at a time.

For most users the command-line interface works the same way as before, but several options have been added, renamed, or removed, and the state file format has changed. The sections below describe what changed.

Running jobs in parallel

The new -j/--jobs option controls how many submit/process/store jobs run in parallel:

1
$ emtellipro-db-client process -j 4 example-data/

Each job processes its own batch of documents end-to-end. The default is 1, which preserves the previous single-threaded behaviour.

New options

These are all for the process command.

-j, --jobs
Number of parallel submit/process/store jobs to run. Default: 1.

--sql-count-query
When reading input from a database, an optional query returning the number of rows that --sql-query will produce. If omitted, a SELECT COUNT query is generated automatically. This is used to drive the progress bars.

--sql-limit
An optional limit on how many rows to read from --sql-query.

Renamed options

The ‘submit shard size’ option was renamed for clarity. Update any scripts or config files that use the old names.

Removed options

The following options have been removed:

-r, --recursive
Directories passed as input paths are now always searched recursively, so this flag is no longer needed.

--max-save-shard-size
Saving is now handled by the worker jobs and no longer needs a separate shard size. The saving shard size is the same as --batch-size.

--retry-forever
Removed. Use --max-retries to control retry behaviour.

The --job-id option is no longer accepted by the process and store-result commands. To set a job ID when processing, use process --save-opt instead:

1
$ emtellipro-db-client process --save-opt job_id=<uuid> example-data/

State file changes

The state file (set with -s/--state) has changed. Previously it stored a compressed copy of every input document so a run could be resumed. It now stores only a unique key per document along with its processing status, which keeps the file much smaller while still allowing a run to be restarted from the same state file.

In addition, the state file now records how long each document spent in each processing stage (loading, processing, and saving). This means the run summary can report finer-grained timing information, for example average time per document per stage:

Documents processed: 1200
  saved: 1200
Average time per document per stage:
  loading      0.012s/doc  (14.4s over 1200 docs)
  processing   0.305s/doc  (366.0s over 1200 docs)
  saving       0.041s/doc  (49.2s over 1200 docs)

Because the format changed, state files created by older versions of the client are not compatible with this version.