Release notes for NLP API 2.31

Contents

• Brief Summary
• New Features
• Changes
• Python SDK updates
• Java SDK updates

Brief Summary

Release versions

• NLP API version 2 Release 31.
• Python SDK and client v5.26.0.
• Java SDK and client v5.14.0.
• Python SDK CCD reader plugin v2.2.0.
• NLP API specification v2.17.0: added support for new processing features radplaybook-ontology and entity-guidance.
• JSON result output schema v2.26.1 for emtellipro-json-2: added new found entity attribute guidance, new ontology radplaybook-ontology, new temporality category specific, new date_time argument added to medication relations and added ontology version information.
• visualclient version vc-v2023.03.17-dadc23b8 added support for new processing features entity-guidance and radplaybook-ontology and new specific category of temporality relation.
• New version of the Snowflake CSV upload SQL script.
• New description and relationship tables for SNOMED September 2022 release available on the downloads page.

Highlights

• Added the rad-playbook (LOINC/RSNA Radiology Playbook) ontology.
• Guidance: new attribute for found entities.
• New specific category for temporality relations.
• The version of every ontology in the NLP API output is provided in the result JSON.
• Added date_time argument to medication relations.

New Features

rad-playbook (LOINC/RSNA Radiology Playbook) Ontology

From the LOINC/RSNA Radiology Playbook website

The wide variation in the way radiology procedures are named in radiology departments makes ordering and protocoling less efficient and radiology records difficult to aggregate or compare across sites. The LOINC/RSNA Radiology Playbook was created to address this critical problem. Building on RadLex, RSNA developed names for nearly 1,000 commonly performed radiology procedures. Playbook provides a standard system for naming radiology procedures, based on the elements that define an imaging exam such as modality and body part.

In the output json result format v2.26.1 or later the NLP API adds a new ontology rad_playbook in the output controlled by the NLP API processing feature radplaybook-ontology.

A link to the concept will appear for each found entity that has an rad-playbook code. The link will appear in the list of concept_links for each found entity (in the list found in the entities object).

The example output below is truncated to the relevant parts (the ... indicate the removal of unnecessary details).

1
{
2
    "documents": [
3
        {
4
            ...
5
            "entities": {
6
                "found": [
7
                    {
8
                        ...
9
                        "concept_links": [
10
                            ...
11
                            "Crad_playbook0"
12
                        ],
13
                        "entity_type": {
14
                          "rad_playbook": "CT Chest"
15
                        },
16
                        "text": [
17
                          "CT Chest"
18
                        ]
19
                    }
20
                ],
21
            }
22
            ...
23
            "concepts": {
24
                ...
25
                "rad_playbook": [
26
                  {
27
                    "label": "Crad_playbook0",
28
                    "concept_id": "24627-2",
29
                    "description": "CT CHEST"
30
                  }
31
                ]
32
            }
33
            ...
34
        }
35
    ],
36
    "format": "emtellipro-json-2",
37
    "version": "2.26.1",
38
}

Guidance attribute (beta)

This new attribute captures whether this entity is part of a guidance statement, where someone is suggesting that someone else do something. Say a caregiver writes ‘If the patient experiences abdominal pain, she should return to the ER.’ In this case, ‘abdominal pain’ is not something the patient currently has, but is part of a guidance statement, and thus the value of this attribute for ‘abdominal pain’ would be is_guidance. If an entity is not part of a guidance statement, it is assigned a value of not_guidance. This attribute is present in the output of the NLP API if the entity-guidance feature is enabled and only for Radiology reports and only in FINDINGS, IMPRESSION and ADDENDUM sections.

For the following input processed as a Radiology/generic by the NLP API:

IMPRESSION:
If there is ongoing concern for pneumothorax, an upright chest x-ray should be obtained.

The NLP API adds a new attribute guidance to the entity information for found entities in the output JSON result format v2.26.1 or later.

The output JSON for the above sentence is shown below. The focus in the example below is on just the new guidance attribute. The example output below is truncated to the relevant parts (the ... indicate the removal of unnecessary details).

1
{
2
  "documents": [
3
    {
4
         ...
5
         "entities": {
6
             "found": [
7
               {
8
                 ...
9
                 "section_name": "IMPRESSION",
10
                 "attributes": {
11
                   ...
12
                   "guidance": "is_guidance"
13
                 },
14
                 ...
15
                 "text": [
16
                   "chest x-ray"
17
                 ]
18
               },
19
               ...
20
             ],
21
         }
22
         ...
23
    "format": "emtellipro-json-2",
24
    "version": "2.26.1"
25
  }
26
}

specific category for temporality relations

The temporality relation in the NLP API indicates whether the subject entity is in scope of a temporal_entity. Currently the supported categories of temporality are historical and specific. The historical category was released previously in the NLP API v2.28. In this release we add a new category called specific to the temporality relation. The specific category indicates that the subject entity (like diabetes) is within the scope of a specific time, date, or date range (since 1982) in the sentence He has had diabetes since 1982. The polarity attribute for this relation indicates whether the temporality status itself is negated or asserted.

This output will be present in NLP API output if the temporality-relations feature is enabled.

For the following input processed as a Clinical/generic by the NLP API:

He has had diabetes since 1982.

The output JSON for the above sentence is shown below. The focus in the example below is on one relation, namely temporalities. The example output below is truncated to the relevant parts (the ... indicate the removal of unnecessary details).

1
{
2
  "documents": [
3
    {
4
      ...
5
      "relations": {
6
        ...
7
        "temporalities": [
8
          {
9
            "label": "RT1",
10
            "attributes": {
11
              "polarity": "asserted",
12
              "category": "specific"
13
            },
14
            "args": {
15
              "subject": {
16
                "ref": "E1",
17
                "text": [
18
                  "diabetes"
19
                ]
20
              },
21
              "modifiers": [],
22
              "temporal_entity": {
23
                "ref": "E0",
24
                "text": [
25
                  "since 1982"
26
                ]
27
              }
28
            }
29
          }
30
        ]
31
      },
32
      ...
33
  "format": "emtellipro-json-2",
34
  "version": "2.26.1"
35
  }
36
}

Ontology versions

The version of every ontology in the NLP API output is provided in the result JSON as a new object called ontology_versions.

The output JSON for the above sentence is shown below. The focus in the example below is on one output feature, namely ontology_versions. The example output below is truncated to the relevant parts (the ... indicate the removal of unnecessary details).

1
{
2
  "documents": [
3
    {
4
      ...
5
      "ontology_versions": {
6
        "snomed": {
7
          "release_version": "2022-09-01",
8
          "release_type": "snapshot",
9
          "packaging_date": "2022-11-17",
10
          "language": "en_us"
11
        },
12
        "radlex": {
13
          "release_version": "2020-11",
14
          "packaging_date": "2022-09-27"
15
        },
16
        "rxnorm": {
17
          "release_version": "2022-11-07",
18
          "release_type": "full",
19
          "packaging_date": "2022-11-07"
20
        },
21
        ...
22
        "rad_playbook": {
23
          "release_version": "2022-08-08",
24
          "packaging_date": "2022-11-10"
25
        }
26
      },
27
      ...
28
  "format": "emtellipro-json-2",
29
  "version": "2.26.1"
30
  }
31
}

Only the release_version is guaranteed to exist in each ontology description. All the other information is optional.

New date_time argument in medication relations

The new date_time arugment for medication relations appears as a new argument for medication relations and appears in the result JSON under medications.

For the following input processed as a Clinical/generic by the NLP API:

• lisinopril 10 mg oral tablet, = 1 cap(s), PO, Once Daily, 10 cap(s), 09/22/2008, 10/02/2008, Ordered

The output JSON for the above sentence is shown below. The focus in the example below is on one relation, namely medication-relations. The example output below is truncated to the relevant parts (the ...).

1
{
2
  "documents": [
3
    {
4
      ...
5
      "relations": {
6
        "medications": [
7
          {
8
            "label": "RMD0",
9
            "attributes": {},
10
            "args": {
11
              "drug": { "ref": "E71", "text": [ "lisinopril" ] },
12
              "dosage": [ { "ref": "E72", "text": [ "10 mg" ] } ],
13
              "route": [ { "ref": "E73", "text": [ "oral" ] },
14
                         { "ref": "E75", "text": [ "PO" ] } ],
15
              "quantity": [ { "ref": "E197", "text": [ "1" ] }, 
16
                            { "ref": "E198", "text": [ "10" ] } ],
17
              "mode": [ { "ref": "E74", "text": [ "tablet" ] },
18
                        { "ref": "E199", "text": [ "cap(s)" ] },
19
                        { "ref": "E200", "text": [ "cap(s)" ] } ],
20
              "frequency": [ { "ref": "E76", "text": [ "Once Daily" ] } ],
21
              "duration": [],
22
              "necessity": [],
23
              "modifier": [],
24
              "indication": [],
25
              "date_time": [ { "ref": "E78", "text": [ "09/22/2008" ] },
26
                             { "ref": "E79", "text": [ "10/02/2008" ] } ]
27
            }
28
          }
29
       ]
30
     }
31
   }
32
  ]
33
  "format": "emtellipro-json-2",
34
  "version": "2.26.1",
35
}

Changes

Ontology updates

• SNOMED CT database updated to version 2022-09-01 US Edition.
• Improved longer entity matching to SNOMED CT.
• Added >50K new annotations of synonyms for SNOMED CT concepts.
• RxNorm updated to November 2022 release.
• UMLS updated to version 2022AB.

Changes to entities

• Change SNOMED concept ID for temporal entities to 260863009 | Temporal relationship (attribute) |.
• Fix measurement value emt_sem_type (internal) / semantic_type (external) for SNOMED, corresponding to NLP API UMLS entity type. It was ‘measurement’ but is now ‘Quantitative Concept’.

Accuracy Improvements

• Improved accuracy of imagelink relations.
• Improved accuracy of measurement relations.
• Improved accuracy of entity disambiguation for some common acronyms.
• Improved accuracy of negation detection.
• Improved accuracy of uncertainty detection.
• Add new measurement units CFU/ml and organisms/ml.

NLP API Improvements

• New private interface for identity and access management.
• Improvements to the NLP API admin UI.
• Support for scalesets in cloud computing to use hundreds or thousands of workers as spot instances.

Python SDK updates

The latest NLP API Python SDK is version 5.26.0. The last NLP API release v2.30 was released with Python SDK version 5.25.0. Refer to the changelog for the Python SDK for a full explanation of the updates made since the last release. You will need to use at least version 5.7.0 of the Python SDK or later to send requests to the NLP API v2.29 but if you want to use the new features explained above you will need to use at least version 5.26.0. We recommend using the latest version of the Python SDK 5.26.0 with the NLP API v2.31.

$
emtellipro-db-client --database postgresql://username:password@hostname:port/example_db migrate

rad-playbook (LOINC/RSNA Radiology Playbook) Ontology

When using the emtellipro-db-client Python database client the foundentityconcept table contains rad_playbook and the values in the found_entity_id column can be used to find the mapping to the entity in the foundentityconcept table:

$
# select * from foundentityconcept where concept_ontology='rad_playbook' limit 5;
$
 found_entity_id | concept_id | concept_ontology
$
-----------------+------------+------------------
$
            2521 | 39026-0    | rad_playbook
$
            5963 | 24627-2    | rad_playbook
$
            5585 | 39026-0    | rad_playbook
$
            6731 | 39026-0    | rad_playbook
$
             226 | 39026-0    | rad_playbook
$
(5 rows)
$
$
$
# select * from foundentityspan  where found_entity_id='2521';
$
  id  | start | end  | text | found_entity_id
$
------+-------+------+------+-----------------
$
 2521 |  1235 | 1237 | CT   |            2521
$
(1 row)

Guidance attribute (beta)

Support has been added for the guidance attribute of found entities. This is now found in the SDK as FoundEntity.guidance (where it is stored as a string), and it’s stored in the database in the new foundentity.guidance column. The new feature is called entity-guidance. Using this updated table will require a database migration.

When using the emtellipro-db-client Python database client the foundentity table contains a column for the guidance attribute. The values can be is_guidance or not_guidance. rad_playbook and the values in the found_entity_id column can be used to find the mapping to the entity in the foundentityconcept table:

$
# select * from foundentity where guidance='is_guidance' limit 2;
$
 id | polarity | section_name | uncertainty | known_ambiguity | question_status |  guidance   | document_id
$
----+----------+--------------+-------------+-----------------+-----------------+-------------+-------------
$
  1 | asserted | FINDINGS     | certain     | unknown         | not_question    | is_guidance |           1
$
 15 | asserted | FINDINGS     | certain     | unknown         | not_question    | is_guidance |           1
$
(2 rows)
$
$
# select * from foundentity where guidance='not_guidance' limit 2;
$
 id | polarity | section_name | uncertainty | known_ambiguity | question_status |   guidance   | document_id
$
----+----------+--------------+-------------+-----------------+-----------------+--------------+-------------
$
  2 | asserted | INDICATION   | certain     | unambiguous     | not_question    | not_guidance |           1
$
  3 | asserted | TECHNIQUE    | certain     | unknown         | not_question    | not_guidance |           1
$
(2 rows)

You can use the id from the foundentity table to find out more information about the entity:

$
# select * from foundentityspan where found_entity_id=1;
$
 id | start | end |   text   | found_entity_id
$
----+-------+-----+----------+-----------------
$
  1 |     2 |  10 | CT CHEST |               1
$
(1 row)
$
$
# select * from foundentityspan where found_entity_id=2;
$
 id | start | end |  text  | found_entity_id
$
----+-------+-----+--------+-----------------
$
  2 |    53 |  59 | distal |               2
$
(1 row)

specific category for temporality relations

The new specific category for temporality relations are stored in the same tables as the earlier historical category for this relation:

• temporalityrelation with links to two argument entities: the subject argument subject_id and the temporal entity argument temporal_entity_id.
• temporalityrelationmodifier (a list of modifier arguments for this relation)

$
# select * from temporalityrelation where category='specific' limit 2;
$
 id | polarity | category | temporal_entity_id | subject_id
$
----+----------+----------+--------------------+------------
$
  7 | asserted | specific |                200 |        201
$
 10 | asserted | specific |                565 |        440
$
(2 rows)

The temporal_entity_id can be viewed in the entity table:

$
# select * from entity where id='200';
$
 id  | type_ | document_id | entity_id
$
-----+-------+-------------+-----------
$
 200 | found |           2 |       199
$
(1 row)

The text span information for this temporal entity argument can then be viewed in the foundentityspan table:

$
# select * from foundentityspan where id='199';
$
 id  | start | end |   text    | found_entity_id
$
-----+-------+-----+-----------+-----------------
$
 199 |   151 | 160 | 2021-5-30 |             199
$
(1 row)

A sample SQL query for temporality relations is provided below:

SELECT d.filename,
       t.category        AS category,
       sub_fes.text      AS subject,
       temporal_fes.text AS temporal,
       s.text            AS sentence
FROM temporalityrelation t
       JOIN entity sub_e ON t.subject_id = sub_e.id AND sub_e.type_ = 'found'
       JOIN foundentity sub_fe ON sub_e.entity_id = sub_fe.id
       JOIN foundentityspan sub_fes ON sub_e.entity_id = sub_fes.id
       JOIN foundentitylocation sub_fel on sub_fe.id = sub_fel.found_entity_id
       JOIN location l on sub_fel.location_id = l.id AND l.type_ = 'sentence'
       JOIN sentencelocation s on l.location_id = s.id
       JOIN document d ON l.document_id = d.id
       JOIN entity temporal_e ON t.temporal_entity_id = temporal_e.id AND
                                 temporal_e.type_ = 'found'
       JOIN foundentity temporal_fe ON temporal_e.entity_id = temporal_fe.id
       JOIN foundentityspan temporal_fes
            ON temporal_fe.id = temporal_fes.found_entity_id;

Ontology versions

Support is now available for the ontology_versions information returned by the NLP API for each processed document. These are available as AnnotatedDocument.ontology_versions and in the ontologyversion table. The database migration will insert rows in the ontologyversion table based on the documentconcepts table, but with release_version = NULL since previous ontology version data is not available.

$
# select * from ontologyversion;
$
$
 document_id |    ontology     | release_version
$
-------------+-----------------+-----------------
$
           1 | snomed          | 2022-09-01
$
           1 | radlex          | 2020-11
$
           1 | rxnorm          | 2022-11-07
$
           1 | rad_playbook    | 2022-08-08
$
...

Since each document in a batch might be processed with a different set of ontologies, the release version information is repeated for each document_id.

New date_time argument in medication relations

The new date_time argument of medication relations is stored in the Python SDK in emtellipro.data.MedicationRelation.date_times (as a list), and in the medicationrelationdatetime database table when using emtellipro-db-client.

Java SDK updates

The latest NLP API Java SDK is version 5.14.0. Refer to the changelog for the Java SDK for a full explanation of the updates made since the last public release. You will need to use version 5.14.0 or later to take advantage of the new features described above.