Release notes for NLP API 2.29

Contents

• Brief Summary
• New Features
• Changes
• Python SDK updates
• Java SDK updates

Brief Summary

Release versions

• NLP API version 2 Release 29.
• Python SDK and client v5.24.0.
• Java SDK and client v5.9.0.
• Python SDK CCD reader plugin v1.9.0.
• NLP API specification v2.16.0: added support for new processing features snomedicd10cm-ontology ontology and reportedevent-relations.
• JSON result output schema v2.21.0 for emtellipro-json-2: added new output for snomed_icd10_cm concepts and reportedevents relations and the NLP API engine-version used to process the submitted documents.
• visualclient version vc-v2022.07.04-ef5da396 added support for new processing features snomed_icd10_cm ontology and Reported Event relations.

Highlights

• Added ICD-10-CM ontology.
• Added new Reported Event relation with category Critical Results Communication for Radiology reports (beta).
• Improved accuracy and coverage of measurement relations.
• Improved coverage for historical events in temporality relations.
• SNOMED CT database updated to version 2022-04-11 US Edition.
• UMLS updated to version 2021AB.

New Features

ICD-10-CM Ontology Support

The ICD-10 Clinical Modification, ICD-10-CM, is a modification of the ICD-10, and is primarily used as a source for diagnostic codes in the United States of America.

In this release, the NLP API now supports ICD-10-CM as a new ontology, snomedicd10cm-ontology, which will output ICD-10-CM codes derived from SNOMED-based named entity recognition. Please note that in many cases, this will output different ICD-10 codes from the NLP API’s currently supported snomedicd10-ontology feature.

In the output json result format v2.21.0 or later the NLP API adds a new ontology snomed_icd10_cm.

For the following input processed as a Clinical/generic by the NLP API, and using the snomedicd10cm-ontology feature:

This has the appearance of breast cancer.

A link to the concept will appear for each found entity that has an ICD-10-CM code. The link will appear in the list of concept_links for each found entity (in the list found in the entities object).

The example output below is truncated to the relevant parts (the ... indicate the removal of unnecessary details). Note that NLP API’s snomedicd10-ontology for breast cancer is C50.9 (Malignant neoplasm of breast of unspecified site) while for the snomedicd10cm-ontology, two codes are produced, C50.919 and C50.929 (“Malignant neoplasm of unspecified site of unspecified female breast” and “Malignant neoplasm of unspecified site of unspecified male breast”). The two codes come with a description that contains information to help decide which code to choose based on the gender of the patient.

1
{
2
    "documents": [
3
        {
4
            ...
5
            "entities": {
6
                "found": [
7
                    {
8
                        ...
9
                        "concept_links": [
10
                            ...
11
                            "Csnomed_icd100",
12
                            "Csnomed_icd10_cm0",
13
                            "Csnomed_icd10_cm1",
14
                        ],
15
                        "entity_type": {
16
                            ...
17
                            "snomed_icd10_cm": "disorder",
18
                            "snomed_icd10": "disorder",
19
                            ...
20
                        },
21
                        "text": [
22
                            "breast cancer"
23
                        ]
24
                    }
25
                ],
26
            }
27
            ...
28
            "concepts": {
29
                ...
30
                "snomed_icd10": [
31
                    {
32
                        "label": "Csnomed_icd100",
33
                        "concept_id": "C50.9",
34
                        "description": "Malignant neoplasm of breast of unspecified site (Neoplastic Process)"
35
                    },
36
                    ...
37
                ],
38
                "snomed_icd10_cm": [
39
                    {
40
                        "label": "Csnomed_icd10_cm0",
41
                        "concept_id": "C50.919",
42
                        "description": "Malignant neoplasm of unspecified site of unspecified female breast (Neoplastic Process)"
43
                    },
44
                    {
45
                        "label": "Csnomed_icd10_cm1",
46
                        "concept_id": "C50.929",
47
                        "description": "Malignant neoplasm of unspecified site of unspecified male breast (Neoplastic Process)"
48
                    }
49
                ]
50
            }
51
            ...
52
        }
53
    ],
54
    "format": "emtellipro-json-2",
55
    "version": "2.21.0",
56
}

Reported Event relation (beta)

Beta version of a new reportedevent relation indicating whether the subject entity is the topic of communication from one group of entities to another. Currently the only supported category of reportedevent is critical_results.

This relation indicates that the subject (e.g. the “findings” of the report) has been communicated by a group of entities (could be a singleton) in the list argument from_entities to another group of entities (also could be a singleton) in the list argument to_entities. The mode of communication is provided in the argument methods while the time_expression argument captures when the communication took place.

Currently the only category produced by the NLP API is the category of critical_results which indicates that the communication was sent to a healthcare profession and that the sentence occurs in the following sections: FINDINGS, IMPRESSION, CALLED REPORT, CRITICAL RESULTS COMMUNICATION, COMMENTS and only from reports submitted with an NLP API category of Radiology with any subcategory.

This new feature is available in the result output by asking for the reportedevent-relations processing feature when submitting to the NLP API.

For the following input processed as a Radiology/generic by the NLP API:

CRITICAL RESULTS COMMUNICATION:
The above findings and recommendations were discussed in person with Dr. Smith on 2149-4-2 .

The NLP API adds a new relation reportedevents to the list of relations in output JSON result format v2.21.0 or later. The output JSON for the above sentence is shown below. The focus in the example below is on one relation, namely reportedevents. The example output below is truncated to the relevant parts (the ... indicate the removal of unnecessary details).

1
{
2
  "documents": [
3
    {
4
      ...
5
      "relations": {
6
        ...
7
         "reportedevents": [
8
             {
9
                 "label": "RR0",
10
                 "attributes": { "category": "critical_results", "polarity": "asserted" },
11
                 "args": {
12
                     "subject": { "ref": "E4", "text": [ "findings" ] },
13
                     "to_entities": [ { "ref": "E5", "text": [ "Dr. Smith" ] } ],
14
                     "from_entities": [],
15
                     "methods": [ { "ref": "E6", "text": [ "in person" ] } ],
16
                     "time_expressions": [ { "ref": "E7", "text": [ "2149-4-2" ] } ]
17
                 }
18
             },
19
             {
20
                 "label": "RR1",
21
                 "attributes": { "category": "critical_results", "polarity": "asserted" },
22
                 "args": {
23
                     "subject": { "ref": "E8", "text": [ "recommendations" ] },
24
                     "to_entities": [ { "ref": "E5", "text": [ "Dr. Smith" ] } ],
25
                     "from_entities": [],
26
                     "methods": [ { "ref": "E6", "text": [ "in person" ] } ],
27
                     "time_expressions": [ { "ref": "E7", "text": [ "2149-4-2" ] } ]
28
                 }
29
             }
30
         ]
31
      },
32
      ...
33
  "format": "emtellipro-json-2",
34
  "version": "2.21.0"
35
}

Changes

Ontology updates

• UMLS updated to version 2021AB.
• SNOMED CT database updated to version 2022-04-11 US Edition.
• ICD-10-CM ontology added.
• Removed mapping advice for ICD-10 ontology and replaced it with UMLS Preferred Term and sementic group, e.g. Malignant neoplasm of breast of unspecified site (Neoplastic Process).

Accuracy Improvements

Entities

• Numerous additions and updates to entity linking for multiple ontologies.
• Addition of less common units of measurement for matching of entire measurement value.
• Updates to latest version of SNOMED and UMLS.
• Identification of new measurements with the measurement units mL/hr, pack year, fL (femtoliter), centimeters per second, x10^3/uL, 10E6/uL and variants, L/min/m2, mg/24 hr, mg/dL, cc/hr and {M/uL}.
• Include <, <=, >= and > in measurement values.
• Add support for ranges of measurement values, e.g. 120-159.
• Fixed false positive of HR (heart rate) as part of measurement value. It is now correctly identified as measurement subject.
• Fixed false positive with Mg (magnesium) incorrectly identified as milligrams.
• Fix for time expressions that sometimes blocked reported event relations.
• Give priority to entities with a definitionStatusId of 900000000000073002 (Sufficiently defined by necessary conditions definition status).

Relations

• Remove man, woman and similar ambiguous terms as experiencers in experiencer relations.
• Recognize proper names that start with Dr., Mr., Ms., Mx. and Mrs. and other honorifics for experiencer and reported event relations..
• New temporal relation model with improved consistency and accuracy.
• New measurement relation model with significantly increased accuracy.
• Add , as a possible scope breaker in measurement relations.
• Reported event relation is enabled in Radiology reports (only) in the sections: FINDINGS, IMPRESSION, CALLED REPORT, CRITICAL RESULTS COMMUNICATION, COMMENTS.
• Fix overlapping span bug between arguments in measurement relations.
• Fix false positive for measurement values where you have two separated numbers followed by a unit, e.g. 32 96%.
• Qualifier relations updated to include stage 0 etc. as qualifiers.
• Enable matching of qualifier values in qualifier relations to assessment scale and observable entity entities.

Polarity

• Fix negation detection failure with there is no X to suggest Y pattern.
• Fix negation issue with negated temporality triggers.
• Fixed scope of question of to be rightwards in negation detection.
• Fix negation error in no acute distress.
• Assert some entities which properly contain negation terms.

Misc

• Fix lemmatization bug that was causing incorrect concept ID assignment for mentions like fainting.
• Support for CALLED REPORT radiology heading.
• Several accuracy improvements for the uncertainty attribute on found entities.
• More robust handling of dirty input data with OCR output that appears as Turkish text.
• Improve uncertainty accuracy for is concerning for triggers.

NLP API Improvements

• Added engine-version to the JSON result output and the engine version is also stored in the processingdetails table in the NLP API output database.
• For on-premises deployments, the admin UI has a new look with improved search and paging support for log views.
• Added support for more workers in the API server which has improved the throughput of the NLP API engine.

Python SDK updates

The latest NLP API Python SDK is version 5.24.0. The last NLP API release v2.28 was released with Python SDK version 5.22.0. Refer to the changelog for the Python SDK for a full explanation of the updates made for 5.23.0 and 5.24.0. You will need to use at least version 5.7.0 of the Python SDK or later to send requests to the NLP API v2.29 but if you want to use the new features explained above you will need to use at least version 5.23.0. We recommend using the latest version of the Python SDK 5.24.0 with the NLP API v2.29.

$
emtellipro-db-client --database postgresql://username:password@hostname:port/example_db migrate

ICD-10-CM Support in Output Database

When using the emtellipro-db-client Python database client the foundentity table contains the found entities and the id column can be used to find the concept mapping in the foundentityconcept table:

$
# select * from foundentity where id=443 ;
$
 id  | polarity |          section_name          | uncertainty | known_ambiguity | question_status | document_id
$
-----+----------+--------------------------------+-------------+-----------------+-----------------+-------------
$
 443 | asserted | CRITICAL RESULTS COMMUNICATION | certain     | unambiguous     | not_question    |           1
$
$
# select * from foundentity where id=496 ;
$
 id  | polarity |          section_name          | uncertainty | known_ambiguity | question_status | document_id
$
-----+----------+--------------------------------+-------------+-----------------+-----------------+-------------
$
 496 | asserted | CRITICAL RESULTS COMMUNICATION | certain     | unambiguous     | not_question    |           1
$
$
# select * from foundentityconcept where concept_ontology='snomed_icd10_cm';
$
 found_entity_id | concept_id | concept_ontology
$
-----------------+------------+------------------
$
             443 | K76.9      | snomed_icd10_cm
$
             496 | I82.90     | snomed_icd10_cm
$
...

You can look up the description of the concept_id from the above table in the concept table:

$
# select * from concept where concept_id='K76.9';
$
 concept_id |    ontology     |   description
$
------------+-----------------+---------------------------------------
$
 K76.9      | snomed_icd10    | Liver disease, unspecified (Disease or Syndrome)
$
 K76.9      | snomed_icd10_cm | Liver disease, unspecified (Disease or Syndrome)
$
$
# select * from concept where concept_id='I82.90';
$
 concept_id |    ontology     |   description
$
------------+-----------------+---------------------------------------------------------------
$
 I82.90     | snomed_icd10_cm | Acute embolism and thrombosis of unspecified vein (Disease or Syndrome)

Reported Event relation in Output Database

When using the emtellipro-db-client Python database client the new reported event relation appears in the reportedeventrelation table and associated reportedeventrelationtoentity, reportedeventrelationfromentity, reportedeventrelationmethod and reportedeventrelationtimeexpression tables which correspond to the various arguments of the reported event relation.

The attributes of polarity and category and the identifier of the found entity that is the subject of the relation is provided in subject_id which is the link to the foundentity table which lets you obtain all the information about the subject:

Relation tables

The ER diagram for the Reported Event relation is shown below:

ER Diagram for the Reported Event relation.

The reportedeventrelation table looks like this:

$
# select * from reportedeventrelation;
$
 id | polarity |     category     | subject_id
$
----+----------+------------------+------------
$
  1 | asserted | critical_results |          5
$
  2 | asserted | critical_results |          9
$
  3 | asserted | critical_results |         12
$
  4 | asserted | critical_results |         15
$
  5 | asserted | critical_results |         16
$
...

The reportedeventrelationtoentity table below has the reported_event_id which links back to the reportedeventrelation table and entity_id which is the link to the foundentity table:

$
# select * from reportedeventrelationtoentity;
$
 reported_event_id | entity_id
$
-------------------+-----------
$
...

The reportedeventrelationfromentity table below has the reported_event_id which links back to the reportedeventrelation table and entity_id which is the link to the foundentity table:

$
# select * from reportedeventrelationfromentity;
$
 reported_event_id | entity_id
$
-------------------+-----------
$
...

The reportedeventrelationmethod table below has the reported_event_id which links back to the reportedeventrelation table and entity_id which is the link to the foundentity table:

$
# select * from reportedeventrelationmethod;
$
 reported_event_id | entity_id
$
-------------------+-----------
$
...

The reportedeventrelationtimeexpression table below has the reported_event_id which links back to the reportedeventrelation table and entity_id which is the link to the foundentity table:

$
# select * from reportedeventrelationtimeexpression;
$
 reported_event_id | entity_id
$
-------------------+-----------
$
...

SQL query

A sample SQL query that provides a list of the reported event relations in a database produced by the database client is provided below:

WITH critical_res AS (
   SELECT ev.*, rel.document_id FROM relation rel
        JOIN reportedeventrelation ev ON rel.relation_id = ev.id AND rel.type_='reportedevent'
        WHERE ev.category = 'critical_results'
 ),
 to_persons AS (
  SELECT ev.reported_event_id, string_agg(fes.text, '::') as to_entities FROM reportedeventrelationtoentity ev
    JOIN entity e ON ev.entity_id = e.id AND e.type_='found'
    JOIN foundentity fe ON e.entity_id = fe.id
    JOIN foundentityspan fes ON fe.id = fes.found_entity_id
    GROUP BY ev.reported_event_id
 ),
 from_persons AS (
  SELECT ev.reported_event_id, string_agg(fes.text, '::') as from_entities FROM reportedeventrelationfromentity ev
    JOIN entity e ON ev.entity_id = e.id AND e.type_='found'
    JOIN foundentity fe ON e.entity_id = fe.id
    JOIN foundentityspan fes ON fe.id = fes.found_entity_id
    GROUP BY ev.reported_event_id
 ),
 _methods AS (
  SELECT ev.reported_event_id, string_agg(fes.text, '::') as methods FROM reportedeventrelationmethod ev
    JOIN entity e ON ev.entity_id = e.id AND e.type_='found'
    JOIN foundentity fe ON e.entity_id = fe.id
    JOIN foundentityspan fes ON fe.id = fes.found_entity_id
    GROUP BY ev.reported_event_id
 ),
 time_exprs AS (
  SELECT ev.reported_event_id, string_agg(fes.text, '::') as time_expressions FROM reportedeventrelationtimeexpression ev
    JOIN entity e ON ev.entity_id = e.id AND e.type_='found'
    JOIN foundentity fe ON e.entity_id = fe.id
    JOIN foundentityspan fes ON fe.id = fes.found_entity_id
    GROUP BY ev.reported_event_id
 )
 SELECT
  sub_fes.text as subject,
  sub_c.description as concept,
  sub_c.concept_id,
  tp.to_entities,
  frp.from_entities,
  m.methods,
  te.time_expressions,
  cr.polarity as event_polarity,
  sl.text as sentence,
  d.filename
FROM critical_res cr
 -- get subject
 JOIN entity sub_e ON cr.subject_id = sub_e.id AND sub_e.type_='found'
 JOIN foundentity sub_fe ON sub_e.entity_id = sub_fe.id
 JOIN foundentityspan sub_fes ON sub_fe.id = sub_fes.found_entity_id
 JOIN foundentityconcept sub_fec on sub_fe.id = sub_fec.found_entity_id AND sub_fec.concept_ontology = 'snomed'
 JOIN concept sub_c on sub_fec.concept_id = sub_c.concept_id and sub_fec.concept_ontology = sub_c.ontology
 -- get critical result communication details
 LEFT JOIN to_persons tp ON cr.id = tp.reported_event_id
 LEFT JOIN from_persons frp ON cr.id = frp.reported_event_id
 LEFT JOIN _methods m ON cr.id = m.reported_event_id
 LEFT JOIN time_exprs te ON cr.id = te.reported_event_id
 -- get document info
 JOIN document d ON cr.document_id = d.id
 JOIN foundentitylocation fel ON sub_fe.id = fel.found_entity_id
 JOIN location l on fel.location_id = l.id AND l.type_='sentence'
 JOIN sentencelocation sl ON l.location_id = sl.id
ORDER BY cr.document_id, sub_fes.start;

Which produces a table of results with the following columns.

 subject | concept | concept_id | to_entities | from_entities | methods | time_expressions | event_polarity | sentence | filename
---------+---------+------------+-------------+---------------+---------+------------------+----------------+----------+----------

Java SDK updates

The latest NLP API Java SDK is version 5.9.0. Refer to the changelog for the Java SDK for a full explanation of the updates made since the last public release. You will need to use version 5.9.0 or later to take advantage of the new features described above.