Release notes for NLP API 2.35

Contents

• Brief Summary
• New Features
• Changes
• Python SDK updates
• Benchmarking

Brief Summary

Release versions

• NLP API version 2 Release 35.
• Python SDK and client v6.9.0.
• Java SDK and client v5.15.0.
• NLP API specification v2.20.0: added support for new processing feature entity-concept-link-confidence.
• JSON result output schema v2.33.0 for emtellipro-json-2: added support for new processing feature entity-concept-link-confidence and new output attribute heading_status and new location headings; added confidence scores for relations.
• visualclient version v2025.01.07-184b7f: added support for new processing feature entity-concept-link-confidence.

Highlights

• Confidence scores for SNOMED entity linking.
• Confidence scores for relations.
• New heading_status found entity field in output.
• New heading locations added to output.
• Page locations from OCR output on submitted PDF files added to output.
• Improved ICD-10-CM output to include useful mapping information that can be used to infer more accurate ICD-10 codes based on the document context.
• Updated experiencer, qualifier and temporality models for improved accuracy and improved GPU batching.

New Features

• New heading status attribute
• New heading locations
• New page locations
• Confidence Scores for SNOMED Concepts
• Relation Confidence Scores

New heading status attribute

The heading_status found entity attribute indicates if the entity is inside of or outside of a section heading. For example, if we have a document with a section like this:

Allergies:
LYRICA 75 MG CAPSULE - DEPRESSION

Then the NLP API will identify a found entity on “Allergies”, which will have a heading_status value of inside_heading. It will also identify found entities on the remaining text, which will have heading_status of outside_heading. See also heading locations. Truncated output JSON for the above is shown below.

1
{
2
        "entities": {
3
            "found": [
4
                {
5
                    "label": "E4",
6
                    "spans": [
7
                        {
8
                            "start": 0,
9
                            "end": 9
10
                        }
11
                    ],
12
                    "section_name": "ALLERGIES",
13
                    "attributes": {
14
                        "heading_status": "inside_heading",
15
                        "polarity": "asserted",
16
                        "uncertainty": "certain",
17
                        "question_status": "not_question",
18
                        "guidance": "not_guidance",
19
                        "known_ambiguity": "unknown"
20
                    },
21
                    "concept_links": [
22
                        "Cumls_loinc1",
23
                        "Cradlex1",
24
                        "Cumls_nci3",
25
                        "Cmedcin2"
26
                    ],
27
                    "locations": [
28
                        "H0",
29
                        "S2",
30
                        "SEC-1"
31
                    ],
32
                    "entity_type": {
33
                        "umls_nci": "Pathologic Function",
34
                        "umls_loinc": "Clinical Attribute",
35
                        "radlex": "clinical finding",
36
                        "medcin": "history"
37
                    },
38
                    "concept_confidences": {},
39
                    "text": [
40
                        "Allergies"
41
                    ]
42
                },
43
                {
44
                    "label": "E0",
45
                    "spans": [
46
                        {
47
                            "start": 18,
48
                            "end": 23
49
                        }
50
                    ],
51
                    "section_name": "ALLERGIES",
52
                    "attributes": {
53
                        "heading_status": "outside_heading",
54
                        "polarity": "asserted",
55
                        "uncertainty": "certain",
56
                        "question_status": "not_question",
57
                        "guidance": "not_guidance",
58
                        "measurement_unit": [
59
                            "MG"
60
                        ],
61
                        "known_ambiguity": "unknown"
62
                    },
63
                    "concept_links": [
64
                        "Cumls_loinc0",
65
                        "Cemtelligent0",
66
                        "Csnomed0"
67
                    ],
68
                    "locations": [
69
                        "S3",
70
                        "SEC-1"
71
                    ],
72
                    "entity_type": {
73
                        "emtelligent": "measurement",
74
                        "umls_loinc": "Clinical Attribute",
75
                        "snomed": "attribute",
76
                        "umls": "Quantitative Concept"
77
                    },
78
                    "concept_confidences": {},
79
                    "text": [
80
                        "75 MG"
81
                    ]
82
                }
83
            ]
84
        }
85
}

New heading locations

The NLP API already had location items for sections in the document. It now also has locations for headings that are found to identify sections. For example, using this document part again:

Allergies:
LYRICA 75 MG CAPSULE - DEPRESSION

The NLP API will find one section, which has a heading Allergies. Results will therefore have one section location and one heading location, with heading and section JSON properties providing cross-reference between them. For example, as shown in the JSON fragment below, any entity in the heading will have H0 in its locations, while any entities in the remaining text will not. See also the heading status attribute. Truncated output JSON is shown below.

1
{
2
        "locations": {
3
            "sections": [
4
                {
5
                    "label": "SEC-1",
6
                    "spans": [
7
                        {
8
                            "start": 0,
9
                            "end": 45
10
                        }
11
                    ],
12
                    "name": "ALLERGIES",
13
                    "level": 0,
14
                    "parent": null,
15
                    "heading": "H0",
16
                    "text": [
17
                        "Allergies:\nLYRICA 75 MG CAPSULE - DEPRESSION\n"
18
                    ]
19
                }
20
            ],
21
            "headings": [
22
                {
23
                    "label": "H0",
24
                    "spans": [
25
                        {
26
                            "start": 0,
27
                            "end": 9
28
                        }
29
                    ],
30
                    "section": "SEC-1",
31
                    "text": [
32
                        "Allergies"
33
                    ]
34
                }
35
            ]
36
        }
37
}

Note that not all sections have headings. For example, text at the beginning of a document without an initial heading will be grouped as an intro section, giving a section location with no corresponding heading location. Similarly, if single section labels are used in input, then the NLP API just uses the sections thus specified, and since these sections do not have headings in the text there will be no heading locations. In these cases the heading property on the section locations is null.

New page locations

If the input has pages (currently only in the case of PDF input), then the result will include page locations similar to other location types. Page locations have a JSON property page_index which gives the index into the ordered sequence of pages (starting from zero). Section locations have a new JSON property pages which cross-references any pages a section is on. Found entities on a page will have a reference to the page in their locations (potentially more than one, if the span more than one page). Truncated output JSON for a document with pages is shown below.

1
{
2
        "locations": {
3
            "pages": [
4
                {
5
                    "label": "P1",
6
                    "spans": [
7
                        {
8
                            "start": 0,
9
                            "end": 645
10
                        }
11
                    ],
12
                    "page_index": 0,
13
                    "text": [
14
                        "FINAL \n\nPHOENIX CHILDREN'S HOSPITAL \nDiagnostic Procedure Report \n\nPatient: REDACTED, GIRL \nMRN: 1236 \nDOB: 03/30/2000 \nSex: F \n\nOrder #: 5081074 \nProcedure Description: XR Chest 1 View AP or PA \nReason For Procedure: Evaluate lung fields \nDate: 12/10/2000 \nVisit #: 41236 \nLocation: 511 \n\nAttending Physician 1: ENGELHARDT, KEVIN \nAttending Physician 2:  \nOrdering Physician: Hann, Jennifer \nPrimary Care Physician: STEVENSON, PETER J \n\nEXAM: XR Chest 1 View AP or PA, 12/10/2020 4:28 AM \n\nHISTORY: Tracheostomy. \n\nAge: 8 months \n\nSex: Female \n\nCOMPARISON: 12/7/2020 \n\nTECHNIQUE: Single frontal view of the chest was performed. \n\nFINDINGS:  \n\n\f"
15
                    ]
16
                },
17
                {
18
                    "label": "P34",
19
                    "spans": [
20
                        {
21
                            "start": 645,
22
                            "end": 1496
23
                        }
24
                    ],
25
                    "page_index": 1,
26
                    "text": [
27
                        "LINES/TUBES/SURGICAL: Tracheostomy \n\ntube tip in trachea. \n\nVascular catheter tip at superior cavoatrial junction. \nPostsurgical changes in the mediastinum. \nGastrostomy tube. \nEnteric tube coiled in stomach, tip in proximal duodenum. \n\nLUNGS/AIRWAY: Mild coarse increased interstitial markings. Subsegmental right \nupper lobe atelectasis, unchanged. PLEURA: No pleural effusion. No \npneumothorax. HEART/MEDIASTINUM: Dextrocardia. UPPER ABDOMEN: Normal. \n\nBONES: Normal. \n\nOTHER: None. \n\nIMPRESSION:  \n\nUnchanged subsegmental right upper lobe atelectasis and mild coarse increased \ninterstitial markings. \n\nReviewed by: Eric Graham.  12/10/2020 7:48 AM. \n\nI, Dianna Bardo, personally reviewed the images and if necessary \nI edited the report. I agree with the report as now presented. \n\nElectronically signed by: Dianna Bardo.  12/10/2020 8:59 AM. \n\n\f"
28
                    ]
29
                }
30
            ],
31
            "sections": [
32
                {
33
                    "label": "SEC-0",
34
                    "spans": [
35
                        {
36
                            "start": 0,
37
                            "end": 67
38
                        }
39
                    ],
40
                    "name": "INTRO",
41
                    "level": 0,
42
                    "parent": null,
43
                    "heading": null,
44
                    "pages": [
45
                        "P1"
46
                    ],
47
                    "text": [
48
                        "FINAL \n\nPHOENIX CHILDREN'S HOSPITAL \nDiagnostic Procedure Report \n\n"
49
                    ]
50
                }
51
            ]
52
        }
53
}

Confidence Scores for SNOMED Concepts

The confidence scores produced by the NLP API for SNOMED concepts are not just based on the probabilities for the concept assigned to a mention. Rather, the confidence scores are based on the plausibility of the mention and the SNOMED concept ID assigned to the mention in context of the sentence.

To explain this consider the following sentence:

Tiny MASK in the pelvic veins .

where MASK is one of the three mentions below with the indicated SNOMED concept assigned to each mention and the confidence score assigned to each case.

1. MASK=phleboliths and SNOMED concept ID 37876005 (phlebolithiasis). Plausible mention in context, plausible concept ID in context. Gets confidence score of 0.97.
2. MASK=heart attack and SNOMED concept ID 37876005 (phlebolithiasis). Implausible mention in context, assigned a plausible concept ID in context. Gets confidence score of 0.46.
3. MASK=heart attack and SNOMED concept ID 22298006 (myocardial infarction). Implausible mention in context, implausible concept ID. Gets confidence score of 0.06.

The confidence score measures the compatibility of the mention, concept pair in the context of the document. For example, the confidence score measures how likely is the mention text “phleboliths” a plausible way to replace the mask token in “Tiny MASK in the pelvic veins”?

In the same way that the p-value from a statistical test measures the likelhood that a result could have occurred randomly, each of these scores represents the likelihood that the given pairing is more plausible than a randomly-chosen pair of inputs. Thus a confidence score of 0.97 for “phleboliths” in the context “tiny <mask> in the pelvic veins” implies that “phleboliths” is a more sensible insertion than 97% of all known SNOMED-CT synonyms in the given context.

If you wish to implement a threshold on the confidence scores returned by the NLP API then we recommend a threshold of 0.6 as the cutoff for “good” versus “bad” matches. This corresponds to a model-internal score of 0.75, or a 75% likelihood that the match is better than random.

Relation Confidence Scores

Relations for which confidence scores are available are reported in the NLP API output as a new attribute associated with each relation.

Here is a snippet of the NLP API output that includes the relation confidence scores:

1
"measurements": [
2
  {
3
    "label": "RMS0",
4
    "attributes": {
5
      "confidence": 0.9999994039535522
6
    },
7
    "args": {
8
      "value": {
9
        "ref": "E0",
10
        "text": [
11
          "97/52"
12
        ]
13
      },
14
      "subject": {
15
        "ref": "E3",
16
        "text": [
17
          "Blood pressure"
18
        ]
19
      }
20
    }
21
  }

If the confidence score for relations is unavailable, it will be reported as null.

The confidence scores for relations are not calibrated like the entity confidence scores for SNOMED concepts.

Changes

Ontology updates

• UMLS ontologies updated to 2024AB (November 4, 2024) release.
• SNOMED-CT ontology updated to US Edition May 2024 release.
• Added 5943 and changed 6239 annotations for SNOMED-CT entities.

Model updates

• Sentence boundary model enforces a maximum sentence length, including splitting on comma.
• New measurement model.
• Update to new specific temporality model.
• New qualifier relation model.
• Add person recognition for Physician Assistants and Nurse Practitioners.
• Address recognition improvements to improve detection of PIN codes and ZIP codes.

Negation

• Handle negation for internally negated annotations, e.g. CAD without angina now has polarity asserted instead of negated.
• Handle negation in cases like : No and : None etc.
• Fix negation regex for not indicated in context.
• Negation issue with lists that use hyphens as item markers.
• Improved negation detection for list coordination with or.

Section parsing

• Some new heading patterns in section parsing.
• Add clinical/lab results report kind.
• Restored radiology recommendation headings.
• New generic category for gastroenterology report category, with generic primary subcategory of endoscopy.

Bug fixes

• Fix section disambiguation in entitylinker.
• Handle cases like bp2 in mention spans.
• Fix bug in some cases of matching for plurals.
• Fix bug in address matching.
• Fix crashing bug in remove mention overlaps.
• Bug fix to not produce section categories labeled as !.

Requirements update

• Updated minimum Python version to 3.11.

Model Updates

• Qualifier relation.
• Experiencer relation.
• Temporality relation.
• Measurement relation.

Python SDK updates

The latest NLP API Python SDK at the time of release is version 6.9.0b5. Refer to the changelog for the Python SDK for a full explanation of the updates made since the last release. You will need to use at least version 5.7.0 of the Python SDK or later to send requests to the NLP API v2.29 but if you want to use the new features explained above you will need to use at least version 6.9.0b5.

New features in the Python SDK

• Concept confidence
• Entity heading status
• Page locations
• Heading locations
• Relation Confidence

$
emtellipro-db-client migrate --database postgresql://username:password@hostname:port/example_db

Benchmarking

• Processed 1744 CCDs.
• Total size: 34 MB; Average size: 67.35 KB.
• NLP API deployed with 4 GPU workers
• 32x CPU: Intel(R) Xeon(R) Silver 4310 CPU @ 2.10 - 3.30 GHz w/18 MB Cache; 189G RAM
• 1x GPU: Nvidia L4 w/24GB of RAM
• Processed 1744 reports in 35:11.43s for an average per report processing time of 1.210684s per report
• Saved 1744 documents to database in 1:42.3s (.058656s / document)
• Output JSON file size: 511MB
• NLP API database size using Python database client output to sqlite: 739MB