Grammar-constrained decoding for structured information extraction with fine-tuned generative models applied to clinical trial abstracts
BackgroundIn the field of structured information extraction, there are typically semantic and syntactic constraints on the output of information extraction (IE) systems. These constraints, however, can typically not be guaranteed using standard (fine-tuned) encoder-decoder architectures. This has le...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2025-01-01
|
| Series: | Frontiers in Artificial Intelligence |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/frai.2024.1406857/full |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841556653022904320 |
|---|---|
| author | David M. Schmidt Philipp Cimiano |
| author_facet | David M. Schmidt Philipp Cimiano |
| author_sort | David M. Schmidt |
| collection | DOAJ |
| description | BackgroundIn the field of structured information extraction, there are typically semantic and syntactic constraints on the output of information extraction (IE) systems. These constraints, however, can typically not be guaranteed using standard (fine-tuned) encoder-decoder architectures. This has led to the development of constrained decoding approaches which allow, e.g., to specify constraints in form of context-free grammars. An open question is in how far an IE system can be effectively guided by a domain-specific grammar to ensure that the output structures follow the requirements of a certain domain data model.MethodsIn this work we experimentally investigate the influence of grammar-constrained decoding as well as pointer generators on the performance of a domain-specific information extraction system. For this, we consider fine-tuned encoder-decoder models, Longformer and Flan-T5 in particular, and experimentally investigate whether the addition of grammar-constrained decoding and pointer generators improve information extraction results. Toward this goal, we consider the task of inducing structured representations from abstracts describing clinical trials, relying on the C-TrO ontology to semantically describe the clinical trials and their results. We frame the task as a slot filling problem where certain slots of templates need to be filled with token sequences occurring in the input text. We use a dataset comprising 211 annotated clinical trial abstracts about type 2 diabetes and glaucoma for training and evaluation. Our focus is on settings in which the available training data is in the order of a few hundred training examples, which we consider as a low-resource setting.ResultsIn all our experiments we could demonstrate the positive impact of grammar-constrained decoding, with an increase in F1 score of pp 0.351 (absolute score 0.413) and pp 0.425 (absolute score 0.47) for the best-performing models on type 2 diabetes and glaucoma datasets, respectively. The addition of the pointer generators had a detrimental impact on the results, decreasing F1 scores by pp 0.15 (absolute score 0.263) and pp 0.198 (absolute score 0.272) for the best-performing pointer generator models on type 2 diabetes and glaucoma datasets, respectively.ConclusionThe experimental results indicate that encoder-decoder models used for structure prediction for information extraction tasks in low-resource settings clearly benefit from grammar-constrained decoding guiding the output generation. In contrast, the evaluated pointer generator models decreased the performance drastically in some cases. Moreover, the performance of the pointer models appears to depend both on the used base model as well as the function used for aggregating the attention values. How the size of large language models affects the performance benefit of grammar-constrained decoding remains to be more structurally investigated in future work. |
| format | Article |
| id | doaj-art-868d40a1fc494b5581ea0b5c4f4b8d2c |
| institution | Kabale University |
| issn | 2624-8212 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Artificial Intelligence |
| spelling | doaj-art-868d40a1fc494b5581ea0b5c4f4b8d2c2025-01-07T06:45:46ZengFrontiers Media S.A.Frontiers in Artificial Intelligence2624-82122025-01-01710.3389/frai.2024.14068571406857Grammar-constrained decoding for structured information extraction with fine-tuned generative models applied to clinical trial abstractsDavid M. SchmidtPhilipp CimianoBackgroundIn the field of structured information extraction, there are typically semantic and syntactic constraints on the output of information extraction (IE) systems. These constraints, however, can typically not be guaranteed using standard (fine-tuned) encoder-decoder architectures. This has led to the development of constrained decoding approaches which allow, e.g., to specify constraints in form of context-free grammars. An open question is in how far an IE system can be effectively guided by a domain-specific grammar to ensure that the output structures follow the requirements of a certain domain data model.MethodsIn this work we experimentally investigate the influence of grammar-constrained decoding as well as pointer generators on the performance of a domain-specific information extraction system. For this, we consider fine-tuned encoder-decoder models, Longformer and Flan-T5 in particular, and experimentally investigate whether the addition of grammar-constrained decoding and pointer generators improve information extraction results. Toward this goal, we consider the task of inducing structured representations from abstracts describing clinical trials, relying on the C-TrO ontology to semantically describe the clinical trials and their results. We frame the task as a slot filling problem where certain slots of templates need to be filled with token sequences occurring in the input text. We use a dataset comprising 211 annotated clinical trial abstracts about type 2 diabetes and glaucoma for training and evaluation. Our focus is on settings in which the available training data is in the order of a few hundred training examples, which we consider as a low-resource setting.ResultsIn all our experiments we could demonstrate the positive impact of grammar-constrained decoding, with an increase in F1 score of pp 0.351 (absolute score 0.413) and pp 0.425 (absolute score 0.47) for the best-performing models on type 2 diabetes and glaucoma datasets, respectively. The addition of the pointer generators had a detrimental impact on the results, decreasing F1 scores by pp 0.15 (absolute score 0.263) and pp 0.198 (absolute score 0.272) for the best-performing pointer generator models on type 2 diabetes and glaucoma datasets, respectively.ConclusionThe experimental results indicate that encoder-decoder models used for structure prediction for information extraction tasks in low-resource settings clearly benefit from grammar-constrained decoding guiding the output generation. In contrast, the evaluated pointer generator models decreased the performance drastically in some cases. Moreover, the performance of the pointer models appears to depend both on the used base model as well as the function used for aggregating the attention values. How the size of large language models affects the performance benefit of grammar-constrained decoding remains to be more structurally investigated in future work.https://www.frontiersin.org/articles/10.3389/frai.2024.1406857/fullgrammar-constrained decodingstructured information extractionclinical trialsdeep learninggenerative large language modelsPICO |
| spellingShingle | David M. Schmidt Philipp Cimiano Grammar-constrained decoding for structured information extraction with fine-tuned generative models applied to clinical trial abstracts Frontiers in Artificial Intelligence grammar-constrained decoding structured information extraction clinical trials deep learning generative large language models PICO |
| title | Grammar-constrained decoding for structured information extraction with fine-tuned generative models applied to clinical trial abstracts |
| title_full | Grammar-constrained decoding for structured information extraction with fine-tuned generative models applied to clinical trial abstracts |
| title_fullStr | Grammar-constrained decoding for structured information extraction with fine-tuned generative models applied to clinical trial abstracts |
| title_full_unstemmed | Grammar-constrained decoding for structured information extraction with fine-tuned generative models applied to clinical trial abstracts |
| title_short | Grammar-constrained decoding for structured information extraction with fine-tuned generative models applied to clinical trial abstracts |
| title_sort | grammar constrained decoding for structured information extraction with fine tuned generative models applied to clinical trial abstracts |
| topic | grammar-constrained decoding structured information extraction clinical trials deep learning generative large language models PICO |
| url | https://www.frontiersin.org/articles/10.3389/frai.2024.1406857/full |
| work_keys_str_mv | AT davidmschmidt grammarconstraineddecodingforstructuredinformationextractionwithfinetunedgenerativemodelsappliedtoclinicaltrialabstracts AT philippcimiano grammarconstraineddecodingforstructuredinformationextractionwithfinetunedgenerativemodelsappliedtoclinicaltrialabstracts |