Publications

Ralf Biedert, Georg Buscher, Sven Schwarz, Manuel Möller, Andreas Dengel, Thomas Lottermann: “The Text 2.0 Framework” to appear in Proc. of the International Workshop on Eye Gaze in Intelligent Human Machine Interaction at the International Conference on Intelligent User Interfaces (IUI2010), Hong Kong, China,  7-10 February 2010

Abstract: We created a simple-to-use framework to construct gaze-responsive applications using web technology focussing on text. A plugin enables any compatible browser to interpret a new set of gaze handlers that behave similar to existing HTML and JavaScript mouse and keyboard event facilities. Keywords like onFixation, onGazeOver, and onRead can be attached to parts of the DOMtree and are triggered on the corresponding viewing behavior. The plugin is part of a distributed architecture featuring a remote gaze provider and a number of assisting services and tools. Using this framework we implemented a number of applications providing help on comprehension difficulties.

Daniel Sonntag, Martin Huber, Manuel Möller, Alassane Ndiaye, Sonja Zillner, Alexander Cavallaro: “Design and Implementation of a Semantic Dialogue System for Radiologists”, to appear as a book chapter in “Semantic Web: Standards, Tools and Ontologies”, Nova Science Publishers, Inc., 2010

Abstract: This chapter describes a semantic dialogue system for radiologists in a comprehensive case study within the large-scale MEDICO project. MEDICO addresses the need for advanced semantic technologies in the search for medical image and patient data. The objectives are, first, to enable a seamless integration of medical images and different user applications by providing direct access to image semantics, and second, to design and implement a multimodal dialogue shell for the radiologist. Speech-based semantic image retrieval and annotation of medical images should provide the basis for help in clinical decision support and computer aided diagnosis.

We will describe the clinical workflow and interaction requirements and focus on the design and implementation of a multimodal user interface for patient/image search or annotation and its implementation while using a speech-based dialogue shell.  Ontology modeling provides the backbone for knowledge representation in the dialogue shell and the specific medical application domain; ontology structures are the communication basis of our combined semantic search and retrieval architecture which includes the MEDICO server, the triple store, the semantic search API, the medical visualization toolkit MITK, and the speech-based dialogue shell, amongst others. We will focus on usability aspects of multimodal applications, our storyboard and the implemented speech and touchscreen interaction design.

Daniel Sonntag, Manuel Möller: “A Multimodal Dialogue Mashup for Medical Image
Semantics”, to appear in Proceedings of the International Conference on Intelligent User Interfaces (IUI 2010), Hong Kong, China, 7.-10. Februar 2010

Abstract: This paper presents a multimodal dialogue mashup where different users are involved in the use of different user interfaces for the annotation and retrieval of medical images. Our solution is a mashup that integrates a multimodal interface for speech-based annotation of medical images and dialogue-based image retrieval with a semantic image annotation tool for manual annotations on a desktop computer. A remote RDF repository connects the annotation and querying task into a common framework and serves as the semantic backend system for the advanced multimodal dialogue a radiologist can use.

Sascha Seifert, Michael Kelm, Manuel Möller, Saikat Mukherjee, Alexander Cavallaro, Martin Huber, Dorin Comaniciu: “Semantic Annotation of Medical Images”, to appear in Proc. of SPIE Medical Imaging, San Diego, 13 – 18 February 2010.

Abstract: Diagnosis and treatment planning for patients can be signicantly improved by comparing with clinical images of other patients with similar anatomical and pathological characteristics. This requires the images to be annotated using common vocabulary from clinical ontologies. Current approaches to such annotation are typically manual, consuming extensive clinician time, and cannot be scaled to large amounts of imaging data in hospitals. On the other hand, automated image analysis while being very scalable do not leverage standardized semantics and thus cannot be used across specic applications. In our work, we describe an automated and context-sensitive work based on an image parsing system complemented by an ontology-based context-sensitive annotation tool. An unique characteristic of our framework is that it brings together the diverse paradigms of machine learning based image analysis and ontology based modeling for accurate and scalable semantic image annotation.

Daniel Sonntag, Manuel Möller: “Unifying Semantic Annotation and Querying in Biomedical Image Repositories”, to appear in Proc. of the International Conference on Knowledge Management and Information Sharing (KMIS), Madeira, Portugal, 6 – 8 October 2009

Abstract: In the medical domain, semantic image retrieval can provide the basis for a new generation of sophisticated decision support and computer aided diagnosis systems. However, the acquisition of the necessary medical knowledge about the image contents poses new problems. We present a set of techniques for annotating images and querying image data sets, based on image semantics.  The unification of semantic annotation (using a GUI) and querying (using natural dialogue) in biomedical image repositories is based on a unified view on the knowledge acquisition process. At the core, this system uses central RDF repository to capture both medical domain knowledge as well as image annotations. We understand medical knowledge engineering as an interactive process between the knowledge engineer and the clinician. Our system supports the knowledge engineering in an interative process between the dialogue engineer and the clinician.

Möller, M.; Vyas, N.; Sintek, M.; Regel, S. & Mukherjee, S.: “Visual Query Construction for Cross-Modal Semantic Retrieval of Medical Information”, to appear in Proc. of the Malaysian Joint Conference on Artificial Intelligence (MJCAI), 14th July – 16th July 2009, [PDF]

Abstract: We present an application for semantic annotation and retrieval of medical images. It leverages the MEDICO ontology which covers formal background information from various biomedical ontologies such as the Foundational Model of
Anatomy (FMA), terminologies like ICD-10 and RadLex and also includes various aspects of clinical procedures. By annotating available data semantically our system gains a number of features not available in existing keyword-based search engines. Search can be performed independent of the modality. Thus, x-ray images, Computed Tomography 3D volume data sets and medical records in text format can be searched for the same medical concepts in a unified manner. The work presented here focuses on our approach for an incremental generation of semantic queries. Our technique does not require prior knowledge about structure and available concepts in the ontology. Instead, it allows the user to construct complex queries by exploring the formal background knowledge from the ontologies and to create queries using formal concepts. In this context we also present our approach for the mapping of RadLex and FMA. It allows to combine the rich semantic modeling of the FMA for query expansion with the lightweight radiology-oriented RadLex hierarchy for the user interface.

Björn Forcher, Manuel Möller, Michael Sintek, Thomas Roth-Berghofer: “Explanation of Semantic Search Results of Medical Images in THESEUS MEDICO”, to appear in Proc. of IJCAI Workshop on Explanation-Aware Computing EXACT 2009, Pasadena, CA, July 11-13 2009

Abstract: The research project THESEUS MEDICO aims at developing an intelligent, robust and scalable semantic search engine for medical images and is designated for different kinds of users such as medical doctors, medical IT professionals or patients. Since semantic search results are not always self-explanatory various kinds of explanation are necessary to satisfy different user goals. Our prime concern is to give understandable justications for inexperienced users in the medical domain using semantic networks as form of depiction. In addition, we provide several interaction styles enabling a deeper insight into the medical knowledge.

Pinar Wennerberg, Manuel Möller,  Sonja Zillner: “A Linguistic Approach to Aligning Representations of Human Anatomy and Radiology”, to appear in Proceedings of the International Conference on Biomedical Ontologies (ICBO), Buffalo, NY, USA, July 24-26, 2009, [PDF on Nature Precedings]

Abstract: In the context of medical imaging different domain ontologies are necessary that provide complementary knowledge about anatomy and radiology. This is essential for realizing applications such as medical image search. Consequently, semantic integration of these different but nevertheless related types of medical knowledge from disparate domain ontologies becomes necessary. In our work we interpret semantic integration as aligning a taxonomy on radiology and an ontology on human anatomy to find equivalent concepts that represent their shared view on medical imaging. The resulting alignments describing this common view can then be used to annotate medical images and related textual patient data. Our alignment approach has three main aspects: (a) linguistic-based, (b) corpus-based, and (c) dialogue-based. In this paper, we describe the application of the first aspect on a representation of human anatomy and a representation of radiology and report on the results.

Manuel Möller, Christian Folz, Michael Sintek, Sascha Seifert, Pinar Wennerberg: “Extending the Foundational Model of Anatomy with Automatically Acquired Spatial Relations”, to appear in Proceedings of the International Conference on Biomedical Ontologies (ICBO), Buffalo, NY, USA, July 24-26, 2009, [PDF on Nature Precedings]

Abstract: Formal ontologies have gained a lot of impact in bioscience over the last ten years. Among them, the Foundational Model of Anatomy Ontology (FMA) is the most comprehensive model for the spatio-structural representation of human anatomy. In the research project THESEUS MEDICO we use the FMA as our main source of background knowledge about human anatomy. Our ultimate goals are to use spatial knowledge about anatomy the FMA to (1) improve automatic parsing algorithms for 3D volume data sets generated by Computed Tomography and Magnetic Resonance Imaging and (2) to generate semantic annotations using the concepts from the FMA to allow semantic search on medical image repositories. We argue that in this context more spatial relation instances are needed than currently available in the FMA.  We present a technique for the automatic inductive learning of missing spatial relation instances by generalizing from expert-annotated volume datasets. The result is stored using the formalism of the FMA and subsequently available for spatial reasoning.

Manuel Möller, Sven Regel, Michael Sintek: “RadSem: Semantic Annotation and Retrieval for Medical Images”, to appear in Proceedings of The 6th Annual European Semantic Web Conference (ESWC2009), Heraklion, Greece, 31 May – 4 June 2009

Abstract: We present a platform for semantic medical image annotation and retrieval. It leverages on the MEDICO ontology which covers formal background information from various biomedical ontologies such as the Foundational Model of Anatomy (FMA), terminologies like ICD-10 and RadLex and covers various aspects of clinical procedures. This ontology is used during several steps of annotation and retrieval: (1) We developed an ontology-driven metadata extractor for the medical image format DICOM. Its output contains, e. g., person name, age, image acquisition parameters, body region etc. (2) The output from (1) is used to simplify the manual annotation by providing intuitive visualizations and to provide a preselected subset of annotation concepts. Furthermore, the extracted metadata is linked together with anatomical annotations and clinical ndings to generate a unied view on a patient’s medical history. (3) On the search side we perform query expansion based on the structure of the medical ontologies. (4) Our ontology for clinical data management allows to link and combine patients, medical images and annotations together in a comprehensive result list. (5) The medical annotations are further extended by links to external sources like Wikipedia to provide additional information.