Related services

• AI model evaluation and benchmarking on genomic datasets

  Karolinska Institutet (KI)

  ## Overview This service provides a comprehensive evaluation and benchmarking report for SMEs' AI models, leveraging genomic research datasets. Following agreed-upon protocols and recommendations, the AI model is rigorously tested against curated datasets. This occurs in a secure environment managed by Karolinska Institutet. SMEs submit their models to the Swedish TEF Health node under pre-signed agreements, ensuring full confidentiality. The evaluation and benchmarking process produces detailed insights, resulting in a report that supports SMEs in validating and refining their AI solutions. We provide expertise and technical support in the following areas: - Project design & management - AI model evaluation and benchmarking ### How can the service help you? This service helps address uncertainties about your AI model’s performance on external datasets. By providing thorough validation, it demonstrates the maturity and reliability of your model. The resulting report can serve as a valuable tool to build trust and confidence among stakeholders, supporting your efforts to showcase the model’s capabilities. ### How the service will be delivered? The service will be delivered according to established ethical agreements and guidelines, in collaboration with the SME and researchers from the Swedish TEF-Health node. Data usage is facilitated through a secure virtual environment managed by Karolinska Institutet, ensuring the highest standards of data protection. --- ## Additional information ### Provider description The Swedish TEF-Health node is a collaboration between Karolinska Institutet, SciLifeLab and RISE, and is led by Karolinska Institutet. Together, we offer world-leading services with our unique collection of core facilities. We can grant services in expert consulting, virtual- and physical testing in the range of in vivo imaging, ex vivo OMICS, pharmaceutical development, simulated healthcare environments, AI-system validation and development, advanced data analysis and other data-driven life science. ### Technical description The service evaluates AI models in a secure environment using high-performance computing infrastructure optimized for large genomic datasets. Models are tested against curated and annotated genomic datasets using state-of-the-art frameworks such as TensorFlow and scikit-learn. Evaluation metrics, including accuracy, precision, and recall, are calculated to benchmark performance against industry standards and reference models. Encrypted storage and strict access controls ensure data security. ### Service customization The service can be customized according to your specific needs. It may be required to combine this service with other services on offer. --- ## Use case example ### Context A biotech SME specializing in rare diseases has developed an AI model to predict genetic predispositions for a rare neurological disorder. The model was trained on internal datasets but has not been validated on external genomic data. Investors and clinical partners are reluctant to adopt the solution without independent evaluation and benchmarking to ensure its generalizability and maturity. ### Objective To validate the AI model’s accuracy and robustness on external genomic datasets, benchmark its performance against existing solutions, and deliver a detailed evaluation report to gain stakeholder trust and regulatory approval. ### Solution The SME submits its AI model to the Swedish TEF-Health node for evaluation. Using secure infrastructure and curated genomic datasets relevant to rare diseases, the model undergoes extensive testing and benchmarking against industry standards. ### Implementation #### Ethical Agreement The SME enters into an ethical agreement with researchers from the Swedish TEF-Health node, ensuring all data collection and usage complies with GDPR and national Swedish regulations. #### Secure Access Usage of the collected data is facilitated through a secure virtual environment managed by Karolinska Institutet, ensuring the highest standards of data protection. #### Evaluation and Benchmarking The model is assessed using metrics like sensitivity, specificity, and ROC-AUC, focusing on its performance in predicting rare disease risks. #### Outcome A benchmarking report is generated, highlighting the model's strengths, weaknesses, and recommendations for improvement. ### Benefits - **Validation & Credibility**: Independent validation enhances trust among clinical and regulatory stakeholders. - **Competitive Benchmarking**: Aligning with industry standards provides an advantage in the rare disease AI market. - **Model Refinement**: Insights from the report drive improvements for clinical readiness. ### Impact The SME secures stakeholder confidence, accelerates discussions with clinical partners, and positions its AI solution as a trusted tool for rare disease risk prediction, paving the way for market adoption and broader collaborations.

  View service →
• AI model evaluation/assessment: Clinical model validation

  Unidade Local De Saúde De Coimbra EPE (ULS Coimbra EPE)

  The service offers SMEs expert evaluation and validation of their AI models intended for clinical use. Leveraging the hospital's domain expertise in healthcare and data analytics, this service assesses the accuracy and clinical suitability of AI models in real-world clinical scenarios by assessing models against clinically relevant metrics, benchmarks, and regulatory standards, it ensures their safety, reliability, and effectiveness in real-world healthcare settings.

  View service →
• AI model evaluation/assessment: Clinical model validation

  Centro Hospitalar De Sao Joao Epe (CHSJ)

  The service offers SMEs expert evaluation and validation of their AI models intended for clinical use. Leveraging the hospital's domain expertise in healthcare and data analytics, this service assesses the accuracy and clinical suitability of AI models in real-world clinical scenarios by assessing models against clinically relevant metrics, benchmarks, and regulatory standards, it ensures their safety, reliability, and effectiveness in real-world healthcare settings.

  View service →
• AI Model performance evaluation

  Multitel (MULTITEL)

  This service provides an independent and reproducible evaluation of AI model performance using well-established quantitative metrics. The evaluation is conducted in a controlled and documented execution environment to ensure traceability and repeatability of results. Performance is assessed on client-provided datasets and models, and associated measurement uncertainties are systematically analyzed and reported. The service delivers a detailed and interpretable evaluation report. All activities are performed under ISO 9001 certified processes.

  View service →
• AI performance evaluation based on mixed testing environments

  Laboratoire National De Metrologie Et D'Essais (LNE)

  Based on the domain of the medical device or robot provided by the customer, the simulation will create a test environment relevant to the system to be evaluated using a video-projection system. The service will be provided using the LE.IA Immersion platform, a platform dedicated to the evaluation of AI. It consists of a video-projection system that makes it possible to reconstitute an "artificial nature" and to immerse the systems to be characterised (personal assistance robots, intelligent cameras, civil and military intervention robots, etc.) in a simulated dynamic reality. It allows the system (robot, camera, etc.) to be subjected to a multitude of test scenarios and to evaluate its reactions in a controlled environment.

  View service →
• AI performance evaluation based on testing datasets

  Laboratoire National De Metrologie Et D'Essais (LNE)

  To ensure that their solution works accordingly regarding a task, AI provider have to evaluate their system using a specific dataset. The performance obtained on this dataset helps to prove the adequate performance of their AI solution. However, the evaluation process can be often quite hard for an AI provider to do properly: the evaluation dataset needs to be correctly qualified, and the creation of the evaluation protocol as well as the analysis of the results are not easy tasks. This service allows AI provider to benefit of the LNE expertise with a full evaluation of their AI system: using an evaluation dataset created for the test or provided by a partner of the TEF project, an assessment of the performance of the AI system is done, and an analysis of its behavior provided. This work also includes a quality assessment of the evaluation dataset. The scope of the analysis of the evaluation results can include robustness and resilience evaluation, depending of the needs of the SMEs. With this service, the customer will have a full assessment of its solution, allowing them to answer the accuracy requirements of the AI regulation, while also having a full report following all transparency and reproductibility requirements. This service generally takes around 2 months, depending of the needs of the customers.

  View service →