WP4 - EOSC data analysis services for EU Photon and Neutron national RIs

WP4 provides the coordination, adaptation and alignment of existing data analysis services at national RIs within the EOSC. The services delivered provide users, academics and the public with the ability to run analysis workflows against the EOSC-aligned data services.

Lead partner: DESY

Budgeted participation per partner:

DESY	PSI	Diamond	UKRI	SOLEIL	HZDR	MAX IV	EGI	ALBA	Elettra
45 PMs	18 PMs	18 PMs	18 PMs	18 PMs	18 PMs	18 PMs	9 PMs	9 PMs	7 PMs

Tasks

Progress in each of the work package's tasks are reported in this section, with regular updates, indicated by the dates. To view the original description of work of each task, as it is written in the project's description of work, you can click on the table header cells. Only the last update of the next steps is displayed.

T4.1	Kick-off meeting (DESY)
Task:	To be organised in conjunction with WP1 to define roles and process to coordinate between RIs, keep NGIs informed and connected to EOSC coordination. Expected outcome: list of roles, identities and meeting structure to deliver an operational framework with associated documentation.
Progress:	08 May 2020: An on site kick-off meeting took place as a satellite to the ExPaNDS kick-off meeting (9/2019), followed by an additional meeting at the 6-month mark in Hamburg, to coordinate the work between the contributing RIs and with PaNOSC (2/2020). Contributors agreed on a draft architecture that could integrate into the PaNOSC portal. Since end of March regular WP4 meeting take place (~1 every two weeks). WP4 also contributed to the general architecture description in relation to the EOSC deliverable published with WP3 and WP1 (03/2020). For EOSC coordination follow-up and NGIs reporting, EGI is directly involved in WP4. 20 Apr 2021: Discussions within T4.1 contributed towards the D4.1 organisational framework document defining the organisational framework for during and after the project and guidelines for integrating existing analysis services into EOSC through the PaN portal. This task is now completed. 06 May 2022: This task is now completed.

T4.1

Kick-off meeting (DESY)

Task:

To be organised in conjunction with WP1 to define roles and process to coordinate between RIs, keep NGIs informed and connected to EOSC coordination. Expected outcome: list of roles, identities and meeting structure to deliver an operational framework with associated documentation.

Progress:

08 May 2020:
An on site kick-off meeting took place as a satellite to the ExPaNDS kick-off meeting (9/2019), followed by an additional meeting at the 6-month mark in Hamburg, to coordinate the work between the contributing RIs and with PaNOSC (2/2020). Contributors agreed on a draft architecture that could integrate into the PaNOSC portal.
Since end of March regular WP4 meeting take place (~1 every two weeks).
WP4 also contributed to the general architecture description in relation to the EOSC deliverable published with WP3 and WP1 (03/2020). For EOSC coordination follow-up and NGIs reporting, EGI is directly involved in WP4.
20 Apr 2021:
Discussions within T4.1 contributed towards the D4.1 organisational framework document defining the organisational framework for during and after the project and guidelines for integrating existing analysis services into EOSC through the PaN portal.
This task is now completed.
06 May 2022:
This task is now completed.

T4.2	Select candidate EOSC data analysis services (DESY)
Task:	Select candidate data analysis services by evaluating their impact on the broadest Photon and Neutron European community, as well as their readiness for integration into the EOSC-hub using PaNOSC services. Simultaneously evaluate the current readiness of EOSC and PaNOSC services. This will involve attending PaNOSC and other related activities to acquire the necessary knowledge to adapt the candidate services for integration into EOSC-hub.
Progress:	08 May 2020: A 1-page description to propose a demonstration service to be implemented into EOSC is currently being elaborated by each site, with a matching data set (see T4.4). 20 Apr 2021: The selection from day one of ExPaNDS data services that are aligned with the users workflow will create a desire by facility users to keep those services maintained after the ExPaNDS project funding period ends. One outcome of the analysis of the users workflows was the observation that only 10% or so of them involved Jupyter notebooks, while the majority relied on community software accessed through remote desktops, and that in many cases analysis workflows made use of HPC resources to fulfil compute and storage requirements. We therefore concluded that focussing on Jupyter-based analysis services would have limited application. Instead, ExPaNDS WP4 will focus on implementing existing community data analysis workflows in portable facility-independent containers against HPC back-end infrastructure. The selection of 11 data analysis services representative of our users needs has proceeded in parallel with the selection of appropriate reference datasets (T4.4). The selected workflows are described in D4.2, along with their technique. This task is now completed. 06 May 2022: This task is now completed.

T4.2

Select candidate EOSC data analysis services (DESY)

Task:

Select candidate data analysis services by evaluating their impact on the broadest Photon and Neutron European community, as well as their readiness for integration into the EOSC-hub using PaNOSC services. Simultaneously evaluate the current readiness of EOSC and PaNOSC services. This will involve attending PaNOSC and other related activities to acquire the necessary knowledge to adapt the candidate services for integration into EOSC-hub.

Progress:

08 May 2020:
A 1-page description to propose a demonstration service to be implemented into EOSC is currently being elaborated by each site, with a matching data set (see T4.4).
20 Apr 2021:
The selection from day one of ExPaNDS data services that are aligned with the users workflow will create a desire by facility users to keep those services maintained after the ExPaNDS project funding period ends. One outcome of the analysis of the users workflows was the observation that only 10% or so of them involved Jupyter notebooks, while the majority relied on community software accessed through remote desktops, and that in many cases analysis workflows made use of HPC resources to fulfil compute and storage requirements. We therefore concluded that focussing on Jupyter-based analysis services would have limited application. Instead, ExPaNDS WP4 will focus on implementing existing community data analysis workflows in portable facility-independent containers against HPC back-end infrastructure.
The selection of 11 data analysis services representative of our users needs has proceeded in parallel with the selection of appropriate reference datasets (T4.4). The selected workflows are described in D4.2, along with their technique. This task is now completed.
06 May 2022:
This task is now completed.

T4.3	Alignment of PaNOSC services (Diamond)
Task:	Provide input to, and negotiate with, PaNOSC and other related activities to develop the alignment needed for the integration of data analysis services developed at the national RIs.
Progress:	08 May 2020: The deployment of the PaNOSC portal at ExPaNDS sites was agreed, it is beneficial for us because it represents a step on the right direction for our users, towards integrated remote analysis solutions. Diamond’s integration is advanced enough that it will be demonstrated at the PaNOSC review meeting in June ‘20. For other facilities having to address challenges concerning AAI, data access and HPC resources access, WP4 is picking ‘real world’ use cases at the national RIs and is working to adapt them and make them work in a services model. ExPaNDS partners also agreed in Jan. ‘20 to continue populating the PaN software catalogue. Since the start of the project, WP4 contributors regularly attend PaNOSC’s regular meetings which take place every two weeks. 20 Apr 2021: In addition to the regular meetings that continue to take place between ExPaNDS and PaNOSC, a technical workshop was organised by ALBA in October 2020 on the portal deployment at PaN sites. The preparation of demonstration cases for this workshop and then for the review meeting rehearsal in April 2021 proved highly useful, highlighting key issues to be solved and accelerating progress. 06 May 2022: There will be no portal, all sides will provide to EOSC.
Next steps:	06 May 2022: Alignment with PaN services (ELI) and PaNOSC services.

T4.4	PaN reference data sets (UKRI)
Task:	Identify, prepare and publish Photon and Neutron reference data sets, which can be used to adapt, align and validate the Prototype Data Analysis Services.
Progress:	08 May 2020: This task is much linked to T4.2 and the data sets are being identified along with the matching services. The best way to upload and store them is currently being investigated. 20 Apr 2021: The selected workflows and associated reference data sets were published in D4.2 (see T4.2). 11 reference data sets were selected, from 7 facilities and covering a range of techniques accross neutron and photon science. The actual data was made accessible through the links provided in the deliverable. This task is now completed. 06 May 2022: This task is now completed.

T4.4

PaN reference data sets (UKRI)

Task:

Identify, prepare and publish Photon and Neutron reference data sets, which can be used to adapt, align and validate the Prototype Data Analysis Services.

Progress:

08 May 2020:
This task is much linked to T4.2 and the data sets are being identified along with the matching services. The best way to upload and store them is currently being investigated.
20 Apr 2021:
The selected workflows and associated reference data sets were published in D4.2 (see T4.2). 11 reference data sets were selected, from 7 facilities and covering a range of techniques accross neutron and photon science. The actual data was made accessible through the links provided in the deliverable. This task is now completed.
06 May 2022:
This task is now completed.

T4.5	Testing and validation framework (MAX IV)
Task:	Establish a continuous test and validation framework which assures that the data analysis services can be validated against the reference data sets. This will serve to make sure that the services are working correctly as the different prototypes are adapted by the various partners.
Progress:	20 Apr 2021: We are currently in the process of defining testing requirements based on the services outline described in T4.3, and are simultaneously determining the technical requirements for efficiently implementing those tests. As a first step a Jupyter Notebook validation CI was developed at MAX IV and is to be presented to the WP at the end of April 2021. 06 May 2022: This task is now completed.

T4.5

Testing and validation framework (MAX IV)

Task:

Establish a continuous test and validation framework which assures that the data analysis services can be validated against the reference data sets. This will serve to make sure that the services are working correctly as the different prototypes are adapted by the various partners.

Progress:

20 Apr 2021:
We are currently in the process of defining testing requirements based on the services outline described in T4.3, and are simultaneously determining the technical requirements for efficiently implementing those tests. As a first step a Jupyter Notebook validation CI was developed at MAX IV and is to be presented to the WP at the end of April 2021.
06 May 2022:
This task is now completed.

T4.6	Prototype data analysis services (SOLEIL)
Task:	Adapt the candidate data analysis services to comply with the EU Photon and Neutron Ontologies provided via metadata catalogue services implemented in WP3. Adapt the candidate data analysis services to the APIs, standards and data lifecycle management guidelines provide by WP3. Adapt the candidate data analysis services to use EOSC services such as browser driven remote desktops and Jupyter analysis services.
Progress:	20 Apr 2021: We are currently in the process of implementing the infrastructure required for integration of NRI analysis services into the PaN portal as outlined in T4.3. 06 May 2022: This task is now completed.

T4.6

Prototype data analysis services (SOLEIL)

Task:

Adapt the candidate data analysis services to comply with the EU Photon and Neutron Ontologies provided via metadata catalogue services implemented in WP3. Adapt the candidate data analysis services to the APIs, standards and data lifecycle management guidelines provide by WP3. Adapt the candidate data analysis services to use EOSC services such as browser driven remote desktops and Jupyter analysis services.

Progress:

20 Apr 2021:
We are currently in the process of implementing the infrastructure required for integration of NRI analysis services into the PaN portal as outlined in T4.3.
06 May 2022:
This task is now completed.

T4.7	Deploy data analysis services into the EOSC-hub (DESY)
Task:	Develop validation criteria for deployment of the data analysis services within EOSC. Test the services by inviting test candidates from the user community to use them and provide feedback to the developers. Feed necessary adjustments back to the developers, keeping in mind the application of the service is intended for a wider scope than the prototype case. Anchor the outcome of task T.4.6 within the national RIs organisation and report back the results to WP1. To ensure consistent development of data analysis services, provide well documented usage examples to WP5 which demonstrate the mainstreaming of standards for data management and certification schemes for data repositories, and all relevant supporting activities within the data analysis services.
Progress:	20 Apr 2021: This task is still pending. Rules and best practices for integration of services into EOSC has been outlined in D4.1, while the architecture for accessing NRI analysis services via the PaN portal has been described in relation to T4.3. 06 May 2022: Integration in testing framework? Data policy, access policy, helpdesk and monitoring for EOSC
Next steps:	06 May 2022: Provide training material: workflows.

Deliverables

"Accepted" deliverables are the ones approved by the European Commission and thus published in CORDIS with other EU projects results.

"Delivered" deliverables are the ones we submitted to the European Commission but are not yet approved. They can only be found in our Zenodo community for now.

"Pending" deliverables are still in progress and were not yet submitted by the project to the European Commission.

Accepted deliverables		Partner	Date
4.1	Guidelines for implementing the national RI’s analysis services within the EOSC	DESY	29 Mar 2021
4.2	Photon and Neutron reference data sets prepared and published	UKRI	28 Feb 2021

Delivered deliverables		Partner	Date
4.3	Testing and Validation framework	MAX IV	30 Nov 2021
4.4	Analysis Services	SOLEIL	28 Mar 2022
4.5	Deployment of Analysis Services for EOSC within the EOSC-hub	DESY	30 Dec 2022

Pending deliverables		Partner	Date
No pending deliverables at the moment

Milestones

Achieved milestones		Partner	Date
14	Analysis services prototypes	DESY	28 Mar 2022
15	Analysis services for EOSC	DESY	30 Dec 2022

Pending milestones		Partner	Date
No pending milestones at the moment