Project Executive Board meeting notes - extracts for WP4

14 Feb 2023

WP4 is organizing the VISA workshop next week. The registration closes tomorrow. No one from ILL has registered yet.

Majid thinks about going to the ICALEPS 2023 and presenting ExPaNDS.

31 Jan 2023

No other activities than the final report are planned. Picking up some work from the leaving people.

Including the work into the subgroups of LEAPS and OSCARS. A FAIR subgroup should be formed In LEAPS WG3 to have a forum, where people can talk about FAIR.

Majid will send a save-the-date email for the VISA meeting at SOLEIL, 23/02/ - 28/02/. The first two days will be a more general meeting about VISA, whereas the two last days will be a hands-on-meeting. Juliane will set up an Indico page.

The MoU should be open for all people. Even if they will not use VISA (e.g. PSI).

17 Jan 2023

WP4 is preparing the session for the closing event.

20 Dec 2022

Focussing on D4.5.

06 Dec 2022

Nothing to report. (D4.5 see above.)

22 Nov 2022

WP4 is bringing the last pieces of work into shape.

Silvan and some others are working on D4.5. The meetings are every friday at 10:00.

Majid reports about a joint WP4 event with PaNOSC, which should have a follow up at our final event. He asks Andy if someone from PaNOSC can attend the ExPaNDS final event for this topic. Andy suggests Jean-Luc and Erwan.

08 Nov 2022

The most important task is the last deliverable D4.5.

Majid is writing a MOU for a collaboration with PaNOSC and ExPaNDS. There will be a joint WP4 meeting in Grenoble. He is writing a document on how to proceed with VISA. Patrick is phrasing the text with ILL.

25 Oct 2022

Anton reports that people are drifting away at the end of the project. Furthermore he reminds people to work on the KPIs and the deliverables.

Paul reports about the ELIXIR platform. He heard about it in EOSC future. It is more about biological science, but we should keep it in mind for the review.

11 Oct 2022

Majid asked EGI for D4.5 for a way to have an entry point to the platform. Paul suggested the entry point “infrastructures” and use it for all services. Furthermore keywords like “PaN facilities” or “PaN infrastructure” can be used. EGI is willing to work on this entry point. It would be an icon on the EOSC platform.

Furthermore WP4 is preparing a document for all facilities who are interested in VISA and the continuation of the collaboration.

27 Sep 2022

Nothing to report.

13 Sep 2022

Email Majid

Nothing new from WP4 apart from the VISA meeting on Friday for which the organization seems to be finalized.

Email Anton

In addition to Majid’s report - we are going to ask all partners to summarise their efforts during the project and end status

A summary of the WP4-relevent KPIs would be useful so we ask the right questions so as to be able to efficiently answer those questions - I have them somewhere but good to have the ‘official’ list.

30 Aug 2022

Silvan is working on the use cases in a joint activity with Antoine to make the data (CFEL) openly available.

WP4 is preparing the VISA workshop (16th September).

Anton reports that UKRI is not convinced to register as an EOSC service provider. We need to encourage them to do!

16 Aug 2022

Anton is monitoring the tasks in his WP regularly in the meetings. It is important to focus on what the people want to deliver at the end of the project.

Abigail would like to know how many registrations from the facilities there are already for EOSC as a service provider. Answer: eight facilities (this is also a KPI)

This is also an important task for WP2, because it is also about the fair use policies to be implemented.

Anton shows the document, in which WP4 is monitoring all the tasks.

02 Aug 2022

Nothing to report-

19 Jul 2022

05 Jul 2022

Majid reports about the VISA workshop, which is being organised at the moment (see WP5).

D1.6 was reviewed and there is a final meeting on 08/07.

21 Jun 2022

Silvan speaks with Anton about being the coordinator for the fair assessment survey (for WP2).

07 Jun 2022

Silvan prepared the demo setups. Majid is contributing in D1.6. Majid would like to talk to Jamie about the compute API. The use case demo from XFEL containers will be discussed in 20-30 min.

24 May 2022

Nothing to report.

10 May 2022

The last meetings were dedicated to other WP’s (eg. WP5 and WP2)

Next steps will be the 2f2 meeting in June and the VISA and SciCat workshops in July.

26 Apr 2022

Nothing to report.

12 Apr 2022

Krisztian has set up a weekly meeting to discuss the WP4 part for D2.7. It takes place every Thursday at 3 pm. Everyone is invited to give input. Last friday the WP4/WP5 meeting took place. Majid suggested two use cases, one from DESY and one from SOLEIL. Kat suggested another tomography use case from Diamond, which is for the industry.

29 Mar 2022

The focus is more on visualisation than at the Jupyter notebook at the moment. A WP4-WP5 meeting is planned on the 08/04. This meeting aims to clarify the expectactions of EC and how WP5 could support WP4 activities (identify key materials to be posted to the platform and help WP4 to upload them, discuss WS/training that you need to organise to fulfill WP4 objectives and how these events can be showcased and promoted to EC with the support of WP5/6, submit you our ideas for WS/training ...)

15 Mar 2022

D4.4 is being reviewed at the moment. The technical architecture enhancements (#16) are discussed in regular meetings. The next meeting is on 7th April 2022.

01 Mar 2022

Majid will check issue #22 is covered by the testing framework during the next WP4 meeting. He hopes to have the deliverable on analysis prototypes (D4.4) finished by the end of week, reviewed and then published by mid-March. One pipeline is missing but will be done by Friday and a few facilities' infrastructures still need to be described.

WP4 is testing the use of Singularity instead of Docker to facilitate launching containers on different infrastructures.

15 Feb 2022

Majid is getting the upcoming deliverable together, the 5 pipelines to be made available in EOSC have been chosen and now each facility responsible for one needs to fill in the document.

Anton reported progress on the containerisation, working on the configuration environment to have consistent ways to mount Slurm e.g. and data repositories, so that it is as transparent as possible to the user.

Uwe noted it was difficult to get commitments from the different facilities, with “silent” participants in WP4 meetings being still an issue. We could be more forceful in asking for answers, taking for example the main objective for WP4 as a “contract” to be filled.

Uwe also suggested offering, in addition to training for the use of the 5 pipelines, specific training workshops on VISA.

01 Feb 2022

The main two tasks for WP4 right now are to update the general architecture with WP3 and WP1 and to write the prototype analysis service deliverable before the end of February. Majid discussed the outline of the deliverable (D4.4) this morning with Anton and it will be on the agenda of Friday’s WP4 meeting. It would be as follows:

The aim of the deliverable is not to provide 1 standard solution but rather recognise the diversity of the situation and describe some of the most interesting possibilities.

Majid proposed to have mixed reviewers between WP3 and WP4 so all three deliverables (architecture + prototype analysis + catalogue) are consistent.

18 Jan 2022

07 Dec 2021

23 Nov 2021

Anton reported on the ongoing work to document the jupyter workflows, with the objective to have a how-to manual to easily repeat the procedure of making notebooks portable.

Work on containerisation of non-jupyter workflows is also ongoing, mainly by Silvan and Michael at DESY. Again, this will also result in a how-to manual for reproducibility.

Anton also reported that PaNOSC (in particular Jamie) had been very constructive with VISA, which enabled good progress at both DESY and ALBA sites on testing out the analysis platform.

Several of our facilities are registering to the EOSC (already registered are: DESY, STFC, PSI and EGI). Uwe mentioned an issue with the EOSC AAI that HZDR had to solve, involving groups and authorisation for specific attributes to reveal to the EOSC AAI proxy. He will get in touch with Jean-François in PaNOSC for support. Exchange of experience (good and less good) on EOSC onboarding should be encouraged in any case.

09 Nov 2021

The testing framework is still pending but has been reviewed internally by Krisztian whose comments will be addressed now along with Patrick’s review.

Majid reported good progress on the analysis service prototype developed by SOLEIL. The Openstack infrastructure is now launched and Eric is working regularly with Jamie on the installation of VISA in addition to Jupyter services. The plan is to use this instance in the future at SOLEIL, integrated as the operational solution offered to users.

All were very happy to welcome Majid again in our PEBs.

26 Oct 2021

12 Oct 2021

The testing framework deliverable is in its final stages, it should be delivered before next PEB.

Patrick reported preparation works continuing for the ExPaNDS contribution to the EOSC-Synergy workshop at the EGI conference next Tuesday.

Anton reported he had been busy working on the DESY answer to the LEAPS questionnaire, which raised awareness on the concrete organisational and resources needs to sustain the outcomes of our project.

28 Sep 2021

The workplan is now picking up after a summer slowdown:

Abigail mentioned an interesting discussion at STFC concerning the testing of services once they are registered in EOSC. It would be interesting to include this in the ongoing deliverable on the testing framework. This will be discussed with Zdenek, Anton, Franz and the rest of WP4.

Sophie also added that EOSC-Synergy offered us to present ExPaNDS as a use case for their SQAaaS framework during this EGI conference workshop. Talks are ongoing with Zdenek and Michael to see what we can present, in the spirit of the mid-term review demo. There is a meeting tomorrow morning.

14 Sep 2021

Uwe suggested applying the principles from WP2-3 to WP4’s reference datasets, for more cross-WP consistency. e.g. identifying if these are FAIR and use the metadata schemas of the ontology and if not, identifying the gaps. To be discussed further with Anton during the next PEB.

Each WP4 facility is registering as a service provider in the EOSC portal. Uwe reported the process was slow at the moment on EOSC’s side, having started it a few weeks ago for HZDR. Abigail discussed this issue with other 5b projects and it is currently complicated for everyone. She offered to share the feedback collected.

17 Aug 2021

Following the mid-term review, Anton now has a clear idea on the next steps for WP4:

Andy informed us a parallel portal for the search API that had been developed at ELI is being considered to be taken over at the ESRF, which would be good sustainability news. This still needs to be clarified.

20 Jul 2021

06 Jul 2021

Following the onboarding procedure presentation by Andrea and Owen, Anton will now monitor the onboarding of the individual DA services of our partners in EOSC (N.B: it’s a KPI).

Anton is getting the reference use cases to work at as many facilities as possible, starting with CrystFEL, which is done by a new starter at DESY. This will be the first non-Jupyter example of an ExPaNDS cloud DA service.

Anton also mentioned its NFDI project for Germany-wide PaN data management had been granted. It will be an interesting related project for ExPaNDS.

22 Jun 2021

VISA is now the solution chosen by PaNOSC for the WP4-related functionalities of the PaN portal. For the search API, a parallel development made by ELI is to be used.

To discuss the fact that VISA’s functionalities do not fit all our ExPaNDS services (in particular: authenticated access only and every service needing to be launched via OpenStack), Anton, Patrick and Sophie had a dedicated meeting with Andy, Fabio and Jordi this morning. The result was that PaNOSC might help us fill the gaps by considering to:

This would notably allow us to use VISA also for the pipelines running on Slurm, thanks to the “Jupyter in Slurm” package developed by ESRF.

VISA was already made open source in GitHub by ILL. Contribution rules and an agreement on maintenance of the whole service will now be initiated. ILL intends to maintain the PaN VISA repositories up to date with the production ILL VISA, which is very good for sustainability.

All this still being developed and discussed, it was agreed that our upcoming testing framework shall be developed independently. Side remark, Andy confirmed that the ESRF was using MAX IV’s testing framework for Jupyter.

Anton reported that since several facilities are looking to onboard their data analysis platform directly in EOSC, Andrea will be presenting the onboarding procedure during the next WP4 meeting on Friday.

In terms of resources for WP4, Ana confirmed SOLEIL found a new person to work with Majid: Eric. Patrick also mentioned the recent interviews at DESY for a WP4 position.

25 May 2021

The message from PaNOSC WP4 saying they had stopped the development of the PaN portal to focus on ILL’s VISA was commented on. We will wait for Andy’s return for a proper statement on what it implies for ExPaNDS. Anton explained that, in any case, it does not help him in the definition of the next action steps at each facility and noted the timing was a bit unfortunate with our imminent review. Patrick thinks we will find a solution for the portal if our services are reachable and helpful so that is where we need to focus our efforts for now. Alun also noted the unavailability of the PaN portal was a foreseen risk at the proposal stage already.

11 May 2021

D4.3 - Testing and Validation framework (MAX IV) - 31 Aug 2021
The testing framework is progressing well for the Jupyter-based use cases. Following Zdenek’s presentation at the end of April, Andy mentioned it was being tested at ESRF. Anton reported the WP had to progress on the framework for remote desktop applications now.

On the topic of remote desktops, and as already mentioned by PaNOSC, the PaN portal will only support launching VMs in OpenStack. In ExPaNDS we don’t have the resources to deploy a production OpenStack service at the facilities that don’t have it already. Besides the possibility to interface the PaN portal to our local analysis services ourselves, e.g. to Kubernetes - which was estimated by ILL to take in the order of 55 weeks of work - ExPaNDS has the possibility to integrate local services directly into the EOSC (e.g. EOSC marketplace).

27 Apr 2021

Anton reported he went over the portable container deployment plan during the last meeting and now intends to use the next meeting this Friday (30/04/2021) to finalise the “check-list”.

For the testing framework, Zdenek organised a presentation to show a use case where Jupyter based applications are tested and discuss it with the other partners. The meeting is planned for this Thursday (29/04/2021).

13 Apr 2021

30 Mar 2021

The framework deliverable (D4.1) is now delivered to the EC and made public: 10.5281/zenodo.4569421. It is a month late but Anton appreciated the discussions it raised and thought it was worth the extra time to straighten it up.

The testing framework and how to link it to the portal are currently being discussed. Anton participated in last week’s PaNOSC meeting and is also meeting Fabio, the new PaNOSC WP4 leader this week.

16 Mar 2021

Anton is working to integrate the useful comments he received from the reviewers to his framework deliverable and will be delivering it asap.

Following the conversations with Daniel and Alun on the next steps for WP4, the tasks are being broken down into two teams: one for infrastructure and one for data analysis.

02 Mar 2021

The integration of the PaN portal in EOSC was discussed and it might be that each facility deployment would need to be onboarded and not only a central instance hosted by one of the facilities. Which means that our integration in EOSC has to also involve the e-infrastructures. Anton shared in that regard EOSC-hub’s ‘Integration handbook for service providers’.

Anton updated us on the status of WP4’s deliverables:

The upcoming WP4 meeting on Friday will be the occasion to discuss the roadmap for the technical implementations (see previous minutes) and to progress on the selection of a use case for the review meeting.

16 Feb 2021

02 Feb 2021

19 Jan 2021

15 Dec 2020

Anton reminded WP4 focused on the portal deployment at DESY, working very closely with PaNOSC/XFEL. Patrick drew Andy’s attention to the issue about ILL not developing the portal on Kubernetes but only on VMs. A solution will need to be found with PaNOSC, it is starting to be complicated to go further with the deployment.

Patrick also reported the 2 hands-on sessions on the portal deployment at DESY provided by Michael, which attracted over 40 participants from our different facilities. Anton confirmed it was interesting sessions that encouraged him to develop more CI/CD pipelines and that it will probably require a follow-up.

Patrick also mentioned the issue between EGI check-in (the AAI for our federated cloud) and EduTeams (umbrella ID, the PaN AAI) who are not agreeing. A situation we need to keep an eye on.

01 Dec 2020

Anton reported that last WP4 meeting focused on the upcoming deliverable 4.1 “Guidelines for implementing the national RI’s analysis services within the EOSC”, which was discussed and started.

In parallel, the test instances of the portal continue being prepared, in particular with DESY and XFEL working together on the subject. Patrick mentioned XFEL was considering having the PaN portal as part of the official XFEL infrastructure, thus greatly increasing the chances to have it sustained after the projects end.

Anton also noted that the integration of the data catalogues, i.e. the interface with WP3 needed to be addressed. Alun mentioned WP3 was participating in the PaNOSC WP4 meetings to keep an eye on it.

17 Nov 2020

27 Oct 2020

13 Oct 2020

Following the technical workshop and the last WP4 meeting, Patrick reported that it was decided to have a continuous integration / continuous deployment (CI/CD) testing platform for the portal hosted by DESY and MAX IV, in order to provide two different testing environments to all partners, avoiding local bias.
DESY is also working to make the ExPaNDS reference datasets available in a dCache instance.

Anton confirmed he would be present at the annual meeting now that it was virtual and will discuss with Andy and Patrick the preparation for the WP4² session.

29 Sep 2020

Patrick mentioned that Michael was preparing for the technical meeting, linking DESY’s Jupyter and Kubernetes to the portal’s back-end.

There are many questions on WP4 that hopefully the technical meeting will help answer.

15 Sep 2020

Majid reported last meeting discussions were about the current portal tests, and they introduced the topic of AAI for external users. EGI will try and come up with a proposal for next meeting.

The new recruit at MAX IV due to start in ~Nov will be responsible for the testing framework.

01 Sep 2020

Anton is working on having a server up and running at DESY to emulate the PaNOSC portal and link the datasets. The plan is to have “something” for the October technical workshop. This test instance would then be available for all partners to access and progress on the testing framework.

Daniel reminded the presentation for the technical workshop also aims at sharing what did not work, so DESY’s input will be valuable anyway. Regarding the workshop, he reported 47 people registered already which was really good. HZB, HZDR, ISIS and ESRF also submitted answers to the survey and SOLEIL, ALBA and EUXFEL are planning to do so in the additional two weeks that were granted.
Only the confirmed number of presentations is too low, with PSI, HZDR, ELI and ALBA presenting so far. Daniel emphasised that he hoped every facility would present, even if they didn’t yet progress much (or at all) on the ‘portal experience’. A reminder will be sent in that sense, giving the 11th of Sept as a deadline to be able to build the agenda of the workshop accordingly.

18 Aug 2020

During the last WP4 meeting, Uwe mentioned the survey sent around by Daniel and confirmed he had responded for HZDR. The others were not all aware of it and Uwe recommended to send a reminder. He also raised the topic of AAI and the fact that all facilities didn’t necessarily use UmbrellaID, which was perceived by some as ‘experimental’. AAI is an important part of PaNOSC WP6 and Andy reminded that GEANT had made UmbrellaID sustainable and EOSC compliant, with EduTeams. Patrick mentioned it was good that facilities did the necessary steps to integrate the new version of UmbrellaID (which will soon be operational) but that it was more a ‘facility task’ than an ‘ExPaNDS task’.

The next steps for the reference datasets and their publication are to be clarified.

04 Aug 2020

21 Jul 2020

During last WP4 meeting, Majid mentioned the discussions were focused on the testing platform. Daniel and Sophie reminded about the technical coordination workshop and the registration.

07 Jul 2020

23 Jun 2020

The selection of the reference data sets is finalised. Anton reported the focus was now going to the implementation of the test framework, lead by MAX IV.

09 Jun 2020

26 May 2020

The reference use cases are progressing with the shared document completed by each facility. Majid also reported that regular meeting are now in place, holding every two weeks.

28 Apr 2020

14 Apr 2020

Recruitment update: 0.5 FTE to be filled by current job advertisement from PSI 1 position to start very soon with Anton at DESY 0.2 FTE to be filled by upcoming job advertisement at DESY to work specifically on portal deployment Same as WP3 for Soleil Same as WP2 for ALBA 0.5 FTE to be fully available from July 2020 on for HZDR (Oliver)

Deliverable due M18: Anton and Patrick are adjusting on a common view of what the D4.1 deliverable should be.

31 Mar 2020

A meeting to define the test data sets of areas of interest will hold this Friday to follow-up on last week’s meeting.

WP4 is also working on integration of the PaNOSC portal.

17 Mar 2020

The current focus is still trying to identify the reference datasets and integrating the PaNOSC portal.

03 Mar 2020

18 Feb 2020

04 Feb 2020

Reminder of the WP4 F2F meeting in Hamburg.

21 Jan 2020

The telephone conference planned on the 22.01 will clarify the work for the coming weeks. A new recruit should also join WP4 soon for DESY.

07 Jan 2020

Use cases for data analysis services are being defined (~6 independent use cases) based on PaNOSC works with the objective of being presented or consolidated during upcoming PaN user meetings (e.g. DESY user meeting and MAX IV-I if it is not too short a notice). The reference data sets are also being defined and a workshop with WP3 is to be scheduled to that purpose. The collaboration within the project needs to be reinforced on this important task, especially with HZDR and PaNOSC. WP1 will help ensure the mailing lists are up-to-date ASAP.

03 Dec 2019

13 Nov 2019

Recruiting at DESY for WP4 successful; relevance of involving researchers has been stressed and the task to find appropriate data sets for testing; proposal to contact WP early to remind them on upcoming deliverables (6 months in advance).