Maritime Forum

4th EMODnet Technical Working Group Meeting

Published on: Wed, 15/05/2019 - 10:18
Table of Contents
    The EMODnet Technical Working Group consists of the data portal managers and developers involved in EMODnet projects. The group meets at least twice per year to discuss common challenges and solutions to provide better services to the EMODnet users.

    Date: Monday 01/10/2018 - 13h00 to Tuesday 02/10/2018 - 16h00
    Meeting Venue: DG MARE, rue Joseph II 99, Brussels, Belgium

     

    Attendees

    Name

    Organization

    Iain Shepherd

    DG-Mare

    Selene Alvarez

    EMODnet Secretariat

    Nathalie Tonné

    EMODnet Secretariat

    Jan-Bart Calewaert

    EMODnet Secretariat

    Pascal Derycke

    EMODnet Secretariat

    Antonio Novellino (Remote)

    ETT

    Marco Alba

    ETT

    Giuseppe Manzella (Remote)

    ETT

    Henrik Knudsen

    GEUS

    Ulla Alanen

    GTK

    Graeme Duncan

    JNCC

    Jordan Pinder

    JNCC

    Nick Earwaker

    Lovell Johns Limited

    Dick Schaap

    Maris

    Menashè Eliezer

    OGS

    Alessio Corongiu (Remote)

    TrustIT

    Alexander Barth

    ULiege

    Bart Vanhoorne

    VLIZ

    Filip Waumans

    VLIZ

    Paula Oset

    VLIZ

    Simon Claus

    VLIZ

     

     

    Summary of action points TWG-4

     

    Meeting

    #

    Action point

    Person(s)

    Status

    TWG4

    1

    AP. Pascal D. to upload the Web services documentation template in JIRA.

    https://webgate.ec.europa.eu/CITnet/jira/browse/EMODNET-2

    Pascal D.

     

    TWG4

    2

    AP. All lots to review and fill in the web services documentation template for their specific lot.

    All lots

     

    TWG4

    3

    AP: All lots to provide six entry points following schema of the EU Open Data Portal.

    All lots

     

    TWG4

    4

    AP: Pascal D. and Menashè E. to do a test to with Sextant / Geonetwork and check if a batch import can be done.

    https://webgate.ec.europa.eu/CITnet/jira/browse/EMODNET-3

    Pascal D., Menashè E.

     

    TWG4

    5

    AP: Pascal D. to share the European Atlas of the Seas Web services assessment checklist and all lots to review it.

    https://webgate.ec.europa.eu/CITnet/jira/browse/EMODNET-4

    Pascal D.

     

    TWG4

    6

    AP: All lots to include the INSPIRE metadata URL field in the Web services GetCapabilites.

    https://webgate.ec.europa.eu/CITnet/jira/browse/EMODNET-5

    All lots

     

    TWG4

    7

    AP: Pascal D. will explore use cases on supporting time dimension in geoserver, documenting them in JIRA, using Human Activities vessel density as test case.

    https://webgate.ec.europa.eu/CITnet/jira/browse/EMODNET-6

    Pascal D.

     

    TWG4

    8

    AP: Seabed habitats (GD, JP) and Bathymetry (DS) to share their web services monitoring solutions in JIRA for the rest of the lots to test.

    https://webgate.ec.europa.eu/CITnet/jira/browse/EMODNET-7

    Seabed habitats (GD, JP), Bathymetry (DS)

     

    TWG4

    9

    AP: Secretariat to consider inviting Copernicus to the next TWG meeting.

    Secretariat

     

    TWG4

    10

    AP: Nathalie T. to distribute proposal for menu standardization.

    https://webgate.ec.europa.eu/CITnet/jira/browse/EMODNET-1

    Nathalie T.

     

    TWG4

    11

    AP: All lots to give feedback on the menu standardization proposal (specifically on separating or not data vs. data product access).

    https://webgate.ec.europa.eu/CITnet/jira/browse/EMODNET-1

    All lots

     

    TWG4

    12

    AP: All lots to re-organize their menus according to the agreed items.

    https://webgate.ec.europa.eu/CITnet/jira/browse/EMODNET-1

    All lots

     

    TWG4

    13

    AP: All lots to update the privacy policy, providing separate link in the footer. Example of reference document - privacy statement given by TrustIT (http://www.emodnet.eu/privacy-statement-protection-personal-data). Note: Chemistry has adapted it to mention that they share their information with Central Portal (third party).

    All lots

     

    TWG4

    14

    AP: Chemistry to look into Bart V. suggestion.

    Menashè E.

     

    TWG4

    15

    AP: Secretariat/TrustIT to include a disclaimer/caveat in the reports to indicate that numbers have to be taken with caution and to look at the relative trends over time rather than the absolute numbers.

    Secretariat, TrustIT

     

    TWG4

    16

    AP: all lots to create one page per use case to monitor individual use cases. (TrustIT request)

    All lots

     

    TWG4

    17

    AP: to elaborate official response from all lots, (with special contribution from Simon C. and Dick S.) and convey the response via DG-Mare. Elaborate response by mid-October and send response by the end of the month.

    All lots, Simon C., Dick S., Secretariat-DG-Mare

     

    TWG4

    18

    AP: Nathalie T. to re-distribute the google document with the categorization of data and data products. Establish deadline.

    Nathalie T.

     

    TWG4

    19

    AP: All lots to review, make corrections and suggestions to the categories document, adding and re-locating their data and products accordingly.

    All lots

     

    TWG4

    20

    AP: Nathalie T. and Paula O. to update the portfolio with the final categories.

    Nathalie T., Paula O.

     

    TWG4

    21

    AP: Once these categories are agreed, all lots to add the categories to the metadata (lineage).

    All lots

     

    TWG4

    22

    AP: VLIZ to include also the OSPAR and Habitat areas layers in the Query tool.

    VLIZ

     

    TWG4

    23

    AP: VLIZ to elaborate API documentation of the Query tool processing service.

    https://webgate.ec.europa.eu/CITnet/jira/browse/EMODNET-8

    VLIZ

     

    TWG4

    24

    AP: VLIZ to launch first versions by the end of the year.

    VLIZ

     

    TWG4

    25

    AP: VLIZ to set up work team for QT and define milestones (JIRA).

    VLIZ

     

    TWG4

    26

    AP: VLIZ to publish the EMODnet data in R tutorial. This to be shared with lots, so they can contribute with similar examples for their services. (To publish web services documentation too once is completed by lots).

    https://webgate.ec.europa.eu/CITnet/jira/browse/EMODNET-9

    VLIZ

     

    TWG4

    27

    Pascal D. asked the query tool to be conceived and designed as a portable application.

    VLIZ

     

    TWG4

    28

    AP: VLIZ to report on experience with NetCDF in geoserver (extension not stable?). If output of DIVA model needs adaptations, Alexander B. confirms this could be looked at.

    VLIZ

     

    TWG4

    29

    AP: VLIZ to integrate Chemistry parameter aggregated layers in the Central Portal geoviewer.

    VLIZ

     

    TWG4

    30

    AP: VLIZ to add beach litter sampling effort layer to CP.

    VLIZ

     

    TWG4

    31

    AP: Nathalie T. and Paula O. to add Chemistry new layers to portfolio.

    Nathalie T., Paula O.

     

    TWG4

    32

    AP: All lots to communicate to Secretariat and VLIZ when a new layer/service will be published.

    All lots

     

    TWG4

    33

    Pascal D. to distribute JIRA credentials with TWG members.

    Pascal D.

     

    TWG4

    34

    Pascal D. to generate the JIRA issues related to the implementation of the MarineID. Dick S. to attach the documentation to JIRA. Seabed Habitats will be the test case with assistance from Pascal D., Menashè E. and Dick S.

    https://webgate.ec.europa.eu/CITnet/jira/browse/EMODNET-10

    Pascal D., Dick S., Menashè E.

     

    TWG4

    35

    Dick S. to consider the development and sharing of a base layer for all the lots.

    Dick S.

     

    TWG4

    36

    Participation of EMODnet to the next EU Open Data portal hackathon

    Secretariat

     

    TWG4

    37

    Implementation of ssl protocole for all lots and CP Matomo and harmonization of the web services URL

    All lots

     

    TWG3

    1

    VLIZ to follow up and integrate Physics climatologies in the CP geoviewer. Definition of legend in URL needed? Fix issue with parameter in URL and not in the get capabilities.

    VLIZ

    Ongoing

    TWG3

    2

    Antonio Novellino to provide aggregated (combined) climatology products for European scale (not per sea basin, similar to Chemistry).

    Antonio Novellino

    Ongoing

    TWG3

    3

    VLIZ to integrate Physics Sea Level product in the CP geoviewer.

    VLIZ

    Ongoing

    TWG3

    4

    Paula Oset and Pascal Derycke to review web services documentation page to remove direct links to data (custom package datasets to use during OSL) and other not relevants links.

    Paula Oset, Pascal Derycke

    Ongoing

    TWG2

    1

    Add Access to web coverage service in metadata

    Alessandro Pititto

    Ongoing

    TWG2

    2

    GeoNetwork. Seabed habitats: move towards one product

    Graeme Duncan

    Ongoing

    TWG2

    3

    Move Human Activities from Mapserver

    Alessandro Pititto

    Unknown (Alessandro Pititto not present)

    TWG2

    4

    (Getlegendgraphic: Alex & Bart check how to implement)

    Pascal Derycke request that the three options (html, xml, json) are activated for all portals.

    All

    Ongoing

    TWG2

    5

    Physics: HF Radar, MEOP data: THREDD server: using OGC standard. Bart Vanhoorne to send email and follow up.

    Bart Vanhoorne, Antonio Novellino

    Ongoing

    TWG2

    6

    Data polices: Map existing policies that exist and propose overall data policy EMODnet by next Steering Committee meeting

    Jan-Bart Calewaert

    Ongoing

     

    Meeting Minutes

    Expected outcomes of this meeting (Pascal Derycke)

    Two main goals:

    • To improve coordination and make collaborative work easier, DG-Mare has set up a JIRA (bug tracking, issue tracking, and project management service) for EMODnet. We all share similar systems and face similar challenges and bottlenecks. With JIRA everybody can view and contribute to the different issues & tasks even if it is not directly related with their lot; everybody can be up to date with developments.
    • To work towards harmonization of the portals and improve m2m communication, keeping the OSLII deadline in mind with the objective to have a hackathon package ready.

     

    Status EMODnet web services (Pascal Derycke)

    Web services documentation

    Full documentation needs to be in place for the OSLII (September 2019).

    Pascal D. has prepared a template document with the core information about Web services that each lot should adopt and customize according its own services (server URL, requests and examples, etc.). The document will be distributed and it should be amended and restated by each of the thematic lots before to be published.

    1. AP. Pascal D. to upload the Web services documentation template in JIRA.
    2. AP. All lots to review, amend and restate the Web services documentation template according to their specific lot.

    Where the documentation will be published, is not decided. Pascal D. thinks that, from a user-point of view, the information should be included on every portal since not all users realise that there are several EMODnet portals. Pascal D. thinks each thematic portal should publish this information – by keeping the same structure, but adapting the content. It is a mean to link the portals between them (Web ring approach).

     

    Web services harmonization

    All the data portals and products are different but the way to disseminate has to be standardized to a certain degree, taking into account the particularities of each lot and the technical implementations (e.g. geoserver vs. oceanbrowser). To work towards harmonization on different levels:

    • Server URL naming & SSL protocol
      • Naming: good practice to include the name of emodnet + lot in the URL fo the Web services (e.g. https://ows.emodnet-bathymetry.eu/ows or https://ows.emodnet-seabedhabitats.eu/ows ).
      • Implementation of the SSL protocol: move towards https for all lots not having implemented it yet. It is also required for the central Matomo instance.
      • Seabed habitats has recently changed the naming of the URL and adopted https. It can help other portals to do so.
         
    • Metadata harmonization:
      • Pascal D. presents the experience with the EU Open Data Portal. Some Data products from the EMODnet lots were manually ingested so EMODnet could be visible for the hackathon organized by the EU Open Data Portal. All the data products should be ingested to increase the visibility of EMODnet work. This official portal maintained by the Publications Office promotes the EU open data Policy (The European Union Open Data Portal (EU ODP) gives you access to open data published by EU institutions and bodies. All the data you can find via this catalogue are free to use and reuse for commercial or non-commercial purposes.)
      • EU Open Data Portal enables m2m communication, documenting data with the following 3x2 structure:
        • Documentation: metadata record + website URL of the data provider
        • Visualization: WMS service + map viewer URL
        • Resources: WFS service + download URL
      • Manual ingestion not desired, so we will explore the possibility to do a batch import from central portal Geonetwork. To do so, all the lots have first to provide the 3x2 accesses. Menashè E. and Pascal D. to do a test to with Sextant / Geonetwork and check if a batch import can be successfully performed.
      • Besides the 3x2 entries, Pascal D. to share the Web services assessment checklist of the European Atlas of the Seas in JIRA and all lots to review it and give feedback. Once the checklist is met we can say that m2m is fully enabled. Pascal D. says that an assessment tool based on the checklist needs to be developed in order to assess if a Web service meets the requirements. The tool is expected to produce an assessment report.
        • The already available records in the EU Open Data Portal are combined products, satellite imagery. Some lots (Dick S., Alexander B.) think that the ingestion exercise should be limited for the moment to the data products.

     

    Addendum: end of October, the Director of DG MARE asked EMODnet to participate to the next hackathon organised by the EU OPEN DATA portal.
     

    • GetCapabilities:
      • For INSPIRE compliancy, GetCapabilites of all the web services should contain the INSPIRE metadata URL field (ref: TWG3).
         
    • Supporting time and elevation dimensions.
      • Pascal D. puts emphasis on the importance to distribute historical data and thus to have the Web services supporting the time dimension. This should be done following ISO standards.

     

    1. AP: All lots to provide six entry points following schema of the EU Open Data Portal.
    2. AP: Menashè E. and Pascal D. to do a test to with Sextant / Geonetwork and check if a batch import can be performed.
    3. AP: Pascal D. to share the European Atlas of the Seas Web services assessment checklist and all lots to review it through JIRA.
    4. AP: All lots to include the INSPIRE metadata URL field in the services GetCapabilites.
    5. AP: Pascal D. will explore use cases on supporting time dimension in Geoserver, documenting them in JIRA, using Human Activities vessel density as test case.

         36. AP: Participation of EMODnet to the next EU Open Data portal hackathon

    37. AP: Implementation of ssl protocole for all lots and CP Matomo and harmonization of the web services URL
     

    Monitoring Web services (Update Spatineo trial)

    Pascal D. ran a Spatineo trial to check its suitability to monitor EMODnet web services. Spatineo checks the status and compliance of the server, and gives metrics on WMS and WFS services. However, the results of the test are not considered positive, main reasons being: a) monitoring of WCS services is not implemented and, b) server logs need to be uploaded in the Spatineo server to monitor WFS services.

    Checking the performance and INSPIRE compliance of the service can be done in-house (e.g. Pascal D. knowcean tool), but a solution to monitor WMS and specifically WFS and WCS is needed. One possible solution is to test server log analytics in Matomo. Bart V. states that the solutions should distinguish between WMS, WFS and layer names.

    Seabed Habitats and Bathymetry have in-house solutions. These existing solutions can be shared and documented in JIRA, to test whether they can be applicable to all lots.
    Pascal D. stressed that ideally a common solution which would allow to report on metrics the same way for all lots is sought.

    1. AP: Seabed habitats (GD, JP) and Bathymetry (DS) to share their Web services monitoring solutions in JIRA for the rest of the lots to test.

     

     

    EMODnet Open Sea Lab II Hackathon (Pascal Derycke)

    The OSLII will take place in September 2019, together with Copernicus and ICES. This will help to widen the profile of the participants. We expect to target more students by running the hackathon in the month of September.

    1. AP: Secretariat to consider inviting Copernicus to the next TWG meeting.

     

    Packages, documentation, toolbox

    A hackathon package (user documentation, portfolio, toolbox, …) needs to be finalised sometime before the OSL II. This will allow to engage with some potential participants that will be able to evaluate in advance what can be done during the hackathon. The typical hackathon participant is a data scientist, but we expect (marine) scientists too. The hackathon gives us a deadline. We need to elaborate a set of documents that are valuable for other types of users beyond hackathon participants. The documentary background has to be user-oriented: different types of documentation for users of different areas and levels of expertise. All documentation should be managed via JIRA and then published in the Central portal.

     

    The hackathon package should be a combination of libraries, APIs, documentation for different users. This will make us to be prepared for the hackathon but also to be stronger within the community of marine scientists.

    • Portfolio to discover what is available and video tutorials (TrustIT will help with the elaboration) on where to find data access.
    • QGIS plugin and/or tutorial to use EMODnet web services in QGIS for basic GIS users.
    • OGC web services documentation based on Pascal D. template, for GIS experts and developers.
    • APIs & documentation (data scientists, developers).
    • Libraries / tutorials on how to use EMODnet data and services in different environments (R, Python,…), for scientists and data scientists.
    • Libraries to combine datasets, produce indicators, etc.

     

    Goal: to test this documentation and have it ready by next summer (before the OSL II).

     

    Standardization menu items

    Feedback from users collected by Secretariat (Nathalie T.) shows that users have difficulties finding what they are looking for in the portals, and that in fact the same information is structured differently across the lots.

    The menus of all the thematic portals should be standardized so pages with a similar content (map viewer, services documentation) can be accessed in a similar way (UX improvement in the context of a Web ring). Nathalie T. presents new proposal to organize content:

    About

    Data & data product access

    News & events

    Data policy

    Get involved!

    Helpdesk

    Project/About

    (-> background

    info, WPs, timeframe, ..)

    Map viewer

    News

    Privacy statement

    How to contribute data to EMODnet

    Contact

    Project documents

    (-> ppts, public documents, progress reports, publications, photos)

    Catalogue

    (-> Search, Download, …)

    Events

    How to acknowledge

    Use cases

    Help

    (-> Tutorials, help, demos, glossary)

    Partners

    Web services

    (Blog)

    (Disclaimer; can be shown upon downloading for example)

    Associated Partners

    (-> explanation how to become an AP; link to the Partners-page in About)

     

    (Sources)

    Data definition

    (-> QA/QC protocols, production details, …)

    (Surveys)

         

     

    https://webgate.ec.europa.eu/CITnet/jira/browse/EMODNET-1

     

    The lots discussed the proposed structure and made few comments:

    * Some lots would prefer to have separate menu for Data access and another for data products. This are different things and often offered through different platforms. It helps the users to recognise immediately that both types of data are offered (i.e. there is a (work) flow from data to data products). Another possibility is to keep it under the same menu with a different title (e.g. data and data products, data services).

    ** To be compliant with GDPR, a separate link needs to be provided for the privacy policy. This link will be given at the footer (separate – as is already done in most Portals), but can at the same time be given under the data policy menu.

     

    1. AP: Nathalie T. to distribute proposal for menu standardization, and to already include the agreed changes (e.g. inclusion of “Associated Partners” under the Get involved!-menu).
    2. AP: All lots to give feedback on the menu standardization proposal (specifically on separating or not data vs. data product access).
    3. AP: All lots to re-organize their menus according to the agreed items.
    4. AP: All lots to update the privacy policy, providing separate link in the footer. Example of reference document - privacy statement given by TrustIT and updated by Nathalie T. according to privacy statement VLIZ in the meantime (http://www.emodnet.eu/privacy-statement-protection-personal-data). Note: Chemistry has adapted it to mention that they share their information with Central Portal (third party).

     

     

    Status EMODnet

    EU Open Data portal / European Atlas of the Seas (Pascal D.)

    Eu Open Data portal: see Web services harmonisation / m2m communication.

    The European Atlas of the Seas works as an aggregator of Web services with a system in place (Content management system) which allows to operate a catalogue of more than 200 Web services supplied by various data providers as EMODnet portals. A tool is daily monitoring the status of all services.

    In order to improve the consistency and performance of the Atlas, an assessment protocol of the Web services was written (see Web services harmonisation).

     

    EMODnet progress indicators

    Chemistry (Menashè E.) has made a comparison study for two months between their metrics system and Matomo. They get very different results (related to Indicator 6 in the quarterly progress reports: one table on web pages and one on web sections). Bart V. suggests the differing results could be an issue produced by their server (Post-meeting update: web-server is acting as bot, generating ~500 extra visits daily). However, many of the lots have observed a decrease in the numbers compared to previous systems. This should be highlighted in the progress reports.

    Pascal D. stated that Matomo is one of the most community accepted metrics system (after Google Analytics) and should be considered a fully reliable analytic tool. He lighted that Matomo collects visit hits when AWSTAT analyses server log files which are two different conceptual approaches to deliver metrics.

    1. AP: Chemistry to look into Bart V. suggestion.
    2. AP: Secretariat/TrustIT to include a disclaimer/caveat in the reports to indicate that numbers have to be taken with caution and to look at the relative trends over time rather than the absolute numbers. Also think about the reports that have been published already on every Portal (upon request from DG MARE).
    3. AP: TrustIT to create one page per use case to monitor individual use cases.

     

    Menashè E. thinks that the web pages-indicator table can be removed or moved to a less prominent part of the report, because the pages and sections are the same in case of Chemistry (prove the same metrics). This is not the case for the other portals, so Alessio C (Trust-IT) suggests to keep both tables because they do provide different information, and to identify new web pages for Chemistry worth monitoring.

    Alessio C. mentions that after this reporting period there will be for the first time the opportunity to do some analysis and present trends of the indicators over time (3 full reporting periods considered).

     

    INSPIRE updates

    Dick S. gives feedback on the Chemistry / SDC presentation during the INSPIRE 2018 conference, a use case of implementing INSPIRE data models to map EMODnet nutrients data. By testing the implementation rules for EMODnet data, shortcomings were found and change requests were formulated. New scheme is more rigid and INSPIRE validator has errors; Chemistry metadata is compliant but some data is not. In general, requesting changes is a slow, arduous process. Simon C., in collaboration with Seabed Habitats, has requested INSPIRE to adopt the DarwinCore (DwC) schema. Initial feedback was that they were open to it but there was no follow-up on the INSPIRE side. There is frustration because there is a lot of effort done in making EMODnet INSPIRE compliant but the feedback is limited. DG-Env has to support the member states but no obligation to support EMODnet (which is thus not a priority). It is suggested that it could be a project on its own, a pilot to implement INSPIRE in the marine data community. It is agreed that there should be a response to move things forward at the level of DG-Mare. There was a meeting on how to link EMODnet and INSPIRE. It was said that input can be provided to INSPIRE (new phase), by means of a form (e.g. Biology and Chemistry did). The INSPIRE JRC team gave follow-up feedback to the EMODnet comments:

    https://webgate.ec.europa.eu/fpfis/wikis/display/MarINSP/feedback+and+solutions

    1. AP: to elaborate official response from all lots, (with special contribution from Simon C. and Dick S.) and convey the response via DG-Mare. Elaborate response by mid-October and send response by the end of the month to Iain Shepherd.

     

    Data products categorization

    Giuseppe M. presents the physics approach to categorize data & data products (“downstream products”). To elaborate this proposal, they looked at existing best practice initiatives (SCOR, QUASIMEME, QARTOD temperature-salinity). Nathalie T. and Paula O. present the current categories suggested for EMODnet data and data products. It differs from the physics approach mainly by the addition of two extra levels (Level 4 data and Level 6 data product). See: https://docs.google.com/document/d/1MERpY1bui67tTTM6vMkZdCoBnrsHWZos7icBy2zh-2E/edit?usp=sharing


    Level 4 is in general accepted by the lots (many portals have data that flow from level 3 to 4 (individual dataset to harmonized collections), but Skype connection with Giuseppe M. is lost.

    The final goal is to publish these categories together with the portfolio, assigning the showcased data and products with their correspondent categories. Paula and Nathalie indicated the definitions should be ready before 20 October. A final version of the portfolio has to be ready to be printed for the EOOS Conference (21-23 November 2018).

    1. AP: Nathalie T. to re-distribute the google document with the categorization of data and data products. Establish deadline within email.
    2. AP: All lots to review, make corrections and suggestions to the categories document, adding and re-locating their data and products accordingly.
    3. AP: Nathalie T. and Paula O. to update the portfolio with the final categories.
    4. AP: Once these categories are agreed, all lots to add the categories to the metadata (lineage).

     

    Central Portal data services update

    • CP data services are organized with a user-targeted approach (different services for different users).
    • Updates related to Geoviewer and Geonetwork catalogue are presented.
    • The concept of the query tool is presented.
      • The tool is though for a preliminary assessment and early phase planning of a particular activity, with the practitioner or marine planner user in mind. It retrieves information from multiple thematic data products via one single interface, and produces a summary report of the area selected for the layers selected.
      • The technical set up is also presented. APIs documentation will be provided.
      • Timeline: to implement two more layer types (WCS, WFS point), with layers from different thematic portals by the end of the year 2018.
    • VLIZ has also elaborated a tutorial (html page) for using EMODnet data and services in R. Examples of layers from Bathymetry, Seabed Habitats plus Biology API. The tutorial covers:
      • How to download and use data in R.
      • How to use EMODnet products in R using WFS service (SH) and WCS (Bathymetry).
      • How to visualize WMS services in R.

     

    1. AP: VLIZ to include also the OSPAR and Habitat areas layers in the Query tool.
    2. AP: VLIZ to elaborate API documentation of the Query tool processing service.
    3. AP: VLIZ to launch first version by the end of the year.
    4. AP: VLIZ to set up work team for QT and define milestones (JIRA).
    5. AP: VLIZ to publish the EMODnet data in R tutorial. This to be shared with lots, so they can contribute with similar examples for their services (To publish Web services documentation too once is completed by lots).
    6. Pascal D. asked the query tool to be conceived and designed as a portable application.

    Pascal D. asked information about the technology supporting the implementation of the query tool. Filip W. said that the query tool is being developed in JAVA. Pascal D. asked the reasons for choosing JAVA.
     

    Overview of new data products & developments by thematic portals

    EMODnet Ingestion (Dick S.)

    • Over 240 submissions, mostly in the first phase (ingestion), with about 25 submissions in the second phase (digestion, integration in EMODnet portals).
    • There are wide differences in the number of submissions per country. It could be to various reasons, including the degree of existing organization that is already in place.
    • In some cases, they are also working into m2m communication (e.g. BODC), establishing specific data exchange procedures.
    • Main challenge remains involving industry.
    • SH is not formally represented at Ingestion but the communication is fluent.

     

    Bathymetry (Dick S.)

    • New DTM with 1/16 * 1/16 arc (approx. 115m) and greater spatial coverage. Derived data products updated (e.g. source reference layer).
    • 3D visualization tool -> documented and will be shared in GitHub.
    • Download registration form.

     

    Geology (Henrik K.)

    • GeoNetwork CSW for OGC access.
    • Data entity indexes are ready.
    • Restructured portal content.
    • New data products and updates presented.
    • Upcoming products: multiscale substrate map, seafloor quaternary geology, seafloor geomorphology, etc…
    • Working on stability of servers and webGIS download (possibility to draw polygon).

     

    Biology (Simon C.)

    • Second week of October there will be a data product workshop.
    • Deliverables will be compiled in the Atlas of Marine Life which will contain different sections: a combination of tools, models, Geoviewer with mapped products. The Web services and workflows will be documented (R-markdown).
    • Data products on the Geoviewer of the Atlas of Marine Life will be basic data maps (e.g. gridded abundances), indicators (e.g. non-indigenous species) and story maps, distribution and trait-based models (e.g. low resilience benthic communities). Products will be structured around EOV (some of the EOV categories are covered by Seabed Habitats).
    • It is desired to work only with publically available data so that the user can replicate the analysis by using documented workflows.
    • Main technical challenge is to upload DIVA output – NetCDF into Geoserver.
    1. AP: VLIZ to report on experience with NetCDF in Geoserver (extension not stable?). If output of DIVA model needs adaptations, Alexander B. confirms this could be looked at.

     

    Human Activities (Nick E.)

    • Visual guidelines implemented, common terminology adopted, responsive design, blog updates.
    • Live chat implemented, managed by Cogea (9-17h), plus contact form.
    • User needs to fill in form to access to Web service URL. This should be changed.
    • Updated datasets: protected areas, bathing waters, fish catches, ports, cables, waste disposal. New: waste at ports.
    • Coming soon: vessel density maps. Output unit: hours/sqkm per month. Currently improving processing speed. Initial launch will be one-year data -> update map viewer to allow time sliding. Challenge: to implement WCS for vessel density (no technical experience with WCS).

     

    Seabed habitats (Graeme D. and Jordan P.)

    • WP4 survey point data update (ground-truthing data).
      • Current status: beta testing. UK data transformed and uploaded (already in Geoserver but testing mode).
      • More European data to do further ingestion and testing.
      • Format DwC OBIS-ENV based. It allows to store different classification systems (EUNIS, Habitats Directive) for a single point, together with measurement device (BODC vocab), metatada and dataset identifications and different contact points.
      • The table accessible via Geoserver is a flattened version with only public data.
      • Bboxes will be displayed for areas where there is restricted data to let user know the data exists.
      • Working on developing BODC/NERC vocabulary for classification systems.
      • Next steps: finish testing with wider datasets and publish, and work with VLIZ towards OBIS integration.
    • Other updates:
      • 180 new maps uploaded, and more third party will come in the next year.
      • EUNIS MSFD for the Black Sea and recent data from OSPAR uploaded.
      • Looking into revamp of the map viewer, to move forward to Geoserver based implementation.
      • Working towards automatization of updates and INSPIRE compliancy.

    Chemistry (Dick S.)

    • Harvested nutrient data and now working on the product (DIVA), which will replace the previous products. We learn from previous runs.
    • Harvesting contaminants (data from new areas like Norway) à data sent over to research coordinators for harmonization, to be published the in future.
    • Further work on marine litter and micro-litter, trying to access data from more areas (Mediterranean is difficult, but in contact with the area). We are now enriching the maps. Both datasets progressively being populated.
    • DG-Env has accepted EMODnet Chemistry as a source for the MSFD implementation (as an official reporter).
    • Plots: P36 layers per aggregated parameters have been created (action point from TWG2). For each parameter group you get position of CDIs, with a link to the CDI service (you get a list of all relevant datasets for a specific region).

     

    1. AP: VLIZ to integrate Chemistry parameter aggregated layers in the Central Portal Geoviewer.
    2. AP: VLIZ to add beach litter sampling effort layer to CP.
    3. AP: Nathalie T. and Paula O. to add Chemistry new layers to portfolio.
    4. AP: All lots to communicate to Secretariat and VLIZ when a new layer/service is about to be published.

     

    Physics (Marco A.)

    • New products on underwater noise. Two products:
      • Pulse base, starting from data from ICES and expanded to Mediterranean Sea with other sources. Already available.
      • Noise map, currently being developed/calculations ongoing. Time consuming process. First test area around Barcelona. For other areas is difficult because lack of accessible data and 3-4 weeks processing to make a one-month layer.
      • Spatial coverage of the product: Europe except Black Sea, Biscay Bay. It’s point and polygon data.
      • HF radar services: published on request of DG MARE, and now nearest time in EAS (European Atlas of the Seas).

     

    Technical support & other updates

    JIRA

    JIRA and Confluence will be used to improve coordination and make collaborative work easier. Pascal D. will email with credentials and skype calls can be made with the different lots to explain how it works.

    1. Pascal D. to distribute JIRA credentials with TWG members.

     

    MarineID

    Technical details and advantages of implementing MarineID where discussed during TWG3 and SC in Mallorca. It will be implemented in all portals (part of new contract), step-wise approach. Information on usage of each portal will be stored by the specific lot (separate from the MarineID user information at Ifremer).

    MarineID grants access (is a gateway), and it will be possible for the user to switch from one to another portal within one session. It does not register when user downloads; this is one through a download application which makes a track record.

    1. Pascal D. to generate the JIRA issues related to the implementation of the MarineID. Dick S. to attach the documentation to JIRA. Seabed Habitats will be the test case with assistance from Pascal D., Menashè E. and Dick S.

     

    Background layer

    All the lots agree in the convenience to have a common background layer for all the viewers. Background layer should have global coverage, be empty on the marine side and support other projections (e.g. polar for Physics). A proposal by Dick S. was presented at the 3TWG. The initiative was well received but there is no budget at the moment.

    1. Dick S. to consider the development and sharing of a base layer for all the lots.

    The lots agree on the new structure with the exception of the distinction between data and data product access, to be further discussed.

     

     

    Revision of action points from previous meetings

     

    Action point

    Person(s)

    Status

    Meeting

    Paula Oset to circulate email list of both TWG and OSL coaches to be updated.

    Paula Oset

    Completed

    TWG3

    Each thematic data portal to develop an action plan to resolve identified bottlenecks/issues with regards to web services, data accessibility, harmonization of documentation.

    Thematic coordinators

    Completed (TWG4 agenda)

    TWG3

    Pascal Derycke to report the list of issues and actions needed from the different partners to harmonize web services and facilitate machine to machine communication (i.e. metadataURL)

    Pascal Derycke

    Completed (TWG4 agenda)

    TWG3

    Write a comprehensive general documentation on “accessing EMODnet data” (Web services, APIs, Case studies,…)

    Pascal Derycke, Paula Oset

    Completed (TWG4 agenda)

    TWG3

    Secretariat to communicate again and remind login details for the doc manager to partners.

    Secretariat

    Completed

    TWG3

    Secretariat to define who has to be contacted in the future whenever partners want to add new use cases.

    Secretariat

    Completed

    TWG3

    VLIZ to set up Search page as default and possibly hide the INSPIRE page.

    VLIZ

    Completed

    TWG3

    VLIZ to check with Ifremer responsible for Sextant, what are the issues with Chemistry harvests. Keep Pascal Derycke in the loop.

    VLIZ

    Completed

    TWG3

    VLIZ to provide deep link from metadata record to geoviewer display of the layer.

    VLIZ

    Unclear

    TWG3

    VLIZ and Pascal Derycke to work towards the harmonization of metadata.

    VLIZ, Pascal Derycke

    Completed (TWG4 agenda)

    TWG3

    Paula Oset to filter out Chemistry combined layers before 2000.

    Paula Oset

    Completed

    TWG3

    VLIZ to follow up and integrate Physics climatologies in the CP geoviewer. Definition of legend in URL needed? Fix issue with parameter in URL and not in the get capabilities.

    VLIZ

    Ongoing

    TWG3

    Antonio Novellino to provide aggregated (combined) climatology products for European scale (not per sea basin, similar to Chemistry).

    Antonio Novellino

    Ongoing

    TWG3

    VLIZ to integrate Physics Sea Level product in the CP geoviewer.

    VLIZ

    Ongoing

    TWG3

    VLIZ, Gerrit Hendriksen, Dick Schaap (ongoing from TWG-2), GetFeatureInfo to actually retrieve bathymetry values (not RGB).

    VLIZ, Gerrit Hendriksen, Dick Schaap

    Completed

    TWG3

    Paula Oset and Pascal Derycke to review web services documentation page to remove direct links to data (custom package datasets to use during OSL) and other not relevants links.

    Paula Oset, Pascal Derycke

    Ongoing

    TWG3

    VLIZ to develop a proposal of the Query tool 2.0 main functionalities with input from Secretariat; to be presented at Marine Knowledge Expert Group meeting.

    VLIZ

    Completed

    TWG3

    Develop work plan/action plan for the development of the query tool and other central portal data services

    VLIZ

    Completed (TWG4 agenda)

    TWG3

    Set up a user group + task force group (selected members of TWG and OSL Coaches/participants). The users group should be a small group (~5) to provide feedback during development

    VLIZ

    Completed (TWG4 agenda)

    TWG3

    Simon Claus to share presentation of overview of EMODnet data services in intranet.

    Simon Claus

    Completed

    TWG3

    Simon Claus (and all lots) to channel material that can demonstrate the value generated by EMODnet in relation to MSFD.

    Simon Claus, all lots

    Completed

    TWG3

    Secretariat to check with individual portals what is their opinion about the proposal for a new common base layer.

    Secretariat

    Completed

    TWG3

    TrustIT to send email with clear guidance on the new list of indicators.

    TrustIT

    Completed

    TWG3

    All thematic lots to report using new indicators from April onwards. If possible, overlap old and new indicators for a period. When not possible to report on a particular indicator (e.g. WMS), give an indication about the progress in setting up procedure to report on that indicator.

    All lots

    Completed

    TWG3

    Bart Vanhoorne to share the solution to track user interaction in map viewer with individual portals.

    Bart Vanhoorne

    Completed

    TWG3

    Dick Schaap to report and share with the rest of the partners about the solution implemented to track services.

    Dick Schaap

    Completed (TWG4 agenda)

    TWG3

    VLIZ to lead on implementing a common solution to monitor web services.

    VLIZ

    Completed (TWG4 agenda)

    TWG3

    Secretariat to circulate doodle with dates for TWG4 (end September – beginning of October).

    Secretariat

    Completed

    TWG3

    Add Access to web coverage service in metadata

    Alessandro Pititto

    Ongoing

    TWG2

    GeoNetwork. Geology: currently repeat metadata+> static xml that we harvest for metadata

     

    Completed

    TWG2

    GeoNetwork. Geology: currently repeat metadata+> static xml that we harvest for metadata

    Bjarni Pjetursson

    Unknown (follow via email)

    TWG2

    GeoNetwork. Seabed habitats: move towards one product

    Graeme Duncan

    Ongoing

    TWG2

    Move Human Activities from Mapserver

    Alessandro Pititto

    Unknown (Alessandro Pititto not present)

    TWG2

    (Getlegendgraphic: Alex & Bart check how to implement)

    Pascal Derycke request that that the three options (html, xml, json) are activated for all portals.

    All

     

    TWG2

    Map chemistry plots: deep links?

    Discussion over the technical complexity. Chemistry to provide a map with overview of observations per P36 (general) parameters, from which to point to the portal. Follow up via email.

    Alexander Barth, Paula Oset, Gerrit Hendriksen, Pascal Derycke, Alessandra Giorgetti

    Completed

    TWG2

    Physics: HF Radar, MEOP data: THREDD server: using OGC standard. Bart Vanhoorne to send email and follow up.

    Bart Vanhoorne, Antonio Novellino

    Ongoing

    TWG2

    Seabed Habitats. Move from mapserver to Geoserver/PostgreSQL

    Graeme Duncan

    Completed

    TWG2

    Bathymetry. Mean depth full coverage as background layer CP

    Dick Schaap

    Completed (TWG4 agenda)

    TWG2

    Bathymetry. Getfeature info: colour

    VLIZ, Gerrit Hendriksen

    Completed

    TWG2

    Create page describing WMS/WFS/WCS

    All lots

    Completed (TWG4 agenda)

     

    Single sign-on: List & compare attributes between user databases

    TL, Bart Vanhoorne

    Completed (TWG4 agenda)

     

    Data polices: Map existing policies that exist and propose overall data policy EMODnet by next Steering Committee meeting

    Jan-Bart Calewaert

    Ongoing

     

     

    Annex 1 - Agenda

    Monday 1 October 2018

    13:00-13:15 - Welcome & introduction (Jan-Bart Calewaert)

    • Welcome to participants 5’
    • Practical arrangements 5’
    • Tour de Table 5’

    13:15-13:45 - Progress & objectives (Jan-Bart Calewaert)

    • Updates since last meetings (revision of action points from previous meetings) (Paula Oset) 15’
    • Expected outcomes of this meeting (Pascal Derycke) 15’

    13:45-14:45 – Status EMODnet web services

    • Web services documentation 20’
    • Web services harmonization 20’
    • Monitoring Web services (Update Spatineo trial) 20’

    14:45-15:00 – Coffee Break

    15:00-16:00 - EMODnet Open Sea Lab II Hackathon (Pascal Derycke)

    • Packages 20’
    • Toolbox 20’
    • Required actions/contributions 20’

     

    Tuesday 2 October 2018

    09:30-11:30 - Status EMODnet

    • EU Open Data portal / GEOSS (geoportal.org)/ European Atlas of the Seas (Pascal Derycke) 30’
    • INSPIRE updates 30’
    • Data services (VLIZ) 1h
      • Data and data products categorization 30’
      • Query tool update 30’

    11:30-13:30 - Overview of new data products & developments by thematic portals

    Thematic lot representatives present (15’ max) an overview of existing/upcoming data products, functionalities and Web services & links to the Central portal. Particular attention should be given to technical challenges of common concern.

    • Data Ingestion 15’

    13:30-14:30 – Lunch Break

    14:30-15:45 – Technical support & other updates

    • JIRA 15’
    • Marine ID 15’
    • Background layer and other reference layers 15’
    • Specific questions & technical issues 15’
    • Other updates 15’

    15:45-16:00 - Wrap up, actions, next meeting (Secretariat)

     

    Add new comment

    It is not necessary to be logged in to provide comments but in that case comments will need to go through a moderator.
    CAPTCHA
    This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.