WPE Overview English (en) français (fr)

From ESSnet Big Data
Jump to: navigation, search

Objectives

Taking into account the developments of the successfully concluded work in the ESSnet big data programme during 2016-2018 (see ESSnet Big Data I WP4 AIS data), the aim of WPE is to develop functional production prototypes including setting up procedures and developing technical solutions, to promote and support the collection, processing and analysis of (big) data from AIS (Automatic Identification System) for statistical production in the participating NSIs related to statistics on maritime, inland waterways and environmental statistics.

The output of the work should become available to the ESS and adequately enable other NSIs, in the next implementation phase, to put in place the necessary system(s) and deploy or adapt the proposed solution(s) in order to produce statistical output. From the practical point of view, the output should in addition develop key artefact like target architecture definition and requirements and guidance for the next implementation and deployment phase, hence producing specifications for full-fledged implementation at the ESS level. Therefore, the active involvement of subject matter experts, statisticians, enterprise architects and other stakeholders (for example Working Groups of Inland waterways and maritime statistics, Transport and Evironment Reporting Mechanism (TERM) and Task Forces) is important. Subject matter experts for the NSIs working in the relevant national statistical production units will be involved in all phases of work. Participation to relevant Eurostat Working groups and Task Forces will be part of the work of WPE. This includes presentation of findings and work to the relevant working groups at an early stage.

Along the production of the relevant methodologies, recommendations, specifications and statistical software, the production of experimental statistics demonstrating the capabilities to produce statistics is a prominent objective.

Methodological, quality and technical results of WPE, including intermediate findings, will be used as inputs for WPF Process and architecture and WPK Methodology and quality. When carrying out the tasks listed below, care will be taken that these results will be stored for later use at the ESSnet Big Data website.

Description of work

With respect to the development and implementation of functional production prototypes according to the work carried out during 2016-2018 by the preceding ESSnet, this WP aims at completing the following activities that would lead to the implementation and integration of AIS data within the national statistical production process(es) in the NSIs:

Task 1 - Interaction with WPF

Performed by CBS (Statistics Netherlands)

Aim of this task is coördinating the interaction with WPF. The aim of WPF is the definition of reference architectures necessary to carry out big data production both at national and European levels. This task includes (i) designing and adopting application and information architectures (WPF Task 1) and (ii) providing input to WPF Task 2.1 on solution architectures.

Task 2 – Data access

Performed by CBS (Statistics Netherlands)

AIS are internationally standardised encoded data available for national territories and the entire European territory, as well as at global level. The used European datasource in ESSnet big data programme 2016-2018 isn’t recent enough to use for implementation.

WPE aims to implement AIS in maritime statistics as well as in inland waterway statistics. That is why it is important to have European AIS data on maritime ships available as well as national data on Inland waterway ships. The European Maritime Safety Agency (EMSA) was requested for European AIS data from 2017 and 2018 on Maritime ships. Dutch AIS data on inland ships will be provided by Rijkswaterstaat for implementing AIS in the statistical process of inland waterways.

This task aims to:

  • Collect the data on maritime ships at a European level and collect the national data on Inland waterway ships for the Netherlands
  • Set a sustainable long term collaboration frame with EMSA for future use at European level. This will require clear roles related to data, pre-processing, processing and production of statistics. Preparation of this future use of European AIS data from EMSA is also part of this task.

For this project it is essential to get at least access to European AIS data (from EMSA). At this point it seems that EMSA could provide the data needed for this project, but there is no final agreement yet. Costst for buying European AIS data from a commercial organisation, in the case EMSA can not provide the data, are not budgeted yet.

Task 3 – Definition

Performed by ELSTAT (Statistics Greece, CBS (Statistics Netherlands) and GUS (Statistics Poland)

Aim of this task is to define the specific outcome of this WP. What would be the delivered statistical product (both new statistics and enhanced existing statistics) at the end of this ESSnet? In what statistical processes AIS would be implemented? How would subject matter experts, statisticians, enterprise architects and other stakeholders (Working Groups of inland waterways and maritime statistics and Task Forces) be involved?

Key elements of this task are:

  • Definition of statistical products/prototypes. Including a plan for possibilities for testing these prototypes in other countries.
  • Identification of statistical production processes and capabilities that may be affected at national level.
  • Guideline for involving subject matter experts, statisticians, enterprise architects and other stakeholders. This includes involving subject matter experts for the NSIs working in the relevant national statistical production units in all phases of work as well as participation to relevant Eurostat Working groups and Task Forces. EMSA will follow the progress during this project. EMSA is also available for (methodological) advice.
  • Description of an adequate Reference Methodological Framework for processing AIS data in statistical processes. Setting a sustainable long term collaboration frame with EMSA for future use of AIS data at European level will be included in building the necessary reference architecture using European data.
  • Development of data models (output in XML or other standard), data validation and data exchange protocols for all the selected statistical processes for implementation

Task 4 – Prototypes

Performed by ELSTAT (Statistics Greece), CBS (Statistics Netherlands) and GUS (Statistics Poland)

The aim of this task is the development of fully functional prototypes of the identified statistical production processes in task 3. Including a description on the adaptation and consolidation of the methodology, the procedures and the tools for collecting. Output would include software as well. This task also includes a definition of:

  • the implementation requirements (report) of prototypes in the relevant statistical production processes at European and national level (all)
  • a quality management framework (report) to ensure quality of outputs and process, leading to complementary statistics (indicators) (EL, NL)
  • required metadata throughout the process (report, metadata schema) (all)
  • a risk plan with mitigation scenarios (all)

Testing of the prototypes (software, architecture, technical requirements, etc.) will take place (if possible) in (at least) the participating countries of this WP. This testing phase would then allow corrections or development of alternatives to be adapted to national specificities of (at least) the partners in this WP.

Besides this, there will be taken into account a couple of recommendations resulting from the previous ESSnet on big data. These two topics will be further investigated in this ESSnet:

  • Combining AIS with online information and/or information on IHS to provide more reliable information on the type and capacity of ships (NL, EL).
  • Developing an algorithm for signaling anomalies in the movements of ships (e.g. different traffic intensities or travelling speeds) to get information on disruptions or (changed) regulations in ports and fairways (NL, PL).

The outcomes of the two above-mentioned investigations will be really helpful for further implementation of AIS in official statistics in this ESSnet. The three topics beneath will be further investigated in this WP as well:

  • Developing an algorithm that can construct a reference frame of ports. The next step would be to construct a reference frame of terminals. Combining this with type of visiting ship could assist with ship visits where type of ships or goods is missing. Now ports and terminals have to be defined manually, this is not feasible for countries with many ports (NL).
  • Improving the port to port distance matrix by Investigating more journeys and factors. Optimizing algorithms could also result in using actual AIS journey data instead of the average distance matrix. Optimizing algorithms could also result in using actual AIS journey data instead of the average distance matrix (NL).
  • Using AIS as a fast indicator of international trade, where international comparisons can be made quickly. National offices are really interested in fast indicators of economic development (NL)

On some of these topics there are already some first findings available, resulting from the ESSnet big data programme 2016-2018. It is not foreseen that the outcomes of these investigations will directly be implemented in one of the statistical offices during this ESSnet, but they will provide very valuable insights for the ESSnet as a whole.

The results of all investigations mentioned will be produced as experimental statistics on Eurostat’s dedicated section (task 4).

Task 5 – Experimental statistics

Performed by ELSTAT (Statistics Greece), CBS (Statistics Netherlands) and GUS (Statistics Poland)

The aim of this task is to produce experimental statistics by visualising the first results of the outcomes on the prototypes in task 4. The experimental statistics will be published most likely in the form of visualisations and they will be hosted by Eurostat's dedicated section and on a national level.

Task 6 – Implementation

Performed by ELSTAT (Statistics Greece), CBS (Statistics Netherlands) and GUS (Statistics Poland)

This task involves (the preparation of) implementating AIS in statistical processes. After finishing this task there will be at least: • A description (process model) on the future process of all selected processes in task 2, describing the processes including tools, infrastructure and conditions (for example skills needed, hardware requirements).

  • A report describing the activities needed for coming from the experimental statistic to the future process, for all selected processes in task 2. What needs to be done before the future process is ready for use and regular output can be published?
  • One statistical process where AIS is fully implemented, including:
  1. Production of tools and software (source code) that can be applied by other NSI’s
  2. Description of used methodologies
  3. A developer's and user's handbook
  4. A procedure for testing and maintenance.

Besides the above-mentioned deliverables, the following more general actions will be performed in this task:

  • Address issues related to sustainability of data sources, data use by NSIs, as well as data sharing between NSIs (apply risk management principles).
  • Develop and evaluate scenarios for data governance and data management in the wider aspect of data sharing and collaboration processes management.

Task 7 – Future perspectives

Performed by ELSTAT (Statistics Greece), CBS (Statistics Netherlands) and GUS (Statistics Poland)

Aim of this task is to summarise the project results in a final detailed report with the outcomes of this WP. Guidelines and documented material for full-fledged implementation in the ESS including relevant software. Setting a sustainable long term collaboration frame with EMSA for future use at European level will require clear roles related to data, pre-processing, processing and production of statistics. Preparation of this future use of European AIS data from EMSA will be will be part of this final report. The results will also include an assessment of sustainability over time.The results should include an assessment of sustainability over time.

Milestones and deliverables

See here for an overview of available milestones and deliverables.

WPE milestones

  EM1   Report on the WP meeting mid-2019   Month 9
  EM2   Report on the WP meeting mid-2020   Month 20

WPE deliverables

  E1  Report on the results of the definition process, concerning the Reference Methodological Framework and data models   Month 7  Results of Task 3.
  E2  Report on implementation requirements of relevant statistical production processes at European and national level, quality framework, required metadata and a risk plan   Month 12  Results of Task 4.
  E3  Interim technical report concerning the conditions for using the data, the methodology and the procedures to be used for producing statistics (development of functional prototypes, development toolkit and software), recommendations and guidelines for full-fledged implementation in the ESS.   Month 16  
  E4  Final detailed report with the outcomes: guidelines and documented material for full-fledged implementation in the ESS including relevant software, description of used methodologies, developer's and user's handbook, procedure for testing and maintenance and description of skills needed.   Month 24  The results should also include an assessment of sustainability over time.