We will upgrade the Confluence Wikis platform.
In the meantime, and report any issue to DIGIT-WIKIS-SUPPORT@ec.europa.eu. You will be able to check new features from the release notes.
Skip to end of metadata
Go to start of metadata


1. Purpose

This document provides explanations about the import of SDMX-ML files and Excel files for creating or editing Metadata files and Guidelines, along with extra annotations used to be more accurate and compliant with the application ESS-MH.

These annotations cover the following subjects:

  • Importation of concept values of various representation types
  • Images
  • Annexes – at the concepts level & global
  • Publication flag (for the whole Metadata file)
  • Restricted for publication flag (at the concepts level)

 

2. Introduction

A metadata file can be exchanged by:

Using only an SDMX-ML file if there is no annex and no image,

  • Should follow the SDMX standard (with a <header> and specific tags…)
  • Including the content itself following a valid MSD (Metadata Structure Definition, defining especially the reported concepts and its structure).

Using only an xlsx file if there is no annex and no image,

  • Should follow a certain template complying with the SDMX standard
  • Including the content itself following a valid MSD (Metadata Structure Definition, defining especially the reported concepts and its structure).

Using a zip file

  • "SDMX" zip file (necessary if the file refers to annexes or images). Such an "SDMX" zip file should have a specific structure and contain an SDMX-ML file.
  • "Excel" zip file (necessary if the file refers to annexes or images). Such an "Excel" zip file should have a specific structure and contain an xlsx file.


2.1 "SDMX" zip file

It should be used when the metadata file refers to annexes or images.

In ESSMH the files used to be called like xx.sdmx.zip (where xx is the name of the metadata file). Note that the file to be imported can have a different name, as long as the extension .sdmx.zip is kept.


Example: DMBB_SIMS_A_4D_2020_0000.sdmx.zip.
            

This zip file contains:

  • One SDMX-ML file named xx.sdmx.xml (where xx is preferably the metadata file name)
    In our case: DMBB_SIMS_A_4D_2020_0000.sdmx.xml
  • If there are annexes, one directory Annexes/ containing one file per annex:
    • Each annex is named like xx_annn.ext (where xx is the metadata file name, nn is a number starting from 1, ext is the extension of the annex according to the type of document).
      Example: DMBB_SIMS_A_4D_2020_0000_an1.xlsx, DMBB_SIMS_A_4D_2020_0000_an2.docx
    • Each annex should have a reference in the SDMX-ML file (like "Annexes/xx_annn.ext").
    • Note that you can put your own name (instead of xx_annn), but you should keep the extension, put the file in the "Annexes" directory, and refer to it in the SDMX-ML file by using "Annexes/YourOwnName.ext".
  • If there are images, one directory Images/ containing one file per image:
    • Each image is named like xx_imgnn.ext (where xx is the metadata file name, nn is a number starting from 1, ext is the extension of the image).
      Example: DMBB_SIMS_A_4D_2020_0000_img1.png
    • Each image should have a reference in the SDMX-ML file.
    • Note that you can put your own name (instead of xx_imgnn), but you should keep the extension, put the file in the "Images" directory, and refer to it in the SDMX-ML file by using "Images/YourOwnName.ext".
  • Optionally one MSD file (in SDMX-ML format) named yy.msd.xml (where yy is the name of the MSD in the SDMX registry).
    Example: SIMS_MSD.msd.xml

For annexes, all formats are available but keep in mind that every person should be able to open it and sometimes might need a specific program to open them. Therefore we propose to use "standard" files like PDF, XLSX, DOCX, jpg, png for example.

For images, JPEG (.jpg) is probably the lightest format but .png, .gif and .bmp are also accepted.

Currently there is a limit for the zip file of 5 MB.

2.2 "Excel" zip file

It should be used when the metadata file refers to annexes or images.

In ESSMH we are used to call the file like xx.xlsx.zip (where xx is the name of the metadata file). Note that the file to be imported can have a different name, as long as the extension .xlsx.zip is kept.


Example: DMBB_SIMS_A_4D_2020_0000.xlsx.zip

This zip file contains:

  • One xlsx file named xx.xlsx (where xx is preferably the metadata file name)
    In our case: DMBB_SIMS_A_4D_2020_0000.xlsx
  • If there are annexes, one directory Annexes/ containing one file per annex:
    • Each annex is named like xx_annn.ext (where xx is the metadata file name, nn is a number starting from 1, ext is the extension of the annex according to the type of document).
      Example: DMBB_SIMS_A_4D_2020_0000_an1.xlsx, DMBB_SIMS_A_4D_2020_0000_an2.docx
    • Each annex should have a reference in the xlsx file (like "Annexes/xx_annn.ext").
    • Note that you can put your own name (instead of xx_annn), but you should keep the extension, put the file in the "Annexes" directory, and refer to it in the xlsx file by using "Annexes/YourOwnName.ext".
  • If there are images, one directory Images/ containing one file per image:
    • Each image is named like xx_imgnn.ext (where xx is the metadata file name, nn is a number starting from 1, ext is the extension of the image).
      Example: DMBB_SIMS_A_4D_2020_0000_img1.png
    • Each image should have a reference in the xlsx file.
    • Note that you can put your own name (instead of xx_imgnn), but you should keep the extension, put the file in the "Images" directory, and refer to it in the xlsx file by using "Images/YourOwnName.ext".
  • Optionally one MSD file (in SDMX-ML format) named yy.msd.xml (where yy is the name of the MSD in the SDMX registry).
    Example: SIMS_MSD.msd.xml

For annexes, all formats are available but keep in mind that every person should be able to open it and sometimes might need a specific program to open them. Therefore we propose to use "standard" files like PDF, XLSX, DOCX, jpg, png for example.

For images, JPEG (.jpg) is probably the lightest format but .png, .gif and .bmp are also accepted.

Currently there is a limit for the zip file of 5 MB.


3. SDMX-ML file

In order to retrieve the correct template for the SDMX-ML used for the import, it is advised to download first an example file from ESS-MH, optimally from a report based on the same MSD/typology.

The name of the file to be prepared for the import is like xx.sdmx.xml (where xx is preferably the metadata file name)
Example: DMBB_SIMS_A_4D_2020_0000.sdmx.xml



3.1 <Header> tag

The <Header> is mandatory and should contain all the tags presented in the example.


Some of the tags in the <Header> are mandatory to contain a value,


ItemsUsage statusSpecificationExample
<ID>optionalNo value needed here.DMBB_SIMS_A
<Test>optionalNo value needed here.false
<Name>mandatoryThe name of the metadata file is required here. The name is basically a composition of the Target values that need to be provided in the <MetadataSet> tag. It follows the structure DataFlow_DataProvider_TimeDimension

DMBB_SIMS_A_4D_2020_0000

<Prepared>optionalNo value needed here, but may contain a date in the format yyyy-nn-ddThh:mn:ss (like 2020-01-01T00:00:00)2020-11-20T11:34:00
<Sender id>mandatoryThe <Sender> should have the same value as the DataProvider in the <TargetValues> in <MetadataSet> tag (<genericmetadata:ComponentValue component="DATA_PROVIDER" object="DataProvider">).4D0
<DataSetID>optionalCan take any value, but may optimally be the same as in <Name>. Upon file creation, the DataSetID is generated automatically by the application ESS MH and takes the following form: DataFlow_n where n is an application specific identifier automatically generated for storing the file in the ESS MH database.DMBB_SIMS_A_1605868455406
<DataSetAction>optional

To edit an existing file, normally the Import is used by checking the "overwrite" box in ESS-MH UI. The content of the new file replaces the old one completely.

With the <DataSetAction> tag, an incremental import is facilitated: The application is able to import partial files of the same report while keeping the existing content. Based on the value in the (optional) <DataSetAction> tag in the <Header> the following actions can be carried out:

  • Append: New data (i.e. concepts that are not empty) may be added, but cannot overwrite or delete existing ones.
  • Replace: New data (i.e. concepts that are not empty) may be added and also replace existing one. 
  • Delete: New data (both, concepts that are empty or contain a value) replace existing ones and old ones are deleted if the new ones are missing values.
  • [missing value in the<DataSetAction> tag]: the new file replaces the existing one entirely (= “Delete”, complete overwrite of the file).


Note:

  • The "overwrite" checkbox in ESS-MH UI is taken into account only when there is no action tag in the SDMX-ML file. The action tag has priority over the checkbox.
  • A downloaded file contains by default an empty <DataSetAction> tag.

<Extracted>optionalNo value expected here.
<ReportingBegin>optionalNo value needed here, but may contain a date in the format yyyy-nn-ddThh:mn:ss.2020-01-01T00:00:00
<ReportingEnd>optionalNo value needed here, but may contain a date in the format yyyy-nn-ddThh:mn:ss.2020-12-31T00:00:00
<DataSetAgency>optional

The <DataSetAgency> tag contains the Agency, which should be defined in the ESS-MH Application. A downloaded file contains by default "ESTAT".

ESTAT


3.2 <MetadataSet> tag

The <MetadataSet> tag depends completely on the MSD.


3.2.1 <TargetValues>

The name of the file in ESS-MH (on an import) is defined based on the Full Target Identifiers / the tags. (Only the Full target Identifiers are taken into account, the Partial Target Identifiers are ignored)

tagExample

<genericmetadata:ComponentValue component="DATAFLOW" object="DataFlow">

DATAFLOW contains:

  • The code for the Domain (Category), here 'DMBB'.
  • An additional label indicating also the report structure used, as specified in the Metadata flow. Here 'SIMS'.
  • The periodicity, as specified in the Metadata flow. Here 'A' for annual. 
  • The version of the Metadata flow is optional, here ':1.0'
DMBB_SIMS_A:1.0

<genericmetadata:ComponentValue component="DATA_PROVIDER" object="DataProvider">

  • Contains the specific Organisation code that applies for the Metadata file. It should be the same as the Sender ID as provided in the <Header> tag.
4D0

<genericmetadata:ComponentValue component="TIME_PERIOD" object="TimeDimension">

TIME_PERIOD can contain:

  • The year for which we are reporting (here 2020),
  • The periodicity (here 'A' for annual) which is the same as the last character of the DATAFLOW,
  • The value for the period depending on the periodicity (here '0' because there is no period for annual flows). The following periodicities are currently in use in ESS-MH:

Periodicity 

Description

Possible period

2                  

(every 2 years)

0

3                 

(every 3 years)

0

4                 

(every 4 years)

0

5                  

(every 5 years)

0

A                

(annual)

0

M                

(monthly)

1 to 12

Q                

(quarterly)

1 to 4

2020-A0

 

3.2.2 Reported Metadata Attributes - The concepts and their values

  • Are reported in a tag <genericmetadata:ReportedAttribute conceptID=…>
  • Their value should be put in the tag <genericmetadata:Value> (without any '<' or '>').
  • The order and hierarchy should follow the MSD. 
  • The representation types of the concepts are specified in the MSD. They can have String format, Date, reference to a Code List (CL_...) or be of type Boolean. In the table below are described some examples for each representation type.


Representation of Metadata attribute / conceptSpecificationExample
String

Simple plain text

Plain text can be imported.

HTML entities, allowing to introduce HTML tags for special formatting

  • &amp; becomes & (ampersand)
  • &lt; becomes < (less than)
  • &gt; becomes > (greater than)

Images

  • Should be referenced in one concept as an <img> tag (where '<' and '>' should be encoded as '&lt;' and '&gt;')
  • <genericmetadata:Value>&lt;p&gt;&lt;img src="Images/xx_imgnn.ext" alt="" name="xx_imgnn.ext" /&gt;&lt;/p&gt;</genericmetadata:Value>
    • where xx is the metadata file name, nn is a number starting from 1, ext is the extension of the image
    • src contains the reference to the image file in the accompanied Images/ directory
    • name gives the name of the file 
    • Note that you can put your own name (instead of xx_imgnn), but you should keep the extension, put the file in the "Images" directory, and refer to it in the SDMX-ML file by using "Images/YourOwnName.ext".
  • And the file should be present in the subdirectory Images of the "SDMX" zip file with the specified name.

Simple plain text

<genericmetadata:Value>Here is some plain text.</genericmetadata:Value>

HTML entities

<genericmetadata:Value>&lt;p&gt;This is a list with different items&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;ABCDEFGHIJKLMN&lt;/li&gt;
&lt;ul&gt;
&lt;li&gt;abcdefghijklmn&lt;/li&gt;
&lt;ul&gt;
&lt;p&gt;&amp;nbsp;&lt;/p&gt;
&lt;p&gt;Text on level 1 without bullets or numbering&lt;/p&gt;
&lt;p style="padding-left: 30px;"&gt;Text on level 2&amp;nbsp;without bullets or  numbering&lt;/p&gt;</genericmetadata:Value>

Images

<genericmetadata:Value>&lt;p&gt;Eurostat&lt;/p&gt;&lt;p&gt;&lt;img        src="Images/DMBB_SIMS_A_4D_2020_0000_img1.png" alt="" name="DMBB_SIMS_A_4D_2020_0000_img1.png" /&gt;&lt;/p&gt;</genericmetadata:Value>

Date

The dates contained e.g. in the concepts

META_CERTIFIED

META_POSTED

META_LAST_UPDATE

should be inserted as plain text in format dd/mm/yyyy

<genericmetadata:Value>20/11/2020</genericmetadata:Value>
Code list

A concept can refer to a specific Code List from which values are to be chosen. The underlying code list is given in the MSD at the respective Metadata Attribute. Details on the Code list with its different Code values can be retrieved from the MSD from the tag <CodeLists>

ESS-MH accomodates single selection of values from a Code list as well as multiple selection. This definition is made in the application exclusively and is not retrievable from the MSD. For such cases, it needs to be clarified with the Metadata Support or the Production Domain, if single or multiple selection mode applies. In case, they have been made available, tailored Guidelines may also provide useful information.

Single value code list

<genericmetadata:Value>codeID</genericmetadata:Value>

Multiple value code list

<genericmetadata:Value>codeID1</genericmetadata:Value>

<genericmetadata:Value>codeID2</genericmetadata:Value>

<genericmetadata:Value>codeID3</genericmetadata:Value>

<genericmetadata:Value>codeID4</genericmetadata:Value>

Example of values for concepts, referenced to CL_ESMSIP_GRADE_1


Single value code list

</genericmetadata:ReportedAttribute>

<genericmetadata:Value>HIGH</genericmetadata:Value>

</genericmetadata:ReportedAttribute>

Multiple value code list

<genericmetadata:ReportedAttribute conceptID="...">

<genericmetadata:Value>HIGH</genericmetadata:Value>

<genericmetadata:Value>LOW</genericmetadata:Value>

<genericmetadata:Value>MEDIUM</genericmetadata:Value>

<genericmetadata:Value>NOT_APPLICABLE</genericmetadata:Value>

<genericmetadata:Value>NOT_AVAILABLE</genericmetadata:Value>

</genericmetadata:ReportedAttribute>

Boolean

A concept can be of type Boolean, which is specified in the MSD at the respective Metadata Attribute. The concept values can be TRUE or FALSE. The application ESS-MH is not case sensitive in that respect and accepts values also in lowercase true or false.

</genericmetadata:ReportedAttribute>

<genericmetadata:Value>TRUE</genericmetadata:Value>

</genericmetadata:ReportedAttribute>


</genericmetadata:ReportedAttribute>

<genericmetadata:Value>FALSE</genericmetadata:Value>

</genericmetadata:ReportedAttribute>


3.2.3 Annexes

Annexes can be

  • Global to the whole metadata file
  • Or specific to one concept.

Global annexes should be added

  • in the tag <genericmetadata:MetadataSet>,
  • after </genericmetadata:AttributeValueSet>,
  • in a tag <genericmetadata:Annotations>
    and <genericmetadata:Annotation> for each annex.


Concept's annexes should be added

  • in the related tag <genericmetadata:ReportedAttribute conceptID=…>
  • in a tag <genericmetadata:Annotations>
    and <genericmetadata:Annotation> for each annex.


The different types of an annex are:

  • FILE
  • URL


TypeSpecificationExample
FILE
  • A tag <common:AnnotationType> containing "FILE"
  • A title put in the tag <common:AnnotationTitle>
  • Refer to a file put in the tag <common:AnnotationText>
  • And the file should be present in the subdirectory Annexes of the "SDMX" zip file with the specified name.

<common:Annotation>

        <common:AnnotationType>FILE</common:AnnotationType>

        <common:AnnotationTitle>Annex 1 Excel file</common:AnnotationTitle>

<common:AnnotationText>Annexes/DMBB_SIMS_A_4D_2020_0000_an1.xlsx</common:AnnotationText>

</common:Annotation>

URL
  • A tag <common:AnnotationType> containing "URL"
  • A title put in the tag <common:AnnotationTitle>
  • And an URL put in the tag <common:AnnotationURL>

<common:Annotation>

        <common:AnnotationType>URL</common:AnnotationType>

        <common:AnnotationTitle>Eurostat Website</common:AnnotationTitle>

        <common:AnnotationURL>https://ec.europa.eu/eurostat/web/main/home</common:AnnotationURL>

</common:Annotation>


3.2.4 Publication flags and Restriction flags


SpecificationExample
Publication flag for whole Metadata file

If the Metadata file should be published, an annotation for the "Publication flag"

  • should be present
  • and be set to "YES".
  • in the tag <genericmetadata:MetadataSet>,
  • after </genericmetadata:AttributeValueSet>,
  • in a tag <genericmetadata:Annotations>
    <genericmetadata:Annotation>
    <common:AnnotationType> containing "FOR_PUBLICATION"
    <common:AnnotationText> containing "YES".

<common:Annotation>

<common:AnnotationType>FOR_PUBLICATION</common:AnnotationType>

<common:AnnotationText>YES</common:AnnotationText>

</common:Annotation>

Restricted for publication flag at the concepts level

For each concept which values should not be published, an annotation for the "Restricted for publication flag"

  • should be present
  • and set to "YES".

The value "Restricted for publication" will appear as content of the concept when the file is published.

Such flag should be added

  • in the related tag <genericmetadata:ReportedAttribute conceptID=…>
  • in a tag <genericmetadata:Annotations>
    <genericmetadata:Annotation>
    <common:AnnotationType> containing "RESTRICTED_FOR_PUBLICATION"
    <common:AnnotationText> containing "YES".

<common:Annotation>

<common:AnnotationType>RESTRICTED_FOR_PUBLICATION</common:AnnotationType>

<common:AnnotationText>YES</common:AnnotationText>

</common:Annotation>


4. xlsx file 

In order to retrieve the correct template for the xlsx used for the import, it is advised to download first an example file from ESS-MH, optimally a comparable file from a report based on the same MSD/typology.

The name of the file to be prepared for the import is like xx.xlsx (where xx is preferably the metadata file name)
Example: DMBB_SIMS_A_4D_2020_0000.xlsx

The xlsx file comprises three sheets that serve different purposes and require different degree of filling.

  • Metadata
  • Parameters
  • Annexes

4.1 Metadata - "Header" section

Most parts in the Header section are mandatory and should contain the items presented in the example.

The Metadata file will be assembled based on the values provided in cells B4:B8.



ItemsUsage statusSpecificationExample
Filenameoptionalequivalent to SDMX-ML fileDMBB_SIMS_A_4D_2020_0000
Published nameoptionalThe download of a metadata file includes also the published name of it, specified by the code of the node in the DataNavigationTree, the file has been attached to, the typology used and, in case of national files, the country code of the file. 

scitech_sims

(or in case of a national file, e.g. scitech_sims_lu)

Data flow IDmandatoryequivalent to SDMX-ML fileDMBB_SIMS_A
Data flow versionmandatoryequivalent to SDMX-ML file1.0
Organization codemandatoryequivalent to SDMX-ML file4D0
Time dimensionmandatory

equivalent to SDMX-ML file

2020-A0
For publicationmandatory

If the Metadata file should be published, the value "For Publication" should be set to YES, otherwise to NO.


YES
Data set actionoptional

equivalent to SDMX-ML file



4.2 Metadata "Concepts" section 

The Concepts section depends completely on the MSD. The order and hierarchy should follow the MSD. 

Column A contains the "Concept name" with the corresponding numbering based on the typology in ESS-MH. This information is retrievable from the underlying MSD to a certain extent, while the numbering is autogenerated by the application ESS-MH upon building the typology. Otherwise, this information can be obtained directly from the ESS-MH, or can be clarified with the Metadata Support or the Production Domain. 


4.2.1 Concept values

The concept values should be put in cells in column B "Concepts". According to the specifications in the MSD, concept values can be expressed in different representation types:


Representation of Metadata attribute / conceptSpecificationExample
String

Simple plain text

Plain text can be imported. Also carriage returns are accepted and turned into </br> tags upon import.

HTML tags

A downloaded metadata file in xlsx will contain values  with HTML tags, hence they are accepted respectively on import.

  • <p>, </br>, ...
  • <ol>, <ul>, <li>, ...
  • <a>, ...
  • <table>. <tbody>, <tr>, <td>, ...
  • <strong>, <span>, <em>...
  • <sub>, <sup>, ...
  • ...

Images

  • Should be referenced in one concept as an <img> tag 
  • <img src="Images/xx_imgnn.ext" alt="" name="xx_imgnn.ext">
    • where xx is the metadata file name, nn is a number starting from 1, ext is the extension of the image
    • src contains the reference to the image file in the accompanied Images/ directory
    • name gives the name of the file 
    • Note that you can put your own name (instead of xx_imgnn), but you should keep the extension, put the file in the "Images" directory, and refer to it in the xlsx file by using "Images/YourOwnName.ext".
  • And the file should be present in the subdirectory Images of the "Excel" zip file with the specified name.

Simple plain text

Here is some plain text.

HTML tags

<p><strong>This is text in bold.</strong></p>
<p><em>This is text in italics.</em></p>
<p><span style="text-decoration: underline;">This is text with underline.</span></p>
<p><span style="color: #ff0000;">This is text in red.</span></p>
<p><span style="background-color: #ffff00;">This is text with background color yellow.</span></p>
<p>This is text in <sub>subscript</sub>.</p>
<p>This is text in <sup>superscript</sup>.</p>
<p>These are symbols/special character:&nbsp;&amp;&nbsp;&zeta;&nbsp;&phi;</p>

<p><a href="https://ec.europa.eu/eurostat/web/main/home">https://ec.europa.eu/eurostat/web/main/home</a></p>

Images

<p><img src="Images/DMBB_SIMS_A_4D_2020_0000_img1.png" alt="" name="DMBB_SIMS_A_4D_2020_0000_img1.png" /></p>

Date

The dates contained e.g. in the concepts

2.1. Metadata last certified (META_CERTIFIED)
2.2. Metadata last posted (META_POSTED)
2.3. Metadata last update (META_LAST_UPDATE)

may inserted as plain text with apostrophe in format 'dd/mm/yyyy or formatted as Date, e.g. dd-mm-yyyy.

'20/11/2020

20-11-2020

Code list

A concept can refer to a specific Code List from which values are to be chosen. The underlying code list is given in the MSD at the respective Metadata Attribute. Details on the Code list with its different Code values can be retrieved from the MSD from the tag <CodeLists>

ESS-MH accomodates single selection of values from a Code list as well as multiple selection. This definition is made in the application exclusively and is not retrievable from the MSD. For such cases, it needs to be clarified with the Metadata Support or the Production Domain, if single or multiple selection mode applies. In case, they have been made available, tailored Guidelines may also provide useful information.


Single value code list

codeID in uppercase

Multiple value code list

codeID1,codeID2,codeID3,codeID4,...

Example of values for concepts, referenced to CL_ESMSIP_GRADE_1


Single value code list


Multiple value code list

Boolean

A concept can be of type Boolean, which is specified in the MSD at the respective Metadata Attribute. The concept values can be TRUE or FALSE


4.2.2 Restricted for publication flag (at the concepts level)

For each concept which should not be published, the corresponding cell in column C headed by "Restricted from publication" should be set to YES. Otherwise NO.

Then the value "Restricted for publication" will appear as content of the concept when the file is published.


4.3 Parameters

The second sheet is a Parameter sheet that specifies how the items and data in sheet "Metadata" should be coded and implemented in the Metadata files in ESS MH. Generally, when you work with a downloaded template of a comparable Metadata file from ESS MH, in sheet "Parameters" editing is neither expected nor advised.

806px700

In column A "Element name" you can find all items as per column A in the Metadata sheet.

Column B "Element code" lists the related IDs for the items and concepts.

Column C "Type" specifies the items further in terms of function and representation in the Metadata file while column D "Position" indicates the rows of the items of Metadata sheet, i.e. references the elements from the two sheets Metadata and Parameters and defines, which values will be taken into account for the importation. 

Only, if you plan to set up your file import per Excel file from scratch with a blank file, or you plan to import only a subset of concepts to ESS-MH, you should ensure to align the Parameters sheet with the Metadata sheet respectively. 


4.4 Annexes

Equivalent to SDMX-ML files, Annexes can be attached

  • specifically to one concept,
  • globally to the whole metadata file.

The different types of an annex are:

  • FILE

  • URL

Note: You can attach as many Annexes as necessary, provided you keep the maximum size of the entire Excel.zip file of 5MB.


Annexes need to be added in the corresponding "Annexes" sheet, both for concepts and globally for the whole metadata file.


In Column A "Concept", you specify whether an Annex is supposed to be attached

  • to a concept by providing here the corresponding concept ID (can be looked up from the Parameters sheet),
  • to the metadata file globally by leaving the cell blank.

In column B "Annex Type" you specify whether an Annex is of the type

  • FILE, or
  • URL.

In column C "Annex Name", you reference the Annex by providing

  • which file needs to be present from the subdirectory Annexes of the "Excel" zip file with the specified name.
  • the concrete URL.

In column D "Annex Description", you specify the custom name of the Annex, as you wish it to appear in the HTML view of the metadata file later.


5. Guidelines

Users with role VALIDATOR may also create and edit Guidelines by file import. The importation mechanism is equivalent to the one for Metadata files, you can choose between 

Using only an SDMX-ML file if there is no annex and no image,

  • Should follow the SDMX standard (with a <header> and specific tags…)
  • Including the content itself following a valid MSD (Metadata Structure Definition, defining especially the reported concepts).

Using only an xlsx file if there is no annex and no image,

  • Should follow a certain template complying with the SDMX standard
  • Including the content itself following a valid MSD (Metadata Structure Definition, defining especially the reported concepts).

Using a zip file

  • "SDMX" zip file (necessary if the file refers to images). Such an "SDMX" zip file should have a specific structure and contain an SDMX-ML file.
  • "Excel" zip file (necessary if the file refers to images). Such an "Excel" zip file should have a specific structure and contain an xlsx file. 


Also for the guidelines, it is advised to download first an example file from ESS-MH, optimally from a report based on the same MSD/typology in order to retrieve the correct template for the SDMX-ML or Excel files used for the import.

Note: Annexes and Publication flags or 'Restricted from publication' flags are not part of the guidelines.

5.1 SDMX-ML file - specifics for "Guidelines"

In ESS-MH the guidelines used to be called like esms-xx-YYYY.sdmx (where xx is the name of the metadata flow and YYYY the reference year from which on the guidelines is valid). 


The imported file should be archived in an sdmx.zip file, when the guidelines file refers to images. Note that the file to be imported can have a different name, as long as the extension .sdmx.zip is kept.

Example for guidelines containing Images: esms-DMAA_ESMS_A-2019.sdmx.zip.
The zip file contains:

  • One SDMX-ML file named esms-xx-YYYY.sdmx.xml (where xx is preferably the metadata flow name)
    In our case: esms-DMAA_ESMS_A-2019.sdmx.xml
  • If there are images, one directory Images/ containing one file per image.

For images, JPEG (.jpg) is probably the lightest format but .png, .gif and .bmp are also accepted.

Currently there is a limit for the zip file of 5 MB.

5.1.1 <Header> tag

The <Header> is mandatory and should contain all the tags presented in the example.

Except from the <Sender id>, these tags are informative.
The <Sender> should have the same value as the Data Provider in the Full target Identifier / <genericmetadata:ComponentValue component="DATA_PROVIDER" object="DataProvider"> . The <Sender> is generally "ESTAT".

The tags <ReportingBegin> and <ReportingEnd> should contain a reference year in the format yyyy (like 2019 and 2020). 


5.1.2 <MetadataSet> tag

The <MetadataSet> tag depends completely on the MSD.


5.1.2.1 <TargetValues>

The name of the file in ESS-MH (on an import) is defined based on the Full Target Identifiers / the tags. (Only the Full target Identifiers are taken into account, the Partial Target Identifiers are ignored)



tagExample
<genericmetadata:ComponentValue component="DATA_PROVIDER" object="DataProvider"> is always "ESTAT"ESTAT

<genericmetadata:ComponentValue component="DATAFLOW" object="DataFlow">

DATAFLOW contains:

  • The code for the Domain (Category), here DMAA.
  • An additional label indicating also the report structure used, as specified in the Metadata flow. Here ESMS.
  • The periodicity, as specified in the Metadata flow. Here 'A' for annual. 
DMAA_ESMS_A
<genericmetadata:ComponentValue component="AGENCY" object="Agency"> is always "ESTAT"ESTAT

<genericmetadata:ComponentValue component="START_YEAR" object="StartYear">

  • The reference year from which on the guidelines are applicable.
2019

<genericmetadata:ComponentValue component="END_YEAR" object="EndYear">

  • The reference year until which the guidelines are applicable.
2020

<genericmetadata:ComponentValue component="MDFLOW_VERSION" object="MdFlowVersion">

  • The version of the Metadata flow concerned.
1.1

 

5.1.2.2 Reported Metadata Attributes - The concepts and their values

  • Are reported in a tag <genericmetadata:ReportedAttribute conceptID=…>
  • Their value should be put in the tag <genericmetadata:Value> (without any '<' or '>').
  • The order and hierarchy should follow the MSD. 
  • The representation types of the concepts values can have String format and can be inserted as



SpecificationExample
Simple plain text

Plain text can be imported.

<genericmetadata:Value>Here is some plain text.</genericmetadata:Value>

HTML entities

HTML entities allowing HTML tags to accomode special formatting

  • &amp; becomes & (ampersand)
  • &lt; becomes < (less than)
  • &gt; becomes > (greater than)
<genericmetadata:Value>&lt;p&gt;This is a list with different items&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;ABCDEFGHIJKLMN&lt;/li&gt;
&lt;ul&gt;
&lt;li&gt;abcdefghijklmn&lt;/li&gt;
&lt;ul&gt;
&lt;p&gt;&amp;nbsp;&lt;/p&gt;
&lt;p&gt;Text on level 1 without bullets or numbering&lt;/p&gt;
&lt;p style="padding-left: 30px;"&gt;Text on level 2&amp;nbsp;without bullets or  numbering&lt;/p&gt;</genericmetadata:Value>
Images
  • Should be referenced in one concept as an <img> tag (where '<' and '>' should be encoded as '&lt;' and '&gt;')
  • And the file should be present in the subdirectory Images of the "SDMX" zip file with the specified name.

<genericmetadata:Value>&lt;p&gt;Eurostat&lt;/p&gt;&lt;p&gt;&lt;img        src="Images/DMBB_SIMS_A_4D_2020_0000_img1.png" alt="" name="DMBB_SIMS_A_4D_2020_0000_img1.png" /&gt;&lt;/p&gt;</genericmetadata:Value>

5.2 Excel file - specifics for "Guidelines"

In ESS-MH the guidelines used to be called like esms-xx-YYYY.xlsx (where xx is the name of the metadata flow and YYYY the reference year from which on the guidelines is valid). 


The imported file should be archived in an xlsx.zip file, when the guidelines file refers to images. Note that the file to be imported can have a different name, as long as the extension .xlsx.zip is kept.

Example for guidelines containing Images: esms-DMAA_ESMS_A-2019.xlsx.zip.
The zip file contains:

  • One SDMX-ML file named esms-xx-YYYY.xlsx (where xx is preferably the metadata flow name)
    In our case: esms-DMAA_ESMS_A-2019.xlsx
  • If there are images, one directory Images/ containing one file per image.

For images, JPEG (.jpg) is probably the lightest format but .png, .gif and .bmp are also accepted.

Currently there is a limit for the zip file of 5 MB.

The xlsx file comprises two sheets that serve different purposes and require different degree of filling.

  • Metadata
  • Parameters

5.2.1 Metadata - "Header" section

Most parts in the Header section are mandatory and should contain the items presented in the example.

The Metadata file will be assembled based on the values provided in cells B3:B7.



ItemsUsage statusSpecificationExample
FilenameoptionalEquivalent to SDMX-ML fileesms-DMAA_ESMS_A-2019
Data flow IDmandatoryequivalent to SDMX-ML file, contains:
  • The code for the Domain (Category), here DMAA.
  • An additional label indicating also the report structure used, as specified in the Metadata flow. Here ESMS.
  • The periodicity, as specified in the Metadata flow. Here 'A' for annual. 
DMAA_ESMS_A
Data flow versionmandatoryEquivalent to SDMX-ML file1.1
typology codemandatorySpecifies the ESS-MH specific typology referred to, based on the MSD.esms
Start yearmandatory

Equivalent to SDMX-ML file: The reference year from which on the guidelines are applicable.

2019
End Yearmandatory

Equivalent to SDMX-ML file: The reference year until which the guidelines are applicable.

2020


5.2.2 Metadata "Concepts" section 

The Concepts section depends completely on the MSD. The order and hierarchy should follow the MSD. 

Column A contains the "Concept name" with thee corresponding numbering based on the typology in ESS-MH. This information is retrievable from the underlying MSD to a certain extent, while the numbering is autogenerated by the application ESS-MH upon building the typology. Otherwise, this information can be obtained directly from the ESS-MH, or can be clarified with the Metadata Support or the Production Domain. 


The concept values should be put in cells in column B "Concepts". Concept values can be expressed as string.



SpecificationExample
Simple plain text

Plain text can be imported. Also carriage returns are accepted and turned into </br> tags upon import.

Here is some plain text.

HTML tags

A downloaded metadata file in xlsx will contain values  with HTML tags, hence they are accepted respectively on import.

  • <p>, </br>, ...
  • <ol>, <ul>, <li>, ...
  • <a>, ...
  • <table>. <tbody>, <tr>, <td>, ...
  • <strong>, <span>, <em>...
  • <sub>, <sup>, ...
  • ...

<p><strong>This is text in bold.</strong></p>
<p><em>This is text in italics.</em></p>
<p><span style="text-decoration: underline;">This is text with underline.</span></p>
<p><span style="color: #ff0000;">This is text in red.</span></p>
<p><span style="background-color: #ffff00;">This is text with background color yellow.</span></p>
<p>This is text in <sub>subscript</sub>.</p>
<p>This is text in <sup>superscript</sup>.</p>
<p>These are symbols/special character:&nbsp;&amp;&nbsp;&zeta;&nbsp;&phi;</p>

<p><a href="https://ec.europa.eu/eurostat/web/main/home">https://ec.europa.eu/eurostat/web/main/home</a></p>

Images
  • Should be referenced in one concept as an <img> tag 
  • And the file should be present in the subdirectory Images of the "Excel" zip file with the specified name.

<p><img src="Images/DMBB_SIMS_A_4D_2020_0000_img1.png" alt="" name="DMBB_SIMS_A_4D_2020_0000_img1.png" /></p>


5.2.3 Parameters

The second sheet is a Parameter sheet that specifies how the items and data in sheet "Metadata" should be coded and implemented in the Metadata files in ESS MH. Generally, when you work with a downloaded template of a comparable guideline file from ESS MH, in sheet "Parameters" editing is neither expected nor advised.

In column A "Element name" you can find all items as per column A in the Metadata sheet.

Column B "Element code" lists the related IDs for the items and concepts.

Column C "Type" specifies the items further in terms of function and representation in the guidelines file while column D "Position" indicates the rows of the items of Metadata sheet, i.e. references the elements from the two sheets Metadata and Parameters and defines, which values will be taken into account for the importation. 

Only, if you plan to set up your file import per Excel file from scratch with a blank file, you should ensure to align the Parameters sheet with the Metadata sheet respectively.