WP1 Sprint 2016 07 28-29 Virtual Ideas (background) English (en) français (fr)

From ESSnet Big Data
Revision as of 13:27, 15 July 2016 by Nswierni (Talk | contribs)

Jump to: navigation, search

Virtual Sprint: 28-29 July:

<span style="font-size:large;"</span>

What is a "virtual sprint"? 

The concept behind this "virtual sprint" is for different members of a team based in different locations to work together to solve a common problem, or set of problems in a highly focused effort. This is in contravention of the 12 Principles of the Agile Manifesto which says that the best way of communicating is "face to face conversation". However, it may be that this event is not so much a "sprint", which normally lasts several weeks, but more like a "hack", which normally only spans from a few hours to a few days. So, it may be more correct to call this a "virtual hack", but it doesn't really matter. This is an experiment in working together and we will see what happens....

How will it work?

The general approach for this virtual sprint has been taken from several "hack days" run within the ONS Big Data team. This has involved team members based in two locations working on a common problem. One person takes the lead in organising the hack and the research tasks are generally related to a specific project that they are working on. Some preparation is involved, for example identifying and assigning tasks, and preparing test data sets etc. These hacks have lasted about 5 hours with a initial phone meeting to ensure that everyone in briefed on what to do and a closing meeting to present and discuss any findings. The experience at ONS has been very positive. They have not only produced some very useful ideas, but they also proved very good for team building, especially across sites.

This "virtual sprint" will be organised on similar lines. The main difference is that it will last two days instead of one. There will be three Webex calls at the following times:

28 July: 9:30 to 10:00 - CET (Final briefing)

28 July: 16:00 to  16:30 - CET (Checkpoint)

29 July: 15:30 to 16:30 - CET (Presentation and discussion of results)

There will be a greater number of locations, but hopefully this will not be a problem. The most important thing is to bring a "hacking mindset". The allocated task and suggested approaches are only meant to be a guide and it is perfectly valid to follow completely different approaches. Finding innovative solutions is the whole point of this exercise.

What will be the tasks?

There seems to be sufficient support for at least one of the tasks to be on methods for identifying duplicates. It is clear that this will be a key challenge when combining data from different portals, but in some cases within job portals and certainly for job search engines. This will require some knowledge of record linkage and machine learning methods.

Since areas of expertise and possibly access to suitable tools and enviroments may be problematic for some countries, it would be desirable to have another task. This could be for example a exercise in collecting job vacancy data, possibly through the use of simpke web scraping tools.

What preparation is needed?

1. Task identification and allocation

2. Background reading

Robert Breton has put together a useful set of links on methods for identifying duplicates

3. Environment and tools

4. Preparation of data

Joint analysis of CEDEFOP data? The query system we have access to has limitations and so we really need access to the underlying data. We are following this up but this may not be available in time

Ideas from Dan Wu (Stats Sweden)

In the final report of project ”Real-time labour market information on skill requirements: feasibility study and working prototype”, a data model developed has been presented, see the figure below. This figure is used to explaining a general approach of web scraping for statistics. In Sweden, four data sets of job advertisements from the state employment agency have been exploited, concerning module 2, 3 and 5.


The data sets are xml files; they are then cleaned and transformed into a database, illustrated in module 2 to 3 in the figure. Since the data are from one source, duplication removal is not considered at the first place. Based on the data in module 3, we studied variables e.g. occupation, organization number and enterprise´s sectors.

The data sets have good coverage of occupations and the sectors. Only a small percentage of companies are covered comparing with the business register. However we cannot by the data itself conclude how good the coverage is. Data of other sources need to be complemented.

In the virtual sprint, the interesting questions can be for example:

  1. How to map from the organization numbers in the adverts to the legal units in statistics of job vacancy, so that we can compare the advertisements data with the job vacancy statistics?
  2. How to find the duplicate adverts in one source/multiple sources, can we combine the usage of the structured data and the text analysis?

Duplication detection methods

Below are a series of resources considering duplication detection in Big Data. There are not many resources presented here, as this is just a fews hours worth of research. This is intended to help others get started. I am sure there are many more resources and ideas that I am missing…


This resource is more theoretical. Chapter 3 considers duplication detection in Big Data. Chapter 3 is available in slide and document form.

The approach suggested is a hashing approach, followed by Jaccard Similarity. Basically, it blocks similar items and then compares only the items within the same hashed block using Jaccard Similarity. This is to avoid having to compare all items to each other.


Python library


I have found a Python Module (Library) which covers duplicated detection. I have not tested this Module. The Module is a similar approach to that outlined above, that is of hashing and Jaccard Similarity detection.


More advanced tools:

Below is a brief exploration into the use of distributed technologies (Spark) in record matching. I don’t think we need Spark for the Sprint. However, it is useful to research: As it would be needed for large scale problems. Thus, we may need such tools later in the project.


Using Spark to merge Brazilian Health Databases 


This is an example from Brazil: concerning the joining together of several large health databases. The databases do not contain shared unique identifiers. Hence, they need to use Similarity matching, at scale. 


Hashing, Blocking and Similarity matching using Spark

They use the Spark(scala) environment. They standardise the features (data wrangling). The features are hashed (bigrams to bit vectors [10110000110101]), blocked (by region) and similarity matched using Sorensen.


Similarity measure

Sorensen ( D(a,b) = 2h / ( |a| + |b| ) ): which compares the position of 1’s in both vectors a and b. The common positions are counted in h, then divided by the total number of 1’s in a + b. (where 1 is a complete match).


Machine Learning Methods

Machine learning can also be used for duplication detection. The method suggested below is the use of supervised machine learning (for record linkage).


Record matching with Spark ML (video)



Slides (for the video above)