Teaching resources

 

These are mostly in French.

Software resources

 

soft_cofact is a set of Matlab scripts, developped by N. Seichepine, that compute Soft Nonnegative Matrix Co-Factorisation with IS or KL divergence and l1 or l2 coupling. Check the related publications.

sv_nmf is a Matlab script that computes Nonnegative Matrix Factorisation (NMF) using single-class Support Vector Machines (SVM). Hopefully there will soon be a Python version. Check the related publications.

Yaafe is "yet another audio feature extractor" developed by Benoit MATHIEU at Telecom ParisTech. It is a program designed for the efficient computation of many audio features simultaneously. Yaafe automatically organizes the computation flow so that the intermediate representations (FFT, CQT, envelope, etc.), on the basis of which most audio features are composed, are computed only once. Further, the computations are performed block per block, so yaafe can analyze arbitrarily long audio files.

TPYaafeExtension is an extension of the Yaafe toolbox, developed at Telecom ParisTech by Benoit MATHIEU. It provides several useful mid-level audio features (CQT, Chromas, Chords, ...).

More software resources by the AAO team here.

Research datasets

 

The 3DLife/Huawei ACM MM GC 2011 dataset consists of multimodal recordings of Salsa dancers, captured at different sites, in particular at our local studio, with different pieces of equipment.
Read more »

More datasets by the AAO team here.

Research demos

 

Here is a selection of demos related to recent work I have contributed to. Credits go to the Masters and PhD students who made most of the following and are hereafter namely mentioned, as well as colleagues who took part in this research.

Dance movement analysis using Gaussian processes

This work focuses on the decomposition of dance movements into elementary motions. Placing this problem into a probabilistic framework, Gaussian processes are exploited to accurately model the different components of the decomposition. Video illustrations of the proposed method can be seen here.

Music-to-score alignment

Audio examples by Cyril Joder, a previous PhD student of mine, can be found here. Cyril worked on music-to-score alignment with graphical models. One of the main contributions of Cyril is the introduction of the Conditional Random Fields (CRF) framework for this task. Check the related publications.

Audio-driven dance performance analysis

This work addressed the Huawei/3Dlife Grand challenge by proposing a set of audio tools for a virtual dance-teaching assistant. These tools are meant to help dance students develop a sense of rhythm to correctly synchronize their movements and steps to the musical timing of the choreographies to be executed. They consist of three main components, namely a music (beat) analysis module, a source separation and remastering module and a dance step segmentation module. These components enable to create augmented tutorial videos highlighting the rhythmic information using, for instance, a synthetic dance teacher voice, but also videos highlighting the steps executed by a student to help in the evaluation of his/her performance. Examples of such videos, prepared by Robin Tournemenne, are given hereafter. Check the related publications.

This is an example of dance teacher videos augmented with audio effects highlighting the musical timing information.

Original video for the 5 th choregraphy

Synthetic teacher voice Synthetic hand clapping

 

 

This is an example of videos of a student dancer augmented with audio effects highlighting the automatically detected steps.

Original video Step evaluation with beeps sounding on detected steps only

 

Step verification with a mix of onfloor piezo sounds and
beeps on steps detected
Musical evaluation to control if steps are consistent
with the musical timing
 

Enhanced Visualisation of Dance Performances from Automatically Synchronised Multimodal Recordings

The Huawei/3DLife Grand Challenge Dataset provides multimodal recordings of Salsa dancing, consisting of audiovisual streams along with depth maps and inertial measurements. In this work, we proposed a system for augmented reality-based evaluations of Salsa dance performances. The following videos, prepared together with Jean Lefeuvre, illustrate the functionalities of the software application that was developped in this work. Check the related publications.

Camera Layout view

Viewpoint and audio stream selection

Audiovisual augmentations illustrating automatic step analysis

Automatic alignment of two dancers