Google created a 'bird-sound expert' neural network to separate birdsong in the wild for classification

Birds surround us everywhere, and by simply listening to them we can learn a great deal about the environment. Ecologists use birds to understand food systems and forest health. For example, if there are more woodpeckers in a forest, it means there is a lot of dead standing wood. Because birds communicate and mark territory with songs and calls, identifying them by ear is the most effective approach. In fact, experts can identify 10 times more birds by ear than by sight.

In recent years, autonomous recording units (ARUs) have made it easy to record thousands of hours of sound in forests, which can be used to better understand ecosystems and identify critical habitats. However, reviewing audio data manually takes a great deal of time, and birdsong specialists are rare. 

A machine learning (ML) approach can significantly reduce the number of expert assessments needed to understand a habitat. However, machine learning-based audio classification of bird species can be challenging for several reasons. First, birds often sing at the same time, especially during the 'dawn chorus', when many birds are most active. 

Furthermore, there are no clean recordings of individual birds to learn from: almost all available training data was recorded in noisy outdoor conditions, where wind, insects and other environmental sounds are often present. As a result, existing birdsong classification models struggle to identify quiet, distant and overlapping vocalizations. 

Some of the most common species often appear unlabeled in the background of training recordings for less common species, causing models to overlook common species. These difficult cases are very important for ecologists who want to identify endangered species using automated systems.

To address the general problem of training machine learning models to automatically separate audio recordings without access to examples of isolated sounds, Google researchers recently proposed a new unsupervised method called mixture invariant training (MixIT) in the paper 'Unsupervised Sound Separation Using Mixture Invariant Training'. 

Additionally, in the paper 'Improving Bird Classification with Unsupervised Sound Separation', they use MixIT training to separate birdsong and improve species classification. The scientists found that including separated sound in classification increases accuracy and classification quality across three independent soundscape datasets. 

Separating birdsong sound

MixIT learns to separate single-channel recordings into multiple distinct tracks and can be fully trained on noisy real-world recordings. To train the separation model, researchers create a 'mixture of mixtures' (MoM) by mixing two real-world recordings together. The separation model then learns to decompose the MoM into multiple channels, minimizing a loss function that uses the two original real-world recordings as ground-truth references. 

The loss function uses these references to group the separated channels so that they can be mixed together to recreate the two original real-world recordings. Because it is impossible to know how the different sounds in the MoM were grouped together in the original recordings, the separation model has no choice but to split out individual sounds, and thus learns to place each singing bird on a different output audio channel, and to separate it from wind and other background noise.

The researchers trained the new MixIT separation model using birdsong recordings from Xeno-Canto and the Macaulay Library. They found that for separating birdsong, this new model outperformed a MixIT separation model trained on a large amount of general sound from the AudioSet dataset. 

The scientists measure quality by mixing two recordings together, applying separation, and then remixing the separated audio channels so that they reconstruct the two original recordings. The signal-to-noise ratio (SNR) of the remixed sound is measured against the original recordings. 

The researchers also found many examples where the system worked incredibly well, separating very hard-to-distinguish calls in real data. 

Classifying bird species

To classify birds in real sound recorded with ARUs, the scientists first split the audio into five-second segments and then created a mel spectrogram of each segment. They then train an EfficientNet classifier to identify bird species from mel spectrogram images of audio from Xeno-Canto and the Macaulay Library. 

The researchers introduced several new methods to improve classifier training. Taxonomic training asks the classifier to provide labels for each level of the species taxonomy (genus, family and order), which allows the model to learn groups of species before learning sometimes subtle differences between similar species. 

Taxonomic training also allows the model to benefit from expert knowledge about taxonomic relationships between different species. They found that random low-pass filtering is useful for simulating distant sounds during training: as the sound source moves farther away, high-frequency parts fade before low-frequency parts. This was especially effective for identifying species from the High Sierra region, where birdsong travels very long distances without interference from trees.

Classifying separated audio

Engineers found that separating sound with the new MixIT model before classification improved classifier performance on three independent real-world datasets. Separation was especially successful at identifying quiet and background birds, and in many cases helped with overlapping vocalizations.

The separation model has some potential limitations. Sometimes over-separation occurs, where a single song is split across multiple channels, which can lead to misclassification. Also, when several birds sing, the most prominent song often receives a lower score after separation. 

This may be due to loss of environmental context or other effects of separation that do not appear during classifier training. At present, scientists get the best results by running the classifier on both the separated channels and the original sound and taking the maximum score for each species. Further work will aim to reduce over-separation and find better ways to combine separation and classification. 

Future directions

Google engineers are currently working with partners from the California Academy of Sciences to understand how habitats and species composition change after forest fires, by applying these models to ARU sounds collected over many years.

"We also foresee many potential applications of unsupervised separation models in ecology beyond birds. For example, separated sound could be used to create better acoustic indices, which can measure ecosystem health by tracking overall activity of birds, insects and amphibians without identifying specific species," the researchers said.

Similar methods could also be adapted for underwater use to monitor the condition of coral reefs.