Model Card for NatureLM-audio

NatureLM-audio is the first audio-language foundation model specifically designed for bioacoustics. It is trained on a diverse dataset of text-audio pairs spanning bioacoustics, speech, and music, enabling it to perform tasks such as species classification, detection, captioning, and lifestage classification. The model demonstrates strong generalization to unseen taxa and tasks, setting a new state-of-the-art on several bioacoustics benchmarks.

Model Details

Model Description

NatureLM-audio is an audio-language model designed to address bioacoustic tasks such as species classification, detection, and captioning. It leverages a combination of bioacoustic, speech, and music data to learn robust representations that generalize across domains.

Developed by: David Robinson, Marius Miron, Masato Hagiwara, Milad Alizadeh, Gagan Narula, Sara Keen, Benno Weck, Matthieu Geist, Olivier Pietquin (Earth Species Project)
Funded by: More info at https://www.earthspecies.org/about-us#support
Shared by: Earth Species Project
Model type: Audio-language foundation model
Language(s) (NLP): English
License: CC-BY-NC-SA
Finetuned from model: Llama-3.1-8B-Instruct, Fine-tuned BEATs_iter3+ (AS2M) (cpt2)

Model Sources

Repository: https://github.com/earthspecies/naturelm-audio
Paper: NatureLM-audio: An Audio-Language Foundation Model for Bioacoustics
Demo: https://earthspecies.github.io/naturelm-audio-demo/

Uses

Direct Use

NatureLM-audio can be used directly for bioacoustic tasks such as species classification, detection, and captioning. It is particularly useful for biodiversity monitoring, conservation, and animal behavior studies.

Example prompts:

Prompt: What is the common name for the focal species in the audio?
Answer: Humpback Whale

Prompt: Which of these, if any, are present in the audio recording? Single pulse gibbon call, Multiple pulse gibbon call, Gibbon duet, None.
Answer: Gibbon duet

Prompt: What is the common name for the focal species in the audio?
Answer: Spectacled Tetraka

Prompt: What is the life stage of the focal species in the audio?
Answer: Juvenile

Prompt: What type of vocalization is heard from the focal species in the audio?
Answer with either 'call' or 'song'.

Prompt: Caption the audio, using the common name for any animal species.

Downstream Use

The model can be used to structure audio for ethology research, be integrated into larger ecological monitoring systems, or be fine-tuned for specific bioacoustic tasks.

Out-of-Scope Use

The model is not designed for tasks outside of bioacoustics. It was not tested for tasks such as individual-id, and call-type and lifestage classification tasks have only been tested on birds Tasks beyond those evaluated in the paper may require in-context learning or fine-tuning. The model does not currently perform fine-grained detection with exact time stamps.

Bias, Risks, and Limitations

Bias: The model may exhibit biases towards bird vocalizations due to the overrepresentation of bird datasets in the training data. This could limit its effectiveness for other taxa. Further, the model may inherit biases from the parent Llama model.
Risks: The model’s ability to detect and classify endangered species could be misused for illegal activities such as poaching.
Limitations: The model’s performance may be limited for under-represented taxa.
Red-teaming results: We ran a red-teaming assessment by first defining 16 risk categories adapted for AI safety in the context of animals, ecosystems, and the environment, such as Wildlife Exploitation, Non-Compliance with Environmental Laws, and Biodiversity Loss. Then, we used an LLM to generate adversarial prompts that could potentially lead to harmful output, and the responses were then evaluated to determine their safety. While the majority of responses from NatureLM-audio were safe, often providing no content for problematic prompts, we identified several scenarios where the model's responses were potentially harmful, including cases where the model failed to discourage unethical actions related to wildlife exploitation and environmental harm.

Recommendations

Users should be aware of the risks, biases, and limitations of the model. It is recommended to use the model in conjunction with other ecological monitoring tools and to validate its predictions in real-world settings.

How to Get Started with the Model

Instantiating the model:

from NatureLM.models import NatureLM

# Download the model from HuggingFace
model = NatureLM.from_pretrained("EarthSpeciesProject/NatureLM-audio")
model = model.eval().to("cuda")

Using the model:

from NatureLM.infer import Pipeline

audio_paths = ["assets/nri-GreenTreeFrogEvergladesNP.mp3"]  
queries = ["What is the common name for the focal species in the audio? Answer:"]

pipeline = Pipeline(model=model)

# Run the model over the audio in sliding windows of 10 seconds with a hop length of 10 seconds
results = pipeline(audio_paths, queries, window_length_seconds=10.0, hop_length_seconds=10.0)

print(results)
# ['#0.00s - 10.00s#: Green Treefrog\n']

Refer to the GitHub repository for more details.

Training Details

Training Data

The model is trained on a diverse dataset of text-audio pairs, including bioacoustic recordings, general audio, speech, and music datasets. The training data includes datasets such as Xeno-canto, iNaturalist, and Watkins. We release the training dataset on huggingface.

Training Procedure

The model is trained in two stages:

Perception Pretraining on species classification.
Generalization Fine-tuning on a variety of bioacoustic tasks.

Training Hyperparameters

Learning rate: 9.0e-5 (peak), 2.0e-5 (end)
Batch size: 128
Training steps: 5.0e5 (Stage 1), 1.6e6 (Stage 2)

For the full list of hyperparameters consult the NatureLM-audio repository.

Evaluation

Testing Data, Factors & Metrics

Testing Data

The model is evaluated on the BEANS-Zero benchmark, which includes tasks such as species classification, detection, and captioning.

Metrics

Accuracy for classification
F1 for detection
SPIDEr for captioning

Results

The model achieves state-of-the-art performance on several bioacoustics tasks, including zero-shot classification of unseen species. <INCLUDE TABLE?>

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

Hardware Type: 8xH100
Hours used: 216
Cloud Provider: Lambda labs
Compute Region: central-texas

Technical Specifications

Model Architecture and Objective

The model uses a BEATs audio encoder, Q-Former for connecting audio embeddings to the LLM, and Llama-3.1-8B-Instruct as the text generator.

Compute Infrastructure

Hardware: 8xH100
Software: Pytorch

Citation

BibTeX:

@inproceedings{naturelm-audio,
    title={NatureLM-audio: An Audio-Language Foundation Model for Bioacoustics},
    author={Robinson, David and Miron, Marius and Hagiwara, Masato and Pietquin, Olivier},
     booktitle={Proceedings of the International Conference on Learning Representations},
    year={2025}
}

APA:

Robinson, D., Miron, M., Hagiwara, M., & Pietquin, O. (2025). NatureLM-audio: An Audio-Language Foundation Model for Bioacoustics. ICLR 2025

Glossary

Bioacoustics: The study of sound production and reception in animals.
Zero-shot learning: The ability of a model to perform tasks it has not explicitly been trained on.
Taxa: A group of organisms, such as species, genus, or family.

More Information

For more information, please visit the project page.

Model Card Authors

David Robinson (Earth Species Project)
Marius Miron (Earth Species Project)
Masato Hagiwara (Earth Species Project)
Milad Alizadeh (Earth Species Project)
Gagan Narula (Earth Species Project)
Sara Keen (Earth Species Project)
Benno Weck (Earth Species Project)
Matthieu Geist (Earth Species Project)
Olivier Pietquin (Earth Species Project)

Model Card Contact

Contact: [email protected]

EarthSpeciesProject
/

NatureLM-audio