metadata

language: en
license: apache-2.0
library_name: transformers
tags:
  - distilbert
  - text-classification
  - token-classification
  - intent-classification
  - slot-filling
  - joint-intent-slot
  - smart-home
  - generated:Enfuse.io
pipeline_tag: token-classification
model-index:
  - name: distilbert-joint-intent-slot-smarthome
    results:
      - task:
          type: token-classification
          name: Slot Filling
        dataset:
          name: enfuse/joint-intent-slot-smarthome
          type: enfuse/joint-intent-slot-smarthome
          config: default
          split: test
        metrics:
          - type: micro_f1
            value: 0.88
            name: Slot F1 (Micro)
          - type: precision
            value: 0.88
            name: Slot Precision (Micro)
          - type: recall
            value: 0.88
            name: Slot Recall (Micro)
      - task:
          type: text-classification
          name: Intent Classification
        dataset:
          name: enfuse/joint-intent-slot-smarthome
          type: enfuse/joint-intent-slot-smarthome
          config: default
          split: test
        metrics:
          - type: accuracy
            value: 0.9208
            name: Intent Accuracy

DistilBERT for Smart Home Joint Intent Classification and Slot Filling

Model Description

Produced By: Enfuse.io

This model is a fine-tuned version of distilbert-base-uncased specifically adapted for joint intent classification and slot filling in the smart home domain. Given a user command related to controlling smart home devices (like lights or thermostats), the model simultaneously predicts:

The user's intent (e.g., set_device_state, get_device_state).
The relevant slots (entities like device_name, location, state, attribute_value) within the command, using BIO tagging.

Intended Use and Limitations

Primary Intended Use: This model is intended for hobbyist experimentation and educational purposes related to Natural Language Understanding (NLU) for smart home applications. It can be used as a baseline or starting point for understanding how to build NLU components for simple device control.

Disclaimer: This model is NOT intended for use in production environments. Enfuse.io takes no responsibility for the performance, reliability, security, or any consequences arising from the use of this model in production systems or safety-critical applications. Use in such contexts is entirely at the user's own risk.

Out-of-Scope Use:

The model is not designed for general conversation or tasks outside the specific smart home intents and slots it was trained on.
It has no built-in mechanism for handling out-of-domain requests (e.g., asking about weather, playing music). It will likely attempt to classify such requests into one of the known smart home intents, potentially leading to incorrect behavior.
It has not been evaluated for fairness, bias, or robustness against adversarial inputs.

Training Data

The model was fine-tuned on the enfuse/joint-intent-slot-smarthome dataset, specifically the generated_smarthome_2016_unique.jsonl version containing 2016 unique synthetic examples.

This dataset was generated by Enfuse.io using a combination of mistralai/Mistral-7B-Instruct-v0.1 and openai/gpt-4o, followed by validation and de-duplication. Please refer to the dataset card for more details on the data generation process and limitations.

Training Procedure

Preprocessing

The text was tokenized using the distilbert-base-uncased tokenizer. Slot labels were converted to a BIO tagging scheme. Input sequences were padded and truncated to a maximum length of 128 tokens.

Fine-tuning

The model was fine-tuned using the Hugging Face transformers library Trainer on a single NVIDIA RTX 5090.

Epochs: 10
Batch Size: 16 (per device)
Learning Rate: 5e-5 (with linear decay)
Optimizer: AdamW
Precision: FP16
Dataset Split: 80% Train (1612), 10% Validation (202), 10% Test (202)
Best Model Selection: The checkpoint with the highest eval_intent_accuracy on the validation set during training was selected for the final model (corresponding to Epoch 8 or 10 in the 10-epoch run).

Evaluation Results

The following results were achieved on the test set (202 examples) using the best checkpoint saved during training:

Intent Accuracy: 92.08%
Slot F1 Score (Micro): 88.00%
Slot Precision (Micro): 88.00%
Slot Recall (Micro): 88.00%

(Note: These results are specific to this particular training setup and may vary with different hyperparameters or training runs.)

How to Use

(You would typically add code examples here showing how to load and use the model with the Transformers pipeline or custom code, similar to the logic in infer.py. Since the user requested no code, this section is omitted but would normally be present.)

Model Card Contact

Enfuse.io

Citation

If you use this model, please cite the dataset:

@misc{enfuse_smarthome_intent_slot_2024,
  author = {Enfuse.io},
  title = {Enfuse Smart Home Joint Intent and Slot Filling Dataset},
  year = {2024},
  publisher = {Hugging Face},
  journal = {Hugging Face Hub},
  howpublished = {\url{https://huggingface.co/datasets/enfuse/joint-intent-slot-smarthome}}
}