README.md

---

base_model:
- huawei-noah/TinyBERT_General_4L_312D
datasets:
- QuotaClimat/frugalaichallenge-text-train
language:
- en
library_name: transformers
license: apache-2.0
metrics:
- accuracy
- f1
---


# Model Card: climate-skepticism-classifier

## Model Overview
This model implements a novel approach to classifying climate change skepticism arguments 
    by utilizing Large Language Models (LLMs) for data rebalancing. The base architecture uses BERT with 

    custom modifications for handling imbalanced datasets across 8 distinct categories of climate skepticism. 

    The model achieves exceptional performance with an accuracy of 99.92%.


    The model categorizes text into the following skepticism types:

    - Fossil fuel necessity arguments

    - Non-relevance claims

    - Climate change denial

    - Anthropogenic cause denial

    - Impact minimization

    - Bias allegations

    - Scientific reliability questions

    - Solution opposition

    

    The unique feature of this model is its use of LLM-based data rebalancing to address the inherent class 

    imbalance in climate skepticism detection, ensuring robust performance across all argument categories.


## Dataset
- **Source**: Frugal AI Challenge Text Task Dataset
- **Classes**: 7 unique labels representing various categories of text
- **Preprocessing**: Tokenization using `BertTokenizer` with padding and truncation to a maximum sequence length of 128.

## Model Architecture
- **Base Model**: `huawei-noah/TinyBERT_General_4L_312D`
- **Classification Head**: cross-entropy loss.
- **Number of Labels**: 7

## Training Details
- **Optimizer**: AdamW
- **Learning Rate**: 2e-5
- **Batch Size**: 16 (for both training and evaluation)
- **Epochs**: 3
- **Weight Decay**: 0.01
- **Evaluation Strategy**: Performed at the end of each epoch
- **Hardware**: Trained on GPUs for efficient computation

## Performance Metrics (Validation Set)
The following metrics were computed on the validation set (not the test set, which remains private for the competition):

| Class | Precision | Recall | F1-Score | Support |
|-------|-----------|--------|----------|---------|
| not_relevant | 0.88 | 0.82 | 0.85 | 130.0 |

| not_happening | 0.82 | 0.93 | 0.87 | 59.0 |
| not_human | 0.80 | 0.86 | 0.83 | 56.0 |

| not_bad | 0.87 | 0.84 | 0.85 | 31.0 |
| fossil_fuels_needed | 0.87 | 0.84 | 0.85 | 62.0 |
| science_unreliable | 0.78 | 0.77 | 0.77 | 64.0 |

| proponents_biased | 0.73 | 0.75 | 0.74 | 63.0 |


- **Overall Accuracy**: 0.83
- **Macro Average**: Precision: 0.82, Recall: 0.83, F1-Score: 0.83
- **Weighted Average**: Precision: 0.83, Recall: 0.83, F1-Score: 0.83


## Training Evolution
### Training and Validation Loss
The training and validation loss evolution over epochs is shown below:

![Training Loss](./training_loss_plot.png)

### Validation Accuracy
The validation accuracy evolution over epochs is shown below:

![Validation Accuracy](./validation_accuracy_plot.png)

## Confusion Matrix
The confusion matrix below illustrates the model's performance on the validation set, highlighting areas of strength and potential misclassifications:

![Confusion Matrix](./confusion_matrix.png)

## Key Features
- **Class Weighting**: Addressed dataset imbalance by incorporating class weights during training.
- **Custom Loss Function**: Used weighted cross-entropy loss for better handling of underrepresented classes.
- **Evaluation Metrics**: Accuracy, precision, recall, and F1-score were computed to provide a comprehensive understanding of the model's performance.

## Class Mapping
The mapping between model output indices and class names is as follows:
0: not_relevant, 1: not_happening, 2: not_human, 3: not_bad, 4: fossil_fuels_needed, 5: science_unreliable, 6: proponents_biased

## Usage
This model can be used for multi-class text classification tasks where the input text needs to be categorized into one of the eight predefined classes. It is particularly suited for datasets with class imbalance, thanks to its weighted loss function.

### Example Usage
```python

from transformers import AutoModelForSequenceClassification, AutoTokenizer



# Load the fine-tuned model and tokenizer

model = AutoModelForSequenceClassification.from_pretrained("climate-skepticism-classifier")

tokenizer = AutoTokenizer.from_pretrained("climate-skepticism-classifier")



# Tokenize input text

text = "Your input text here"

inputs = tokenizer(text, return_tensors="pt", padding="max_length", truncation=True, max_length=128)



# Perform inference

outputs = model(**inputs)

predicted_class = outputs.logits.argmax(-1).item()



print(f"Predicted Class: {predicted_class}")

```

## Limitations
- Performance may vary on extremely imbalanced datasets
    - Requires significant computational resources for training
    - Model performance is dependent on the quality of LLM-generated balanced data
    - May not perform optimally on very long text sequences (>128 tokens)
    - May struggle with novel or evolving climate skepticism arguments
    - Could be sensitive to subtle variations in argument framing
    - May require periodic updates to capture emerging skepticism patterns

## Citation
If you use this model, please cite:
    @article{your_name2024climateskepticism,

        title={LLM-Rebalanced Transformer for Climate Change Skepticism Classification},

        author={Your Name},

        year={2024},

        journal={Preprint}

    }


## Acknowledgments
Special thanks to the Frugal AI Challenge organizers for providing the dataset and fostering innovation in AI research.