trihoang131/distilroberta-movies-embeddings

SentenceTransformer based on sentence-transformers/all-distilroberta-v1

This is a sentence-transformers model finetuned from sentence-transformers/all-distilroberta-v1 on the movie_dataset_50_k dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-distilroberta-v1
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • movie_dataset_50_k

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: RobertaModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("trihoang131/distilroberta-movies-embeddings")
# Run inference
sentences = [
    'Milk the Maid stars newcomer Tia, who is cute, sexy and hot (not necessarily in this order). This erotic comedy tells the story of Milk, a Japanese sexy maid (Tia) who begins living with a Tokyo family. Initially turned off by the prospect of having another mouth to feed, the men of the family are reluctant to accepts Milk, but her irresistible charm kicks in, and she quickly finds her way into their hearts and bedrooms.',
    'Shankar is a wounded, reluctant and reclusive aghora on a quest to find the cure for his very rare human condition – the one that will be cured, only when he can confront and conquers the haunting ironical questions of his life.',
    'Two rogue vagrants make their living as "manure men", turning the waste from the tenement toilets into fertiliser sold to local farmers. Enter Okiku, the only daughter of a fallen samurai, and amongst the overflowing piles of excrement, a well-nourished love story unfolds.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

• Datasets: ai-movie-validation, ai-movie-test and ai-movie-test
• Evaluated with TripletEvaluator

Metric	ai-movie-validation	ai-movie-test
cosine_accuracy	0.9984	0.9988

Training Details

Training Dataset

movie_dataset_50_k

• Dataset: movie_dataset_50_k at 060877e
• Size: 20,000 training samples
• Columns: overview, negative_overview, and positive_overview
• Approximate statistics based on the first 1000 samples:

  	overview	negative_overview	positive_overview
  type	string	string	string
  details	min: 23 tokens mean: 74.06 tokens max: 317 tokens	min: 24 tokens mean: 72.85 tokens max: 361 tokens	min: 23 tokens mean: 79.02 tokens max: 318 tokens

• Samples:

  overview	negative_overview	positive_overview
  A caboclo's soul wanders through purgatory (or hell), visiting many places until he boards a ship whose destination is unknown.	Documentary on the beginnings of Algerian independence filmed during the summer of 1962 in Algiers. The film was banned in France and Algeria but won the Grand Prize at the Leipzig International Film Festival in 1965. Out of friendship, the production company Images de France sent an operator, Bruno Muel, who later declared: "For those who were called to Algeria (for me, 1956-58), participating in a film on independence was a victory over horror, lies and absurdity. It was also the beginning of my commitment to the cinema."	The friends Curió, Boroca (Dedé Santana), Mexelete and Bateia they venture in search of gold in the mine of Serra Pelada. The area is controlled by the foreigner Von Bermann, whose orders are executed by the bully Bira. Thirsty of being able to, the foreigner smuggles the gold and he wants to take possession of the lands of the Brazilian Ribamar, that refuses to do business before the son's arrival Chicão. With the help of the four dabblers, the boy struggles with the thieves and it helps the father in danger.
  In the feature documentary, Summer 82 - When Zappa Came to Sicily, filmmaker and Zappa fan Salvo Cuccia tells the behind-the-scenes story of Frank Zappa's star-crossed concert in Palermo, Sicily, the wrap-up to a European tour that ended in public disturbances and police intervention. Cuccia had a ticket to the concert but never made it. Thirty years later, collaborating with Zappa's family, he re-creates the events through a combination of rare concert and backstage footage; photographs; anecdotes from family, band members, and concertgoers; and insights from Zappa biographer and friend Massimo Bassoli. The story is also a personal one, as Cuccia interweaves the story of Zappa's trip to Sicily with his own memories from that summer.	Ten strangers are trapped in a deadly experiment by a twisted soon-to-be serial killer. "You have 60 seconds to kill the person in front of you."	On October 20th, 1959, producer Giuseppe Amato is alone in a screening room, watching Federico Fellini's most famous movie. The working print is more than four-hour long. Fellini would not allow any cut, and distributor Angelo Rizzoli wants to drop the movie. It is the hardest moment in Giuseppe Amato's long career.
  After wealthy college boy Ömer and lower class college girl Aysem fall in love and quickly marry, they learn that love and life aren't always perfect.	Since the End of World War II, the People of Tanna, a remote island in the south Pacific in the archipelago of Vanuatu, idolize an American prophet. His name is John Frum. The islanders believe he is an American pilot that returned to the United States after the war, and will come back to Tanna with riches from the US that they call "the cargo". They pray to an American flag, awaiting his return. One man, Isaac the Last One, chief of the "Cargo Cult", claims he is Frum's Son. He has formed an army of GI's to celebrate the return of the prophet John Frum.	Sanem is a working girl sharing a flat with three transvestites in Istanbul. Every day she dreams of a savior who will take her away. One day a young man named Gokhan moves into the neighborhood, and soon Sanem gets his attention. This will be the beginning of a journey during which both will question each other and their choices in life.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

movie_dataset_50_k

• Dataset: movie_dataset_50_k at 060877e
• Size: 2,500 evaluation samples
• Columns: overview, negative_overview, and positive_overview
• Approximate statistics based on the first 1000 samples:

  	overview	negative_overview	positive_overview
  type	string	string	string
  details	min: 23 tokens mean: 75.28 tokens max: 311 tokens	min: 23 tokens mean: 72.03 tokens max: 311 tokens	min: 23 tokens mean: 77.78 tokens max: 284 tokens

• Samples:

  overview	negative_overview	positive_overview
  A medieval nun's convent is the only refuge for women trying to escape from Turkish raiders looking for slaves. But the beautiful mother superior is actually in league with the raiders and her convent is no refuge but a secret market for the beauties who also succumb to her wild desires.	From the award-winning team that brought you The Secret Life of Chaos comes a unique television event on the physics of gravity.	Deloris Van Cartier is again asked to don the nun's habit to help a run-down Catholic school, presided over by Mother Superior. And if trying to reach out to a class full of uninterested students wasn't bad enough, the sisters discover that the school is due to be closed by the unscrupulous chief of a local authority.
  Dastak (English: Knock) is a 1996 Hindi language Indian feature film directed by Mahesh Bhatt, starring Sushmita Sen in her debut film supported by Mukul Dev with Sharad Kapoor as the psychopath.	Nelson is a miniature pig—at least, that’s what the Verbeek family were told when they bought him as a cute little piglet. But seven years and 500 kilos later, Nelson has become a big problem. The neighbors and the housing association think it’s unacceptable to have such a large and smelly animal in a residential neighborhood. Son Brandon is being bullied at school because of his pet.	Dekh Bhai Dekh (Hindi: देख भाई देख) is a 2009 Bollywood comedy film directed by Rahat Kazmi featuring Gracy Singh and Siddharth Koirala in the lead roles
  Originally, AlchemyII Inc. had hoped to create a live-action series using animatronic characters, as Ken Forsse had helped Disney do with Welcome to Pooh Corner and Dumbo's Circus. However, due to production costs and difficulties in this format, Forsse, AlchemyII and Worlds of Wonder decided animation would be a better route and the 65 episode animated series was created. The pilot episode of what would have been the animatronic series was instead released as a stand-alone ABC Movie of the week in 1986 and also aired in syndication as a 2-part episode.The show can be found on videocassette. The "animatronic movie", as it's called by Teddy Ruxpin fans, used primarily the same voice talent as the Teddy Ruxpin toy software had, most of which (with the exception of Phil Baron and Will Ryan) were replaced in the later animated TV series by Canadian voice talent.	Eight miles inland of Miami’s beaches, Liberty City residents fight to save their community from climate gentrification: their land, sitting on a ridge, becomes real estate gold.	Disney's animated adaptation of Prokofiev's masterpiece, in which every character is represented musically by a different instrument. Young Peter decides to go hunting for the wolf that's been prowling around the village. Along the way, he is joined by his friends the bird, the duck and the cat. All the fun comes to end, however, when the wolf makes an appearance. Will Peter and his friends live to tell of their adventures?

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss	ai-movie-validation_cosine_accuracy	ai-movie-test_cosine_accuracy
0.16	100	5.2516	4.1563	0.9984	-
0.32	200	4.1595	4.1559	0.9984	-
0.48	300	4.1594	4.1560	0.9984	-
0.64	400	4.1593	4.1559	0.9984	-
0.8	500	4.1594	4.1560	0.9984	-
0.96	600	4.1591	4.1560	0.9984	-
-1	-1	-	-	0.9984	0.9988

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.4.1
• Transformers: 4.48.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.3.0
• Datasets: 3.4.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}