This paper has contents which may be offensive, or upsetting however this cannot be avoided owing to the nature of the work. Hate speech and offensive texts are examples of damaging online content that target or promote hatred toward a group or individual member based on their actual or perceived features of identification, such as ethnicity, religion, or sexual orientation. Sharing violent and offensive content has had a significant negative impact on society. These hate speech and offensive content generally contains societal biases in them. With the rise of online hate speech, automatic detection of such biases as a natural language processing task is getting popular. However, not much research has been done to detect unintended social bias from these toxic language datasets. This report attempts to summarise what are existing hate speech detection and offensive text detection models are. Then it will reason why hate speech models struggle to generalise, which sums up existing attempts at addressing the main obstacles. Finally, this report introduces a new dataset from an existing toxic language dataset to detect social biases, their categories, and targeted groups in English. The dataset contains instances annotated for five different bias categories, viz., gender, race/ethnicity, religion, political, and LGBTQ . We then report baseline performances of both classification tasks on our curated dataset using transformer-based models. The input to the models is English texts which are probably hate speech or toxic texts. The models will then classify these texts into biased or neutral along with bias categories. Model biases and their miti-gation are also discussed in detail. Our study motivates a systematic extraction of social bias data from toxic languages.