I'm in a very dire situation where I have to preprocess the text but the text in the documents is very random. It is in the form of numerical points.

I want to remove a certain class of points (like bullet points) from those documents but simple preprocessing is not helping as the points are appearing randomly anywhere in the document.

I was thinking of learning a model where the model identifies the points which I want to remove.

Is it viable? And If so what approaches should I look for.

[Natural language processing] Thanks!

  • $\begingroup$ Your question is not clear, please add an example to show what the data looks like and what you're trying to achieve: what do you mean with "random text"? what is a "numerical point" in a text? What is a class of numerical points? $\endgroup$ – Erwan Oct 16 at 11:22
  • $\begingroup$ Data is in the form of text a.k.a PDF files. Each PDF file contains text which is in numerical points (like bullet points) Now I want to remove certain no. of points but the location of those points is random in each pdf file. Therefore I wanted to learn a model to identify those points and then later remove them. I can't do this task manually as there are 1000 pdf files and each pdf contains 50-100 pages (20-100 points). $\endgroup$ – nnd Oct 16 at 14:02
  • $\begingroup$ class refer to certain type of class which we wil learn those points belong to, later we will remove those points! $\endgroup$ – nnd Oct 16 at 14:06
  • $\begingroup$ You mean that your text documents contains only long enumerations of items like "1. ..... 2. ..... 3. ....."? Do you want to remove particular points based on their number/order or based on the text content of the item? Without any detail about how you decide which points to keep or remove it's very difficult to answer the question $\endgroup$ – Erwan Oct 16 at 14:31
  • $\begingroup$ Yes, I just want to remove those based on their text. That's why I'm talking about learning a model. Also, the position of those points and text context varies very much from document to document. I just want to simply identify them and remove from my document. It's basically preprocessing. Could I label certain points (Supervised ML task) and then train a model to identify and later remove them. $\endgroup$ – nnd Oct 16 at 17:29

Based on the details given in the comments, it looks like what you need is a binary classifier which predicts whether or not to remove a "numerical point". This means that each instance is an individual "numerical point", and you would need to annotate a sample of instances to use as a training set.

A simple option is to use a bag of words representation, i.e. represent every instance as a vector over the vocabulary where each cell contains the frequency of the corresponding word (or other variants, e.g. binary or TF-IDF weights). Then many classification algorithms can be used: decision trees, SVM, Naive Bayes, etc.

  • $\begingroup$ Thanks for the answer. I'm gonna try it soon. Also, Do you know any efficient technique to reduce feature matrix size as each pdf file contains 50-100 pages? $\endgroup$ – nnd Oct 17 at 2:10
  • $\begingroup$ The simplest thing to do is to discard the words which appear less than N times in the whole corpus. You could also lemmatize the text and use lemmas instead of words. Removing stop words might help but probably not much. Beyond that there are dimensionality reduction techniques, either based on individual features or by clustering like PCA etc. But it's a bit more work to get it right in this case. $\endgroup$ – Erwan Oct 17 at 10:57
