Take the 2-minute tour ×

Certainly, there must be some way to somewhat create a list of words suggested to be included in an index.

share|improve this question
What do you mean by "suggested to be included in the index"? I would expect it to be the author of the content who should know what is suitable for the index. Or are machines already getting so smart? :) –  José Figueroa-O'Farrill Aug 3 '10 at 2:02
Index creation is regarded as a very difficult task, at the 'what to index' level. I believe that there are people who make their living by being good at it! So automation is probable not easy. –  Joseph Wright Aug 3 '10 at 7:18

7 Answers 7

up vote 9 down vote accepted

I suggest you look at the script make-index.py (and related files) in the scripts folder of the download page at the Stacks Project (http://www.math.columbia.edu/algebraic_geometry/stacks-git/). The index it generates isn't really ideal, but at least their strategy will give you some idea how to get started. They seem to take the approach that (in a gigantic math textbook) the things which most deserve to be in the index are the italicized word(s) or phrase(s) in each definition environment. In my experience using math books, the most common reason I look something up in the index is to learn its definition, so this seems appropriate, although maybe not for books in other subjects. However you might be able to use the Stacks Project script as a guide to automate the creation of an index which suits your own needs, even if they are very different.

share|improve this answer
Simply searching for something like \emph might already give you some ideas of index entries. –  Jukka Suomela Aug 3 '10 at 17:08

As others have mentioned, trying to automate this task would be close to impossible. But if you want to get some very rough hints of words for yourself, this is something I would try (note, requires some scripting):

Use detex or something to strip the TeX markup and then write a small script that counts the number of time each word has been used in the document. The top words in the list will probably be useless words like a, the, is, etc. But, after those, you might be able to find a few promising words.

share|improve this answer

In addition to what Juan A. Navarro suggested, I'd say that words which occur in chapter and section titles are likely candidates for indexing. E.g., if section 2.3 is entitled "The Virasoro Algebra", then that's probably a sufficiently important topic that other occurrences of "Virasoro algebra" should be indexed. You could write a script (in your favourite scripting language) to pull out the arguments to \section commands and the like, throw out the prepositions and articles and sort the remainder by frequency. How your script will know that the words Virasoro and algebra go together . . . well, either you call import skynet and live with the consequences, or you do some manual work with its output.

Other things to check could include words which are capitalized when not at the beginning of a sentence and words set in emphatic type.

share|improve this answer

I just wrote a quick python script to extract the most common words in some tex files. It uses detex to strip tex commands from the files, strips characters like ".", ",", ";", "?", "!" from the end of words, ignores words that contain # or =, ignores case and the 100 most common english words (copied from http://www.duboislc.org/EducationWatch/First100Words.html)


import subprocess, glob, operator

# Tweak output here:
charsToStripFromEnd = ".,;?!"
nonWordChars = "=#"
minOccurrence = 30
skipWords = 'the of and a to in is you that it he was for on are as with his they I at be this have from or one had by word but not what all were we when your can said there use an each which she do how their if will up other about out many then them these so some her would make like him into time has look two more write go see number no way could people my than first water been call who oil its now find long down day did get come made may part e.g i.e'.split()

output = subprocess.check_output( ['detex'] + glob.glob('*.tex') )
wordList = output.split()
words = {}

for w in wordList:
  w = w.rstrip(charsToStripFromEnd).lower()
  if len(w) <= 2: continue

  isARealWord = True
  for c in nonWordChars: 
    if c in w: 
      isARealWord = True
  if not isARealWord: continue
  if w in skipWords: continue

  if not w in words:
    words[w] = 1
    words[w] += 1

sorted = sorted(words.iteritems(), key=operator.itemgetter(1))

for item in sorted:
  print item[0], item[1]
  if item[1] < minOccurrence: break
share|improve this answer

As a rough hack I sometimes use the following lines to get all my definitions in bold+italic, and put them in the index.


share|improve this answer
The question seems to be about what should be indexed, not how. At least the accepted answer is about that, so I assume that is what the OP meant. –  Martin Scharrer Jul 1 '11 at 15:17
This gives an idea of what should be indexed by actually indexing it. ☺ –  Geremia Oct 10 '13 at 17:41

You could use the glossaries package to suggest terms and acronyms for inclusion in an automatically generated glossary. It won't pick out words for inclusion on its own though, that might require a rather advanced level of natural language processing to accomplish.

share|improve this answer

An alternative would be to use detex and pipe that into a frequency-analyzing script:

This will show a list of the word distribution (case-sensitive), with the most frequent words first:

detex input.tex | tr -d '[:punct:]' | tr -d '[:digit:]' | tr ' ' '\n' | sort | uniq -c | sort -rn | less

This will output words that are not in the English dictionary:

detex input.tex | tr -d '[:punct:]' | tr -d '[:digit:]' | tr ' ' '\n' | sort -u | while read i; do if [ -z "`grep -i "$i" /usr/share/dict/words`" ]; then echo "$i"; fi; done

It's pretty slow; there's got to be a faster way, though.

share|improve this answer

Your Answer


