Take the 2-minute tour ×  Basically, I am designing a web search
engine, so I designed a crawler to get web pages.  When read in,
the web pages are in html format, so all the tags are there. I
need to extract keywords from the body and title, so I'm trying
to remove all the tags (anything between '<' and '>')  The code
below works well for small html pages, but when I try to use
this on a large scale (ie starting from http://www.google.com),
I run out of memory.  0 def remove_tags(self, s): 1     while
'<' in s: 2         start = s.index('<') 3         end =
s.index('>') 4         s = s[:start] + " " + s[end+1:] 5
return s.split()  The memory error occurs at line 4. How do I
fix my code so that taking the substrings of s doesn't consume
excessive memory?  share|improve this question Oh man that's
ugly. –  Joel Cornett Jul 29 '12 at 9:14 add comment  1 Answer
up vote 8 down vote accepted  Your general approach is wrong.
Firstly, use a real XML/HTML parser. Something like
BeautifulSoup, which is forgiving