Did I use pickle in a wrong way? If so, what would be the alternative?
<p>I received a very long error message as I attempted to pickle and unpickle. The main failure comes from RunTimeError:</p>

<p><strong>Error</strong></p>

<pre><code>&lt;previous messages cut&gt;
File "/opt/conda/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/opt/conda/lib/python2.7/pickle.py", line 425, in save_reduce
save(state)
File "/opt/conda/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/opt/conda/lib/python2.7/pickle.py", line 655, in save_dict
self._batch_setitems(obj.iteritems())
File "/opt/conda/lib/python2.7/pickle.py", line 669, in _batch_setitems
save(v)
File "/opt/conda/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/opt/conda/lib/python2.7/pickle.py", line 425, in save_reduce
save(state)
File "/opt/conda/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/opt/conda/lib/python2.7/pickle.py", line 655, in save_dict
self._batch_setitems(obj.iteritems())
File "/opt/conda/lib/python2.7/pickle.py", line 669, in _batch_setitems
save(v)
File "/opt/conda/lib/python2.7/pickle.py", line 306, in save
rv = reduce(self.proto)
File "/opt/conda/lib/python2.7/copy_reg.py", line 74, in _reduce_ex
getstate = self.__getstate__
RuntimeError: maximum recursion depth exceeded while calling a Python object
</code></pre>

<p>The main purpose of my code is to scrap information from multiple web pages, dump it and then load it with pickle for further analysis. I aim to do this because I would not like to redo everything from scratch when I perform analysis everytime - pickle sounds like a great tool to have. Here is my code:</p>

<p><strong>Main code</strong></p>

<pre><code>import urllib
import urlparse
import re
from bs4 import BeautifulSoup

Links = ["http://www.newyorksocialdiary.com/party-pictures?page=" + str(i) for i in range(3,27)]
Rs = [urllib.urlopen(Link).read() for Link in Links]
soups = [BeautifulSoup(R) for R in Rs]
As = [soup.find_all('a', href=True) for soup in soups]

pattern = re.compile(r"/party-pictures/")
parties = [[a["href"] for a in soup.find_all('a', href=pattern)] for soup in soups]
flattened_parties = []
for party in parties:
    for y in party:
        flattened_parties.append(y)

## Make new soups with all of the 1193 links
NewLinks = ["http://www.newyorksocialdiary.com" + flattened_parties[i] for i in range(len(flattened_parties))]
NewRs = [urllib.urlopen(NewLink).read() for NewLink in NewLinks]
NewSoups = [BeautifulSoup(NewR) for NewR in NewRs]
</code></pre>

<p><strong>Pickle/Unpickle</strong></p>

<pre><code>import pickle
file_Name = "soupsfile"
fileObject = open(file_Name, 'wb')
pickle.dump(a, fileObject)
fileObject.close()

## open the file for reading
fileObject = open(file_Name, 'r')
#load the object from the file into var b
b = pickle.load(fileObject)

print(a==b)
</code></pre>

<p>Could someone help me understand the best strategy to store and retrieve information via pickle (or other modules)? Thank you!!</p>