{"action":"opened","issue":{"url":"https://api.github.com/repos/datamade/dedupe/issues/337","labels_url":"https://api.github.com/repos/datamade/dedupe/issues/337/labels{/name}","comments_url":"https://api.github.com/repos/datamade/dedupe/issues/337/comments","events_url":"https://api.github.com/repos/datamade/dedupe/issues/337/events","html_url":"https://github.com/datamade/dedupe/issues/337","id":55079201,"number":337,"title":"ZeroDivisionError: float division by zero","user":{"login":"comckay","id":9514814,"avatar_url":"https://avatars.githubusercontent.com/u/9514814?v=3","gravatar_id":"","url":"https://api.github.com/users/comckay","html_url":"https://github.com/comckay","followers_url":"https://api.github.com/users/comckay/followers","following_url":"https://api.github.com/users/comckay/following{/other_user}","gists_url":"https://api.github.com/users/comckay/gists{/gist_id}","starred_url":"https://api.github.com/users/comckay/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/comckay/subscriptions","organizations_url":"https://api.github.com/users/comckay/orgs","repos_url":"https://api.github.com/users/comckay/repos","events_url":"https://api.github.com/users/comckay/events{/privacy}","received_events_url":"https://api.github.com/users/comckay/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-01-21T21:24:47Z","updated_at":"2015-01-21T21:24:47Z","closed_at":null,"body":"I've gotten Dedupe to go through the training, blocking and clustering phases but am getting this as the output : \r\n```\r\nTraceback (most recent call last):\r\n\r\n  File \"<ipython-input-6-914b2b56363a>\", line 1, in <module>\r\n    runfile('H:/My Documents/dedupe-examples-master/Sam_Test/DupeTest.py', wdir='H:/My Documents/dedupe-examples-master/Sam_Test')\r\n\r\n  File \"C:\\Anaconda\\lib\\site-packages\\spyderlib\\widgets\\externalshell\\sitecustomize.py\", line 580, in runfile\r\n    execfile(filename, namespace)\r\n\r\n  File \"H:/My Documents/dedupe-examples-master/Sam_Test/DupeTest.py\", line 146, in <module>\r\n    clustered_dupes = deduper.match(data_d, threshold)\r\n\r\n  File \"C:\\Anaconda\\lib\\site-packages\\dedupe\\api.py\", line 201, in match\r\n    return self.matchBlocks(blocked_pairs, threshold)\r\n\r\n  File \"C:\\Anaconda\\lib\\site-packages\\dedupe\\api.py\", line 119, in matchBlocks\r\n    threshold)\r\n\r\n  File \"C:\\Anaconda\\lib\\site-packages\\dedupe\\core.py\", line 270, in scoreDuplicates\r\n    fillQueue(record_pairs_queue, records, n_map_processes)\r\n\r\n  File \"C:\\Anaconda\\lib\\site-packages\\dedupe\\core.py\", line 306, in fillQueue\r\n    current_rate = n_records/time_delta\r\n\r\nZeroDivisionError: float division by zero\r\n```\r\n\r\nHere is my code:\r\n\r\n```python\r\nimport os\r\nimport csv\r\nimport re\r\nimport collections\r\nimport logging\r\nimport optparse\r\nfrom numpy import nan\r\n\r\n\r\nimport dedupe\r\nfrom unidecode import unidecode\r\n\r\n# ## Logging\r\n\r\n# Dedupe uses Python logging to show or suppress verbose output. Added for convenience.\r\n# To enable verbose logging, run `python examples/csv_example/csv_example.py -v`\r\noptp = optparse.OptionParser()\r\noptp.add_option('-v', '--verbose', dest='verbose', action='count',\r\n                help='Increase verbosity (specify multiple times for more)'\r\n                )\r\n(opts, args) = optp.parse_args()\r\nlog_level = logging.WARNING \r\nif opts.verbose == 1:\r\n    log_level = logging.INFO\r\nelif opts.verbose >= 2:\r\n    log_level = logging.DEBUG\r\nlogging.getLogger().setLevel(log_level)\r\n\r\n\r\n# ## Setup\r\n\r\ninput_file = 'DupeTester.csv'\r\noutput_file = 'DupeTesterOut.csv'\r\nsettings_file = 'csv_example_learned_settings'\r\ntraining_file = 'csv_example_training.json'\r\n\r\n\r\ndef preProcess(column):\r\n    \"\"\"\r\n    Do a little bit of data cleaning with the help of Unidecode and Regex.\r\n    Things like casing, extra spaces, quotes and new lines can be ignored.\r\n    \"\"\"\r\n    import unidecode\r\n    column = column.decode(\"utf8\")\r\n    column = unidecode.unidecode(column)\r\n    column = re.sub('  +', ' ', column)\r\n    column = re.sub('\\n', ' ', column)\r\n    column = column.strip().strip('\"').strip(\"'\").lower().strip()\r\n    return column\r\n\r\n\r\ndef readData(filename):\r\n    \"\"\"\r\n    Read in our data from a CSV file and create a dictionary of records, \r\n    where the key is a unique record ID and each value is dict\r\n    \"\"\"\r\n\r\n    data_d = {}\r\n    with open(filename) as f:\r\n        reader = csv.DictReader(f)\r\n        for row in reader:\r\n            clean_row = [(k, preProcess(v)) for (k, v) in row.items()]\r\n            row_id = int(row['Id'])\r\n            data_d[row_id] = dict(clean_row)\r\n\r\n    return data_d\r\n\r\n\r\nprint 'importing data ...'\r\ndata_d = readData(input_file)\r\n\r\n# ## Training\r\n\r\nif os.path.exists(settings_file):\r\n    print 'reading from', settings_file\r\n    with open(settings_file, 'rb') as f:\r\n        deduper = dedupe.StaticDedupe(f)\r\n\r\nelse:\r\n    # Define the fields dedupe will pay attention to\r\n    #\r\n    # Notice how we are telling dedupe to use a custom field comparator\r\n    # for the 'Zip' field. \r\n    fields = [\r\n        {'field' : 'fulladdr', 'type': 'String',\r\n        \"Has Missing\" : True},\r\n        {'field' : 'zip', 'type': 'Exact'}]\r\n\r\n    # Create a new deduper object and pass our data model to it.\r\n    deduper = dedupe.Dedupe(fields)\r\n\r\n    # To train dedupe, we feed it a sample of records.\r\n    deduper.sample(data_d, 1500)\r\n\r\n\r\n    # If we have training data saved from a previous run of dedupe,\r\n    # look for it an load it in.\r\n    # __Note:__ if you want to train from scratch, delete the training_file\r\n    if os.path.exists(training_file):\r\n        print 'reading labeled examples from ', training_file\r\n        with open(training_file, 'rb') as f:\r\n            deduper.readTraining(f)\r\n\r\n    # ## Active learning\r\n    # Dedupe will find the next pair of records\r\n    # it is least certain about and ask you to label them as duplicates\r\n    # or not.\r\n    # use 'y', 'n' and 'u' keys to flag duplicates\r\n    # press 'f' when you are finished\r\n    print 'starting active labeling...'\r\n\r\n    dedupe.consoleLabel(deduper)\r\n\r\n    deduper.train()\r\n\r\n    # When finished, save our training away to disk\r\n    with open(training_file, 'w') as tf :\r\n        deduper.writeTraining(tf)\r\n\r\n    # Save our weights and predicates to disk.  If the settings file\r\n    # exists, we will skip all the training and learning next time we run\r\n    # this file.\r\n    with open(settings_file, 'w') as sf :\r\n        deduper.writeSettings(sf)\r\n\r\n\r\n# ## Blocking\r\n\r\nprint 'blocking...'\r\n\r\n# ## Clustering\r\n\r\n# Find the threshold that will maximize a weighted average of our precision and recall. \r\n# When we set the recall weight to 2, we are saying we care twice as much\r\n# about recall as we do precision.\r\n#\r\n# If we had more data, we would not pass in all the blocked data into\r\n# this function but a representative sample.\r\n\r\nthreshold = deduper.threshold(data_d, recall_weight=2)\r\n\r\n# `match` will return sets of record IDs that dedupe\r\n# believes are all referring to the same entity.\r\n\r\nprint 'clustering...'\r\nclustered_dupes = deduper.match(data_d, threshold)\r\n\r\nprint '# duplicate sets', len(clustered_dupes)\r\n\r\n# ## Writing Results\r\n\r\n# Write our original data back out to a CSV with a new column called \r\n# 'Cluster ID' which indicates which records refer to each other.\r\n\r\ncluster_membership = {}\r\ncluster_id = 0\r\nfor (cluster_id, cluster) in enumerate(clustered_dupes):\r\n    id_set, scores = cluster\r\n    cluster_d = [data_d[c] for c in id_set]\r\n    canonical_rep = dedupe.canonicalize(cluster_d)\r\n    for record_id, score in zip(id_set, scores) :\r\n        cluster_membership[record_id] = {\r\n            \"cluster id\" : cluster_id,\r\n            \"canonical representation\" : canonical_rep,\r\n            \"confidence\": score\r\n        }\r\n\r\nsingleton_id = cluster_id + 1\r\n\r\nwith open(output_file, 'w') as f_output:\r\n    writer = csv.writer(f_output)\r\n\r\n    with open(input_file) as f_input :\r\n        reader = csv.reader(f_input)\r\n\r\n        heading_row = reader.next()\r\n        heading_row.insert(0, 'confidence_score')\r\n        heading_row.insert(0, 'Cluster ID')\r\n        canonical_keys = canonical_rep.keys()\r\n        for key in canonical_keys:\r\n            heading_row.append('canonical_' + key)\r\n        \r\n        writer.writerow(heading_row)\r\n\r\n        for row in reader:\r\n            row_id = int(row[0])\r\n            if row_id in cluster_membership :\r\n                cluster_id = cluster_membership[row_id][\"cluster id\"]\r\n                canonical_rep = cluster_membership[row_id][\"canonical representation\"]\r\n                row.insert(0, cluster_membership[row_id]['confidence'])\r\n                row.insert(0, cluster_id)\r\n                for key in canonical_keys:\r\n                    row.append(canonical_rep[key].encode('utf8'))\r\n            else:\r\n                row.insert(0, None)\r\n                row.insert(0, singleton_id)\r\n                singleton_id += 1\r\n                for key in canonical_keys:\r\n                    row.append(None)\r\n            writer.writerow(row)\r\n```\r\nI understand that some 0's are in the mix causing this to fail but I cant get a handle on what to change "}}