{"action":"created","issue":{"url":"https://api.github.com/repos/smarnach/pyexiftool/issues/16","labels_url":"https://api.github.com/repos/smarnach/pyexiftool/issues/16/labels{/name}","comments_url":"https://api.github.com/repos/smarnach/pyexiftool/issues/16/comments","events_url":"https://api.github.com/repos/smarnach/pyexiftool/issues/16/events","html_url":"https://github.com/smarnach/pyexiftool/issues/16","id":106207887,"number":16,"title":"Use multiprocessing for further speedups","user":{"login":"fbuchinger","id":287434,"avatar_url":"https://avatars.githubusercontent.com/u/287434?v=3","gravatar_id":"","url":"https://api.github.com/users/fbuchinger","html_url":"https://github.com/fbuchinger","followers_url":"https://api.github.com/users/fbuchinger/followers","following_url":"https://api.github.com/users/fbuchinger/following{/other_user}","gists_url":"https://api.github.com/users/fbuchinger/gists{/gist_id}","starred_url":"https://api.github.com/users/fbuchinger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fbuchinger/subscriptions","organizations_url":"https://api.github.com/users/fbuchinger/orgs","repos_url":"https://api.github.com/users/fbuchinger/repos","events_url":"https://api.github.com/users/fbuchinger/events{/privacy}","received_events_url":"https://api.github.com/users/fbuchinger/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-09-13T09:41:15Z","updated_at":"2015-09-13T18:53:43Z","closed_at":null,"body":"Currently pyexiftool leverages the `stay_open` feature of exiftool to get a speed up compared to invoking one exiftool instance for each image. What if we took the ball further and used the python multiprocessing module to invoke multiple exiftool stay_open instances and split the load between them? Here is a script that compares the \"internal\" batch mode of exiftool, the \"external\" batchmode using pyexiftool and the \"multiprocessed external\" batchmode using multiprocessing:\r\n\r\n```python\r\nimport os\r\nimport timeit\r\nimport time\r\nimport multiprocessing\r\n\r\nimport exiftool\r\n\r\n\r\n\r\nmetadatadir = r\"E:\\dev\\exiftool\\test\\sampleImages\\Canon\" #adapt to your path\r\nresult_queue = multiprocessing.Queue(1)\r\n\r\ndef exiftool_internal_batch():\r\n    \"\"\"internal batchmode of exiftool - ET itself finds out which files to read\"\"\"\r\n    md = os.popen(r\"exiftool.exe -j %s\\\\*.jpg\" % metadatadir).read()\r\n    return md\r\n\r\ndef get_filelist():\r\n    files = []\r\n    for sample in os.listdir(metadatadir):\r\n        files.append(os.path.join(metadatadir, sample))\r\n    return files\r\n\r\ndef chunk_list(l, n):\r\n    \"\"\"Yield successive n-sized chunks from l.\"\"\"\r\n    result = []\r\n    for i in xrange(0, len(l), n):\r\n        result.append(l[i:i+n])\r\n    return result\r\n        \r\n\r\ndef exiftool_stay_open(files, mp = None):\r\n    \"\"\"external batchmode of exiftool - ET is invoked by pyexiftool using a stay_open with a filelist\r\n        filelist - list of files to read\r\n        mp - make function multiprocessing aware yes/no (results will be stuffed into a queue)\r\n    \"\"\"\r\n    with exiftool.ExifTool() as et:\r\n        metadata = et.get_metadata_batch(files)\r\n    if mp is not None:\r\n        result_queue.put(metadata)\r\n    return metadata\r\n\r\nif __name__ == '__main__':\r\n\r\n    internal_batch_start = time.clock()\r\n    exiftool_internal_batch()\r\n    internal_batch_end = time.clock()\r\n    print (\"Exiftool internal batch took %s\" % (internal_batch_end - internal_batch_start))\r\n    \r\n    external_batch_start = time.clock()\r\n    files = get_filelist()\r\n    exiftool_stay_open(files)\r\n    external_batch_end = time.clock()\r\n    print (\"Exiftool Stay Open/External batch took %s\" % (external_batch_end - external_batch_start))\r\n    \r\n    \"\"\"invoke three exiftool stay_open instances at once to split the load between them\"\"\"\r\n    \r\n    mbs_start = time.clock()\r\n    jobs = []\r\n    files = get_filelist()\r\n    fl_chunks = chunk_list(files, 3)\r\n    \r\n    et1 = multiprocessing.Process(target=exiftool_stay_open, args=(fl_chunks[0],True))\r\n    et2 = multiprocessing.Process(target=exiftool_stay_open, args=(fl_chunks[1],True))\r\n    et3 = multiprocessing.Process(target=exiftool_stay_open, args=(fl_chunks[2],True))\r\n    \r\n    jobs.append(et1)\r\n    jobs.append(et2)\r\n    jobs.append(et3)\r\n    \r\n    et1.start()\r\n    et2.start()\r\n    et3.start()\r\n    \r\n    et1.join()\r\n    et2.join()\r\n    et3.join()\r\n    \r\n\r\n    mbs_end = time.clock()\r\n    print (\"Exiftool multiprocessing batch took %s\" % (mbs_end - mbs_start))\r\n```\r\n\r\nWhen I run this script on the Canon sample images of Phil Harvey's metadata repository, I get these results on my machine:\r\n\r\nExiftool internal batch took 14.2978597714 sec\r\nExiftool Stay Open/External batch took 12.7552563562 sec\r\nExiftool multiprocessing batch took 0.714086082069 sec\r\n\r\nI have to admit 0,71 for multiprocessing sounds to good to be true, so there might be some error in the script. nevertheless, a previous version with two workers took 7,77 seconds, so there is quite a performance gain possible.\r\n\r\nDo you plan to add multiprocessing functionality to pyexiftool?"},"comment":{"url":"https://api.github.com/repos/smarnach/pyexiftool/issues/comments/139907623","html_url":"https://github.com/smarnach/pyexiftool/issues/16#issuecomment-139907623","issue_url":"https://api.github.com/repos/smarnach/pyexiftool/issues/16","id":139907623,"user":{"login":"smarnach","id":249196,"avatar_url":"https://avatars.githubusercontent.com/u/249196?v=3","gravatar_id":"","url":"https://api.github.com/users/smarnach","html_url":"https://github.com/smarnach","followers_url":"https://api.github.com/users/smarnach/followers","following_url":"https://api.github.com/users/smarnach/following{/other_user}","gists_url":"https://api.github.com/users/smarnach/gists{/gist_id}","starred_url":"https://api.github.com/users/smarnach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarnach/subscriptions","organizations_url":"https://api.github.com/users/smarnach/orgs","repos_url":"https://api.github.com/users/smarnach/repos","events_url":"https://api.github.com/users/smarnach/events{/privacy}","received_events_url":"https://api.github.com/users/smarnach/received_events","type":"User","site_admin":false},"created_at":"2015-09-13T18:53:43Z","updated_at":"2015-09-13T18:53:43Z","body":"@fbuchinger I had a quick look.  Since you use `/ 3` as the chunk size, this might leave a few single items at the end of the list due to rounding that require another exiftool run, but the effect of that is likely to be minor.\r\n\r\nI don't know much about Windows.  Generally I'd look at CPU usage to decide whether a process is doing anything, but since what exiftool does is mostly I/O, this might be misleading in this particular case."}}