{"action":"opened","issue":{"url":"https://api.github.com/repos/stripe/mosql/issues/90","labels_url":"https://api.github.com/repos/stripe/mosql/issues/90/labels{/name}","comments_url":"https://api.github.com/repos/stripe/mosql/issues/90/comments","events_url":"https://api.github.com/repos/stripe/mosql/issues/90/events","html_url":"https://github.com/stripe/mosql/issues/90","id":67390716,"number":90,"title":"GridFS Support","user":{"login":"apocolipse","id":260871,"avatar_url":"https://avatars.githubusercontent.com/u/260871?v=3","gravatar_id":"","url":"https://api.github.com/users/apocolipse","html_url":"https://github.com/apocolipse","followers_url":"https://api.github.com/users/apocolipse/followers","following_url":"https://api.github.com/users/apocolipse/following{/other_user}","gists_url":"https://api.github.com/users/apocolipse/gists{/gist_id}","starred_url":"https://api.github.com/users/apocolipse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apocolipse/subscriptions","organizations_url":"https://api.github.com/users/apocolipse/orgs","repos_url":"https://api.github.com/users/apocolipse/repos","events_url":"https://api.github.com/users/apocolipse/events{/privacy}","received_events_url":"https://api.github.com/users/apocolipse/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-04-09T15:48:32Z","updated_at":"2015-04-09T15:48:32Z","closed_at":null,"body":"I'm curious if you've looked into GridFS support,  being that gridfs is split across 2 collections, they're consistently named (fs.files, fs.chunks), and the standalone adapter for gridfs file getting (by filename or id), I think it merits its own functionality, rather than just mappign both collections to postgres and trying to do assembly on that side.  I did some preliminary testing to see if it could work (using '$gridfs' special as a source to trigger gridfs, and then using orig document to grab gridfs file by id)\r\n\r\nI'm currently running into some issues with encoding however, some imports succeeding (large plaintext files, some pdfs) and then failing at one point on others on\r\n```\r\n# in transform_to_copy()\r\n'join': incompatible character encodings: UTF-8 and ASCII-8BIT (Encoding::CompatibilityError)\r\n``` \r\nmy modification of fetch_special_source():\r\n```\r\ndef fetch_special_source(db, ns, obj, source, original)\r\n      case source\r\n      when \"$timestamp\"\r\n        Sequel.function(:now)\r\n      when \"$data\"\r\n        dbname, collection = ns.split(\".\", 2)\r\n        if collection == 'fs.files'\r\n          grid = Grid.new(db)\r\n          file = grid.get(original[\"_id\"])\r\n          Sequel::SQL::Blob.new(file.read)\r\n        end\r\n      when /^\\$exists (.+)/\r\n        # We need to look in the cloned original object, not in the version that\r\n        # has had some fields deleted.\r\n        fetch_exists(original, $1)\r\n      else\r\n        raise SchemaError.new(\"Unknown source: #{source}\")\r\n      end\r\n    end\r\n```\r\n\r\n(I also tried various combinations of hex transforms and utf8 encoding, it still ended up eventually giving me that ASCII error, for reference my column type its inserting into is BYTEA)\r\n\r\nAlso, I had to add db adapter arguments in all methods up from fetch_special_source() in shema.rb to import_collections() in streamer.rb inorder to create the gridfs object instance in fetch_special_source(), this seems bad, recommendation for where to stick it? "}}