{"action":"opened","issue":{"url":"https://api.github.com/repos/ssbc/scuttlebot/issues/96","labels_url":"https://api.github.com/repos/ssbc/scuttlebot/issues/96/labels{/name}","comments_url":"https://api.github.com/repos/ssbc/scuttlebot/issues/96/comments","events_url":"https://api.github.com/repos/ssbc/scuttlebot/issues/96/events","html_url":"https://github.com/ssbc/scuttlebot/issues/96","id":55382134,"number":96,"title":"reducing blob-sync load: registering interest","user":{"login":"pfraze","id":1270099,"avatar_url":"https://avatars.githubusercontent.com/u/1270099?v=3","gravatar_id":"","url":"https://api.github.com/users/pfraze","html_url":"https://github.com/pfraze","followers_url":"https://api.github.com/users/pfraze/followers","following_url":"https://api.github.com/users/pfraze/following{/other_user}","gists_url":"https://api.github.com/users/pfraze/gists{/gist_id}","starred_url":"https://api.github.com/users/pfraze/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pfraze/subscriptions","organizations_url":"https://api.github.com/users/pfraze/orgs","repos_url":"https://api.github.com/users/pfraze/repos","events_url":"https://api.github.com/users/pfraze/events{/privacy}","received_events_url":"https://api.github.com/users/pfraze/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ssbc/scuttlebot/labels/discussion","name":"discussion","color":"fef2c0"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-01-24T19:34:56Z","updated_at":"2015-01-24T19:34:56Z","closed_at":null,"body":"i think we're going to have congestion issues if we don't add sophistication to the blob-syncing strategy. even if every user is a good citizen, the dataset will get large fast. if a pub follows 100 users, each of which has posted 10mb of data, that's a gig of data for the initial sync. that can't work!\r\n\r\nmost peers, i think, need to download blobs only after registering interest in them. we can come up with different ways to do this, but i think they'd break down to either\r\n - \"give me every blob this feed posts\"\r\n - \"give me every blob in this message\"\r\n - \"give me this blob\"\r\n\r\ni think phoenix's UX should focus around the latter two, and maybe make the first an option\r\n\r\nwith pubs, we could use the first strategy and auto-download every blob its members post (perhaps under a size limit) but ignore blobs by friends of friends. another thing we could do is let users \"push\" blobs to a server - that is, request that the pub take the blob. this would allow members to upload non-members' blobs.\r\n\r\npubs can deal with a slightly higher load overall, but they may need to have a prioritization tool so that they only fetch blobs when feed-syncing doesnt ned to take precedence."}}