{"action":"created","issue":{"url":"https://api.github.com/repos/MicroB3-IS/osd-analysis/issues/18","labels_url":"https://api.github.com/repos/MicroB3-IS/osd-analysis/issues/18/labels{/name}","comments_url":"https://api.github.com/repos/MicroB3-IS/osd-analysis/issues/18/comments","events_url":"https://api.github.com/repos/MicroB3-IS/osd-analysis/issues/18/events","html_url":"https://github.com/MicroB3-IS/osd-analysis/issues/18","id":86556842,"number":18,"title":"Questions about metagnomic data at EMG compared to documentation","user":{"login":"renzok","id":3148886,"avatar_url":"https://avatars.githubusercontent.com/u/3148886?v=3","gravatar_id":"","url":"https://api.github.com/users/renzok","html_url":"https://github.com/renzok","followers_url":"https://api.github.com/users/renzok/followers","following_url":"https://api.github.com/users/renzok/following{/other_user}","gists_url":"https://api.github.com/users/renzok/gists{/gist_id}","starred_url":"https://api.github.com/users/renzok/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/renzok/subscriptions","organizations_url":"https://api.github.com/users/renzok/orgs","repos_url":"https://api.github.com/users/renzok/repos","events_url":"https://api.github.com/users/renzok/events{/privacy}","received_events_url":"https://api.github.com/users/renzok/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/MicroB3-IS/osd-analysis/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/MicroB3-IS/osd-analysis/labels/documentation","name":"documentation","color":"bfd4f2"},{"url":"https://api.github.com/repos/MicroB3-IS/osd-analysis/labels/metagenomes","name":"metagenomes","color":"f7c6c7"},{"url":"https://api.github.com/repos/MicroB3-IS/osd-analysis/labels/question","name":"question","color":"cc317c"}],"state":"open","locked":false,"assignee":{"login":"renzok","id":3148886,"avatar_url":"https://avatars.githubusercontent.com/u/3148886?v=3","gravatar_id":"","url":"https://api.github.com/users/renzok","html_url":"https://github.com/renzok","followers_url":"https://api.github.com/users/renzok/followers","following_url":"https://api.github.com/users/renzok/following{/other_user}","gists_url":"https://api.github.com/users/renzok/gists{/gist_id}","starred_url":"https://api.github.com/users/renzok/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/renzok/subscriptions","organizations_url":"https://api.github.com/users/renzok/orgs","repos_url":"https://api.github.com/users/renzok/repos","events_url":"https://api.github.com/users/renzok/events{/privacy}","received_events_url":"https://api.github.com/users/renzok/received_events","type":"User","site_admin":false},"milestone":null,"comments":8,"created_at":"2015-06-09T11:07:52Z","updated_at":"2015-06-11T14:47:01Z","closed_at":null,"body":"The below issue(s) questions were raised by Pascal.\r\n-------\r\n\r\nHi,\r\n\r\nAs already discussed, there is little reason to work with the OSD raw reads (other than technical or QC questions), so within the scope of the OSD analysis paper we should mostly all be using the workable reads (and assemblies, when available), but at this point I'm a little confused as to where the main workable reads may be hiding :)\r\n\r\nCan I make a quick sanity-check with you about the location of the OSD raw and workable read data, taking for example ERR770976:\r\nRaw reads (after demultiplexing + adapter clipping as defined on gitHub) are here:\r\n    https://www.ebi.ac.uk/ena/data/view/ERR770976\r\n  With two nice looking fastq PE files:\r\n    ftp://ftp.sra.ebi.ac.uk/vol1/ERA413/ERA413491/fastq/OSD73_R2_shotgun_raw.fastq.gz\r\n    ftp://ftp.sra.ebi.ac.uk/vol1/ERA413/ERA413491/fastq/OSD73_R1_shotgun_raw.fastq.gz\r\n    containing 1149852 * 2 reads\r\nA certain flavor of workable reads (after quality trimming + length filtering as defined on gitHub) are here:\r\n    https://www.ebi.ac.uk/metagenomics/projects/ERP009703/samples/ERS667589/runs/ERR770976/results/versions/2.0\r\n  Which provides a link to the above raw reads, and a link to \"Processed nucleotide reads (FASTA)\":\r\n    https://www.ebi.ac.uk/metagenomics/projects/ERP009703/samples/ERS667589/runs/ERR770976/results/sequences/versions/2.0/export?contentType=text&exportValue=processedReads\r\n  which downloads a file which from its name look like merged reads:\r\nERR770976_MERGED_FASTQ_nt_reads.fasta\r\n  it contains 975769 reads, not incompatible with the raw PE read number above.\r\n\r\nThere are a number of questions:\r\n1/ is this file ERR770976_MERGED_FASTQ_nt_reads.fasta the merged workable reads file?\r\n2/ but then how come this file has an extra \"ambiguous bas filtering\" step which is not documented on gitHub in (as seen in the \"Quality Control\" tab of https://www.ebi.ac.uk/metagenomics/projects/ERP009703/samples/ERS667589/runs/ERR770976/results/versions/2.0)\r\n3/ where are the workable un-merged reads which are described on gitHub as part of the \"shotgun data the output files per sample\".\r\n4/ why is the \"initial\" (I guess \"raw\") number of reads (1,241,922) in the EMG \"Quality Control\" tab different from the raw read number on the ENA page above (1,149,852). Because the EMG analysis appears based on more reads than the supposedly official raw dataset, is it possible that EMG started with the \"pre-raw\" data set, ie the files directly obtained from the sequencing company?\r\n\r\nSo it seems that in fact the data located on the EMG portal are not the workable dataset described on giHub documentation, but instead an EMG specific dataset based on a different raw dataset, using a different read pre-processing?\r\n\r\nI seem to remember we'd agreed that we should all be working with the same data sets, or else how are we to hope to compare and merge our results, plus the mat & met section is going to be one read pre-processing method for each OSD analysis partner?\r\n\r\nThanks ever so much for your help (and patience as this might have been said before - but trust me I did really do my best to search before writing this mail, so it might in fact just be an issue with documentation or my poor understanding).\r\nCheers,\r\nPascal"},"comment":{"url":"https://api.github.com/repos/MicroB3-IS/osd-analysis/issues/comments/111158935","html_url":"https://github.com/MicroB3-IS/osd-analysis/issues/18#issuecomment-111158935","issue_url":"https://api.github.com/repos/MicroB3-IS/osd-analysis/issues/18","id":111158935,"user":{"login":"hingamp","id":12284507,"avatar_url":"https://avatars.githubusercontent.com/u/12284507?v=3","gravatar_id":"","url":"https://api.github.com/users/hingamp","html_url":"https://github.com/hingamp","followers_url":"https://api.github.com/users/hingamp/followers","following_url":"https://api.github.com/users/hingamp/following{/other_user}","gists_url":"https://api.github.com/users/hingamp/gists{/gist_id}","starred_url":"https://api.github.com/users/hingamp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hingamp/subscriptions","organizations_url":"https://api.github.com/users/hingamp/orgs","repos_url":"https://api.github.com/users/hingamp/repos","events_url":"https://api.github.com/users/hingamp/events{/privacy}","received_events_url":"https://api.github.com/users/hingamp/received_events","type":"User","site_admin":false},"created_at":"2015-06-11T14:47:01Z","updated_at":"2015-06-11T14:47:01Z","body":"In fact it looks like the workable data is here:\r\nhttps://zarafa.mpi-bremen.de/owncloud/public.php?service=files&t=f0e297f451b9944b32f4b2366c74cde3&path=/datasets/workable/metagenomes"}}