{"action":"created","issue":{"url":"https://api.github.com/repos/BioJulia/Bio.jl/issues/79","labels_url":"https://api.github.com/repos/BioJulia/Bio.jl/issues/79/labels{/name}","comments_url":"https://api.github.com/repos/BioJulia/Bio.jl/issues/79/comments","events_url":"https://api.github.com/repos/BioJulia/Bio.jl/issues/79/events","html_url":"https://github.com/BioJulia/Bio.jl/issues/79","id":101335249,"number":79,"title":"Rethinking about the parsing interface","user":{"login":"bicycle1885","id":905683,"avatar_url":"https://avatars.githubusercontent.com/u/905683?v=3","gravatar_id":"","url":"https://api.github.com/users/bicycle1885","html_url":"https://github.com/bicycle1885","followers_url":"https://api.github.com/users/bicycle1885/followers","following_url":"https://api.github.com/users/bicycle1885/following{/other_user}","gists_url":"https://api.github.com/users/bicycle1885/gists{/gist_id}","starred_url":"https://api.github.com/users/bicycle1885/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bicycle1885/subscriptions","organizations_url":"https://api.github.com/users/bicycle1885/orgs","repos_url":"https://api.github.com/users/bicycle1885/repos","events_url":"https://api.github.com/users/bicycle1885/events{/privacy}","received_events_url":"https://api.github.com/users/bicycle1885/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-08-17T03:59:22Z","updated_at":"2015-08-18T19:26:18Z","closed_at":null,"body":"Thanks to the great effort of @dcjones, we're getting a powerful framework to parse various data formats at the speed of light.\r\nBut I think that the interface of the parser can be more flexible and idiomatic in the context of Julia.\r\nThe current interface is the `read` method only, and its behavior is significantly different from the methods defined in the standard library.\r\nIn the `Base` module, `read(io, type)` always returns an object of `type` but `read(io, FASTA)` returns an iterator of biological sequences; this is counterintuitive to me.\r\n\r\nI propose the following functions as public APIs on the top of the parser framework:\r\n\r\n* `open(filename, format)`: open `filename` and return a stream indexed by `format`\r\n* `read(stream, type)`: read a `type` object from `stream`\r\n* `read!(stream, object)`: read a record into the preallocated `object` from `stream`\r\n* `each(stream, type)`: return an iterator of `type` objects from `stream`\r\n* `close(stream)`: close stream.\r\n\r\n`open`, `read`, `read!` and `close` behave like methods in the standard library, respectively. `each` is something like `Base.eachline`.\r\nA stream object is indexed by a file format: defined as `Stream{F<:FileFromat}`, and `Stream{FASTA}` is, for example,  a stream type of FASTA format..\r\n\r\nThe rationale behind this idea is that file format and data type are not completely dependent.\r\nFor example, we can read DNA sequences from several formats (including FASTA, FASTQ, SAM, etc.) and a SAM file can generate several kinds of data (including DNA sequence, alignment, quality of sequence, etc.).\r\n\r\nThe next code may illustrate my idea clearly:\r\n\r\n```julia\r\ns = open(\"hg38.fa\", FASTA)\r\nwhile !eof(s)\r\n    seq = read(s, DNASequence)\r\n    ...\r\nend\r\nclose(s)\r\n```\r\nor\r\n```julia\r\nopen(\"hg38.fa\", FASTA) do s\r\n    for seq in each(s, DNASequence)\r\n        ...\r\n    end\r\nend\r\n```\r\n\r\nIf there is numerous number of records in a file, you can save memory allocation using `read!` as follows:\r\n\r\n```julia\r\ns = open(\"sample1234.bam\", BAM)\r\nseq = DNASequence()\r\nwhile !eof(s)\r\n    read!(s, seq)\r\n    ...\r\nend\r\nclose(s)\r\n```\r\n\r\nThe multiple dispatching in Julia would enable this interface very easily; a method `read(::Stream{Format}, ::Type{Record})` will be defined only if a `Record` object can be extracted from a `Format` stream.\r\n\r\nI'd like to know your opinions. Do you think this is reasonable?"},"comment":{"url":"https://api.github.com/repos/BioJulia/Bio.jl/issues/comments/132324819","html_url":"https://github.com/BioJulia/Bio.jl/issues/79#issuecomment-132324819","issue_url":"https://api.github.com/repos/BioJulia/Bio.jl/issues/79","id":132324819,"user":{"login":"dcjones","id":210361,"avatar_url":"https://avatars.githubusercontent.com/u/210361?v=3","gravatar_id":"","url":"https://api.github.com/users/dcjones","html_url":"https://github.com/dcjones","followers_url":"https://api.github.com/users/dcjones/followers","following_url":"https://api.github.com/users/dcjones/following{/other_user}","gists_url":"https://api.github.com/users/dcjones/gists{/gist_id}","starred_url":"https://api.github.com/users/dcjones/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dcjones/subscriptions","organizations_url":"https://api.github.com/users/dcjones/orgs","repos_url":"https://api.github.com/users/dcjones/repos","events_url":"https://api.github.com/users/dcjones/events{/privacy}","received_events_url":"https://api.github.com/users/dcjones/received_events","type":"User","site_admin":false},"created_at":"2015-08-18T19:26:18Z","updated_at":"2015-08-18T19:26:18Z","body":"Thanks for the comments @bicycle1885. I'm generally on board with this idea, but there are a few details to work out.\r\n\r\nWhile you can read a `DNASequence` from a many formats, each has a different set of a metadata (FASTA with name and description, FASTQ with name and quality scores, SAM with alignment info and basically anything with tags). So in some sense each file format has a natural type, and reading anything else involves some conversion. I think your `read(stream, type)` syntax is convenient, but we should also have just `read(stream)` that returns on object of the default type.\r\n\r\nWe could implement `read(stream, type)` as just `read(stream, type) = convert(type, read(stream))`, but I wonder if there's a more clever way to do this. If you are are reading SAM and only interested in the alignments (which is often the case), in theory you could make the parsing faster by not saving or validating sequences or quality scores.\r\n\r\nI very much agree with the need for a `read!` function to do in-place non-allocating reading. With something like FASTQ or SAM/BAM allocation and GC is going to be bottleneck (I think that's already the case with FASTQ), so to be competitive with C we need to provide that low-level escape hatch.\r\n\r\nThe tricky part of this is that Julia strings and Bio.jl sequences are both immutable by convention. So we can overwrite the underlying arrays and avoid allocation, but it could lead to some very surprising results. I guess that's fine, as long we have the appropriate warnings and documentation. I'm tempted to call these functions `unsafe_read!`, just so people really know what they're getting into."}}