{"action":"created","issue":{"url":"https://api.github.com/repos/libextract/libextract/issues/9","labels_url":"https://api.github.com/repos/libextract/libextract/issues/9/labels{/name}","comments_url":"https://api.github.com/repos/libextract/libextract/issues/9/comments","events_url":"https://api.github.com/repos/libextract/libextract/issues/9/events","html_url":"https://github.com/libextract/libextract/issues/9","id":67214909,"number":9,"title":"Mission statement: \"[libextract] provides composable, small functions [..]\"","user":{"login":"rodricios","id":6603965,"avatar_url":"https://avatars.githubusercontent.com/u/6603965?v=3","gravatar_id":"","url":"https://api.github.com/users/rodricios","html_url":"https://github.com/rodricios","followers_url":"https://api.github.com/users/rodricios/followers","following_url":"https://api.github.com/users/rodricios/following{/other_user}","gists_url":"https://api.github.com/users/rodricios/gists{/gist_id}","starred_url":"https://api.github.com/users/rodricios/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rodricios/subscriptions","organizations_url":"https://api.github.com/users/rodricios/orgs","repos_url":"https://api.github.com/users/rodricios/repos","events_url":"https://api.github.com/users/rodricios/events{/privacy}","received_events_url":"https://api.github.com/users/rodricios/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/libextract/libextract/labels/comment/concern","name":"comment/concern","color":"d4c5f9"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-04-08T20:38:28Z","updated_at":"2015-04-09T01:53:19Z","closed_at":null,"body":"\"[..] that can be piped together to process the HT/XML document.\"\r\n\r\nI'd like to put into writing that I think we should not only stick to that mantra (or mission statement?), but also officialize it as this project's description. \r\n\r\nWith that in mind, I think we're reaching a point (or we've reached it already ref: 953c384b3cb1d8ae06f654f893c44e82cc74c835) where the lack of naming/functional-method conventions will delay the progress of this project. \r\n\r\nThere's a few observations I should state:\r\n\r\n1. Perfect extraction is impossible (this one is here for reasons I'll address in another issue/post)\r\n2. Lambda functions are easy for the initiated, difficult for those not.\r\n3. The \"composable, small functions\" we provide in html.py (we should probably rename that file, if we continue to add more functions) can be, to an extent, rewritten as a series of \"pipelined\" lambda functions. \r\n4. Without some set of protocols, the typical user will not be able to simply pick up this library and compose an effective \"strategy\"\r\n\r\nTo stress the problem of point 4, consider the following two extraction cases:\r\n\r\n```python\r\narticle_strategy = (get_etree, get_pairs, histogram, argmax, get_final_text)\r\n\r\ntabular_strategy = (get_etree, get_node_children_pairs, filter_node_children_pairs)\r\n```\r\n\r\nFirst, \"get_etree\", while I'm for keeping method names that describe the action and the returned structure, I believe we should opt for something closer to \"parse\" or \"parse_html\". Not only is it interpretable by most users of lxml.html and the nlp folks (ie. sentence parsing), but it's also, imo, a more concise statement. \r\n\r\nSecond, as is evidenced in the definitions of *get_pairs* and *get_node_children_pairs*, we're doing a lot of *pair getting*. \r\n\r\nIn both types of pairs returned, it's of the format **(HtmlElement, n-dimensional measurement)**. \r\n\r\nJust for completeness, in *get_pairs* we have **(HtmlElement, int)** and in get_node_children_pairs we have **(HtmlElement,** [Counter(key=**HtmlElement**, value=**int*(frequency*))**\r\n\r\nUp until that point, our pipeline's flow of logic, and the names of the methods, matched up nicely. But after that, *histogram*, *argmax*, *get_final_text* and *filter_node_children_pairs* are inelegantly non-synergistic, for lack of a better term. \r\n\r\n\r\n\r\nI don't know if it's too early in the project to be concerned about this, but I think this project will likely head towards a dead end if there isn't some proper set of ground rules. Sorry for the pessimism :(\r\n\r\nI'll work on refactoring the code I've commited in 953c384b3cb1d8ae06f654f893c44e82cc74c835, and see if I can come up with something more constructive than the above."},"comment":{"url":"https://api.github.com/repos/libextract/libextract/issues/comments/91088845","html_url":"https://github.com/libextract/libextract/issues/9#issuecomment-91088845","issue_url":"https://api.github.com/repos/libextract/libextract/issues/9","id":91088845,"user":{"login":"rodricios","id":6603965,"avatar_url":"https://avatars.githubusercontent.com/u/6603965?v=3","gravatar_id":"","url":"https://api.github.com/users/rodricios","html_url":"https://github.com/rodricios","followers_url":"https://api.github.com/users/rodricios/followers","following_url":"https://api.github.com/users/rodricios/following{/other_user}","gists_url":"https://api.github.com/users/rodricios/gists{/gist_id}","starred_url":"https://api.github.com/users/rodricios/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rodricios/subscriptions","organizations_url":"https://api.github.com/users/rodricios/orgs","repos_url":"https://api.github.com/users/rodricios/repos","events_url":"https://api.github.com/users/rodricios/events{/privacy}","received_events_url":"https://api.github.com/users/rodricios/received_events","type":"User","site_admin":false},"created_at":"2015-04-09T01:53:19Z","updated_at":"2015-04-09T01:53:19Z","body":"That looks great! Let's go ahead and merge (rebase?) the changes. "}}