{"action":"created","issue":{"url":"https://api.github.com/repos/synkarius/caster/issues/49","labels_url":"https://api.github.com/repos/synkarius/caster/issues/49/labels{/name}","comments_url":"https://api.github.com/repos/synkarius/caster/issues/49/comments","events_url":"https://api.github.com/repos/synkarius/caster/issues/49/events","html_url":"https://github.com/synkarius/caster/issues/49","id":77353794,"number":49,"title":"make fuzzy symbol searching smarter","user":{"login":"synkarius","id":2197849,"avatar_url":"https://avatars.githubusercontent.com/u/2197849?v=3","gravatar_id":"","url":"https://api.github.com/users/synkarius","html_url":"https://github.com/synkarius","followers_url":"https://api.github.com/users/synkarius/followers","following_url":"https://api.github.com/users/synkarius/following{/other_user}","gists_url":"https://api.github.com/users/synkarius/gists{/gist_id}","starred_url":"https://api.github.com/users/synkarius/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/synkarius/subscriptions","organizations_url":"https://api.github.com/users/synkarius/orgs","repos_url":"https://api.github.com/users/synkarius/repos","events_url":"https://api.github.com/users/synkarius/events{/privacy}","received_events_url":"https://api.github.com/users/synkarius/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/synkarius/caster/labels/enhancement","name":"enhancement","color":"84b6eb"},{"url":"https://api.github.com/repos/synkarius/caster/labels/help%20wanted","name":"help wanted","color":"fbca04"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-05-17T17:26:25Z","updated_at":"2015-11-08T09:55:50Z","closed_at":null,"body":"https://github.com/synkarius/caster/blob/master/caster/lib/pita/selector.py#L11\r\n\r\nMy original shot at a fuzzy string matching algorithm sucks. It needs to be better."},"comment":{"url":"https://api.github.com/repos/synkarius/caster/issues/comments/154796398","html_url":"https://github.com/synkarius/caster/issues/49#issuecomment-154796398","issue_url":"https://api.github.com/repos/synkarius/caster/issues/49","id":154796398,"user":{"login":"chilimangoes","id":1724779,"avatar_url":"https://avatars.githubusercontent.com/u/1724779?v=3","gravatar_id":"","url":"https://api.github.com/users/chilimangoes","html_url":"https://github.com/chilimangoes","followers_url":"https://api.github.com/users/chilimangoes/followers","following_url":"https://api.github.com/users/chilimangoes/following{/other_user}","gists_url":"https://api.github.com/users/chilimangoes/gists{/gist_id}","starred_url":"https://api.github.com/users/chilimangoes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chilimangoes/subscriptions","organizations_url":"https://api.github.com/users/chilimangoes/orgs","repos_url":"https://api.github.com/users/chilimangoes/repos","events_url":"https://api.github.com/users/chilimangoes/events{/privacy}","received_events_url":"https://api.github.com/users/chilimangoes/received_events","type":"User","site_admin":false},"created_at":"2015-11-08T09:55:50Z","updated_at":"2015-11-08T09:55:50Z","body":"No problem. I didn't want to go into excess detail on things you already knew. But I don't mind explaining anything you have questions about. Also, I've studied RNNs quite a bit but don't have any practical experience with them, so this will be a learning experience for me too. And if you're interested in ML, I can't recommend enough Andrew Ng's [ML course](https://www.coursera.org/learn/machine-learning) on Courera. It's a pretty big time investment to actually complete the course and all the homework, but I took it a few years ago and it was one of my favorite courses on any subject ever. The lectures are also available to view if you just want a (relatively) quick primer on some of the subjects.\r\n\r\nTo your questions:\r\n\r\n1) By training, I mean the automated learning process that the algorithm goes through to adjust the parameters of the model (a neutral network in this case) to give good answers for the examples in our training data set. For example, we would feed the algorithm example phrase/symbol pairs from the data we collect in the format you requested above, along with an answer key for whether each pair represents a match or not. The network then calculates what it thinks the probability of a match is, compares its answer to the ground truth answer we gave it, and adjusts the parameters of the model to make its answer closer to the truth. Rinse and repeat for each example in our training data set. Also, generally speaking, the more training data you have, the better. Tens of thousands or millions of examples would be great, but not realistic for us to compile, so I threw out a couple ideas for how we might mitigate this issue by essentially pre-training the network on a large volume of unlabeled data (first option) or automatically generated data (second option). If you want me to go into more detail on either of those options, just ask.\r\n\r\n2) The bulk of processing work in most machine learning algorithms is linear algebra - manipulating and performing arithmetic on vectors and matrices of real numbers. There are a number of linear algebra libraries out there, written in C, with python wrappers. When I talk about training an RNN to produce similar output to say Levenstein, I mean we would encode the string as a series of real numbers, and then the RNN would essentially learn a mathematical transformation on those input numbers to produce an output number close to the number produced by Levenstein for that same input. Because it's all just basic number crunching, it could *potentially* be faster than a manually-coded algorithm that's doing lots of string and character manipulation - but we won't know until we try.\r\n\r\n3) As you probably guessed from (2), I mean a linear algebra vector. In python terms, a vector used in ML would typically be an array of real numbers and a matrix would either be an array of arrays of real numbers or a two-dimensional array of real numbers."}}