{"action":"created","issue":{"url":"https://api.github.com/repos/CocoaPods/cocoapods-search/issues/8","labels_url":"https://api.github.com/repos/CocoaPods/cocoapods-search/issues/8/labels{/name}","comments_url":"https://api.github.com/repos/CocoaPods/cocoapods-search/issues/8/comments","events_url":"https://api.github.com/repos/CocoaPods/cocoapods-search/issues/8/events","html_url":"https://github.com/CocoaPods/cocoapods-search/issues/8","id":107551602,"number":8,"title":"Improving search performance while using --full flag","user":{"login":"manuyavuz","id":1665371,"avatar_url":"https://avatars.githubusercontent.com/u/1665371?v=3","gravatar_id":"","url":"https://api.github.com/users/manuyavuz","html_url":"https://github.com/manuyavuz","followers_url":"https://api.github.com/users/manuyavuz/followers","following_url":"https://api.github.com/users/manuyavuz/following{/other_user}","gists_url":"https://api.github.com/users/manuyavuz/gists{/gist_id}","starred_url":"https://api.github.com/users/manuyavuz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/manuyavuz/subscriptions","organizations_url":"https://api.github.com/users/manuyavuz/orgs","repos_url":"https://api.github.com/users/manuyavuz/repos","events_url":"https://api.github.com/users/manuyavuz/events{/privacy}","received_events_url":"https://api.github.com/users/manuyavuz/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":22,"created_at":"2015-09-21T16:40:59Z","updated_at":"2015-10-06T17:19:26Z","closed_at":null,"body":"Hi,\r\n\r\nI like this `pod search --full` command very much, however i think it has performance drawbacks which makes me hesitate while using it.\r\n\r\nFirst of all, it makes an update index operation whenever it's called, even if an index was generated before. This makes the command very slow to operate.\r\n\r\nSecondly, it generates a hash index such that keys are the pod names and values are strings which are created by concatenating podspec information such as summary, description and authors names. While searching for a given query, it traverses whole these value strings and perform a `gsub()` operation on them.\r\n\r\nI have the following proposal to improve search speed for full text search scenario:\r\n\r\n- Search index generation should be performed only when master repo or another private repo is updated (i.e. after performing `git pull --ff-only`). Operation should perform as a subprocess to not to block user for `pod install` and `pod update` commands.\r\n- Search index should have a different format which is more suitable for search scenarios. Specifically, index should be a hash index where keys are words used in whole spec repository, and values are names of the pods which contains given word inside it's spec information. This eliminates need to run `gsub()` on long strings repeatedly. We need to check query with keys only.\r\n\r\nI've actually implemented proposed scenario, and made some benchmarks. Results are as follows (___default___ means current implementation, ___fast___ means proposed implementation):\r\n\r\n\r\n# Generating Index From Scratch\r\n\r\n```\r\n    user            system      total     real\r\n    defaultIndex:   12.540000   1.130000  13.670000 ( 13.705714)\r\n    fastIndex:      12.800000   0.990000  13.790000 ( 13.797296)\r\n```\r\nThere is no important difference on indexing performance.\r\n\r\n\r\n# Searching 'afnetworking' Query \r\n\r\n```\r\n    user        system     total      real\r\n    default:    7.390000   0.370000   7.760000 (  7.760688)\r\n    fast:       0.470000   0.020000   0.490000 (  0.500742)\r\n```\r\n\r\nHere the proposed method offers a huge performance gain. This gain results from two improvements:\r\n\r\n1. Performing a proactive indexing policy eliminates the need to update index during command execution. We are sure that stored index is up-to-date.\r\n2. Evaluating query only for repo vocabulary (thanks to the new index format).\r\n\r\nTo measure the effects individually, I run one more benchmark, which disables default implementation's update index policy. In this case, it just gets index from the file and performs search on it. Here are the results:\r\n\r\n```\r\n    user                        system      total       real\r\n    defaultWithoutUpdateIndex:  2.140000    0.040000    2.180000 (  2.189329)\r\n    fast:                       0.510000    0.030000    0.540000 (  0.548586)\r\n```\r\n\r\nI think the gain of the new index format is pretty obvious. \r\n\r\n\r\n# Conclusion\r\n\r\nProposed method can remarkably improve search performance for full text search scenario. I will generate a PR containing the implementation, but I have a concern for this. Change contains modifications on three classes each belonging to different repositories (CocoaPods/CocoaPods, CocoaPods/Core, CocoaPods/cocoapods-search). How do you perform PR reviews for these cases? Should I create PR's for each repo and give links to each other? Or are there any other suitable methods?\r\n\r\nThanks so much for giving your time to this long issue description!\r\n"},"comment":{"url":"https://api.github.com/repos/CocoaPods/cocoapods-search/issues/comments/145932600","html_url":"https://github.com/CocoaPods/cocoapods-search/issues/8#issuecomment-145932600","issue_url":"https://api.github.com/repos/CocoaPods/cocoapods-search/issues/8","id":145932600,"user":{"login":"alloy","id":2320,"avatar_url":"https://avatars.githubusercontent.com/u/2320?v=3","gravatar_id":"","url":"https://api.github.com/users/alloy","html_url":"https://github.com/alloy","followers_url":"https://api.github.com/users/alloy/followers","following_url":"https://api.github.com/users/alloy/following{/other_user}","gists_url":"https://api.github.com/users/alloy/gists{/gist_id}","starred_url":"https://api.github.com/users/alloy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alloy/subscriptions","organizations_url":"https://api.github.com/users/alloy/orgs","repos_url":"https://api.github.com/users/alloy/repos","events_url":"https://api.github.com/users/alloy/events{/privacy}","received_events_url":"https://api.github.com/users/alloy/received_events","type":"User","site_admin":false},"created_at":"2015-10-06T17:19:26Z","updated_at":"2015-10-06T17:19:26Z","body":"Nah, it doesn’t sound like the added complexity is worth it, simpler is better :+1: "}}