{"action":"created","issue":{"url":"https://api.github.com/repos/SublimeText/LaTeXTools/issues/611","labels_url":"https://api.github.com/repos/SublimeText/LaTeXTools/issues/611/labels{/name}","comments_url":"https://api.github.com/repos/SublimeText/LaTeXTools/issues/611/comments","events_url":"https://api.github.com/repos/SublimeText/LaTeXTools/issues/611/events","html_url":"https://github.com/SublimeText/LaTeXTools/pull/611","id":119806854,"number":611,"title":"Utility functions: \"Analyze LaTeX document\" and \"Cache\"","user":{"login":"r-stein","id":12573621,"avatar_url":"https://avatars.githubusercontent.com/u/12573621?v=3","gravatar_id":"","url":"https://api.github.com/users/r-stein","html_url":"https://github.com/r-stein","followers_url":"https://api.github.com/users/r-stein/followers","following_url":"https://api.github.com/users/r-stein/following{/other_user}","gists_url":"https://api.github.com/users/r-stein/gists{/gist_id}","starred_url":"https://api.github.com/users/r-stein/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/r-stein/subscriptions","organizations_url":"https://api.github.com/users/r-stein/orgs","repos_url":"https://api.github.com/users/r-stein/repos","events_url":"https://api.github.com/users/r-stein/events{/privacy}","received_events_url":"https://api.github.com/users/r-stein/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-12-01T21:00:17Z","updated_at":"2015-12-01T21:47:54Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/SublimeText/LaTeXTools/pulls/611","html_url":"https://github.com/SublimeText/LaTeXTools/pull/611","diff_url":"https://github.com/SublimeText/LaTeXTools/pull/611.diff","patch_url":"https://github.com/SublimeText/LaTeXTools/pull/611.patch"},"body":"This pull request contains two functionalities. These functionalities don't effect the Enduser so far, but will (hopefully) yield new feature pull requests and improve the performance and soundness of existing features.\r\n\r\nIn short: a cache to improve performance and an analysis to retrieve a lot of information of the LaTeX document.\r\n\r\n\r\n# Cache\r\n\r\nAs the name suggests the cache can be used to cache the results of calculations (e.g. auto-completions).\r\nThese results can either stored local (relative to tex root) or global (in the sublime cache path).\r\n- __file position:__ There are two types of cache files:\r\n    + _local:_ creates a folder `.st_ltt_cache` in the same folder as the tex_root and stores the cache files there\r\n    + _global:_ stores the file in the SublimeText cache path (ST3) or User folder (ST2)\r\n- __clear the cache:__ the cache files are deleted in the `delete_temp_files`-command together with the temporary files.\r\n  Currently there is not other option to clear the cache, but I could implement an extra command or a periodic cleanup with a time interval.\r\n- __cache-able objects:__ the cache uses [pickle](https://docs.python.org/3.1/library/pickle.html). Hence everything can be cached, that can be dumped by pickle (this holds for many objects)\r\n- __usage:__ the functions `cache`, `read` and `write` can be used to access the cache.\r\n  A cache miss is indicated by a `CacheMiss`-Exception. The intended usage is demonstrated in this code snippet:\r\n  ``` python\r\n  try:\r\n      result = cache.read(tex_root, \"unique_name\")\r\n  except cache.CacheMiss:\r\n      result = calculate_result()\r\n      cache.write(tex_root, \"unique_name\", result)\r\n  # do something with result\r\n  ```\r\n\r\n  However the function `cache` is a short form for this:\r\n  ``` python\r\n  # retrieve the data from the cache and pass a pointer\r\n  # to a parameterless function/closure to calculate it\r\n  # if it is not available\r\n  result = cache.cache(tex_root, \"unique_name\", calculate_result)\r\n  ```\r\n\r\nAdvantages\r\n- usually faster results. Hence enables more complex *real-time* functionalities, e.g. auto-completion.\r\n- easy to use/small code overhead\r\n\r\nDisadvantages\r\n- creates a folder near the tex root (but the user can delete it with one keystroke)\r\n- results might be outdated (but the user can manually clear the cache)\r\n\r\n\r\n# Analysis\r\n\r\nThe analysis analyzes the `*.tex`-document and retrieves relevant information. To establish this goal the tex root and every included tex-file is analyzed. The major part of this analysis is to extract all `tex`-commands and create an object for an easy access their content. This is established by a regex-search over every document and creating an object from the result.\r\n\r\nThe result of the analysis contains:\r\n- the tex root\r\n- a map from each file name to its raw text content\r\n- a map from each file name to its content without comments\r\n- a __list of all commands__:\r\n  each command seaches with names:\r\n  ```\r\n  \\command[optargs]{args}[optargs2][optargs2a]{args2}[optargs3]{args3}\r\n  ```\r\n\r\n  for each of these names it contains the fields:\r\n    + 'name': the string in the part\r\n    + 'name'_region: the region of the part as a sublime region\r\n  <br/><br/>e.g. for the `args`-part these two attributes:\r\n    + `args`: the string in the args part\r\n    + `args_region`: the region of the args part as a sublime region in the buffer\r\n  <br/><br/>each command has in addition the attributes:\r\n    + `file_name`: the name of the file, in which the entry appers\r\n    + `text`: the full text of the entry, e.g. `\\command{args}`\r\n    + `start`: the start position in the buffer\r\n    + `end`: the end position in the buffer\r\n    + `region`: a sublime region containing the whole entry\r\n\r\n## Access the analysis\r\nThe access functions are:\r\n- `tex_root`: get the tex root\r\n- `content`: get the content (without comments) of a document\r\n- `raw_content`: get the content (with comments as in the file) of a document\r\n- `rowcol`: creates a rowcol function with the same behavior as the sublime `view.rowcol` function of a document\r\n- `commands`: returns all command entries in the file. You can apply flags for a prefiltering, e.g. remove all `\\begin{}` and `\\end{}` commands.\r\n- `filter_commands`: returns all command entries in the file after applying the filter. As in `commands` it is possible to use flags for a prefiltering.\r\n\r\nThe flags for `commands` and `filter_commands`:\r\n- `NO_BEGIN_END_COMMANDS`: exclude `\\begin{}` and `\\end{}` commands\r\n- `ONLY_COMMANDS_WITH_ARGS`: allows `\\com{args}` and `\\com{}` but not `\\com`\r\n- `ONLY_COMMANDS_WITH_ARG_CONTENT`: only allow `\\com{args}` not `\\com{}` or `\\com`\r\n- `ONLY_PREAMBLE`: only in the preamble, i.e. until `\\begin{document}`\r\n\r\nThe reason for the flag is create easy filters on the one side and improve performance on the other side.\r\n\r\n\r\n# Example Usage\r\n\r\nThese are a few examples to demonstrate the information, which is retrieved in the analysis.\r\n\r\n### Self-defined command completion\r\nRetrieve `\\newcommand`-commands in the document and insert them to the completion system similar to cwl completion.\r\nBasic principal, (see [here](https://github.com/r-stein/LaTeXTools/blob/ana_own_command_completion/latex_own_command_completions.py) for the full file):\r\n``` python\r\nana = analysis.get_analysis(tex_root)\r\nnewcommand_definitions = ana.filter_commands([\"newcommand\", \"renewcommand\"])\r\n# the defined command is in the args part\r\ncompletion_words = [c.args for c in newcommand_definitions]\r\n```\r\n\r\n![own_command_completion](https://cloud.githubusercontent.com/assets/12573621/11513807/7aecc68c-9875-11e5-89ce-7c002103dba0.gif)\r\n\r\n### Jump to command definition\r\nClick on a command and open its definition in Sublime Text (I don't really think this is useful, but it demonstrates the information retrieved by the analysis)\r\n\r\nretrieving information:\r\n``` python\r\n# the command is in the variable 'the_command'\r\nana = analysis.get_analysis(tex_root)\r\nnewcommand_definitions = ana.filter_commands([\"newcommand\", \"renewcommand\"])\r\n# search for the command definition entry\r\ntry:\r\n    entry = next(c for c in newcommand_definitions if c.args == the_command)\r\nexcept StopIteration:\r\n    return  # command not found\r\n\r\n# get the file name and the region\r\nentry_file = entry.file_name\r\nentry_region = entry.args_region\r\n\r\n# open the file with ST and change the selection to entry_region\r\n# ...\r\n```\r\n\r\n![jumpto_command_definition](https://cloud.githubusercontent.com/assets/12573621/11513771/4c9feae8-9875-11e5-8330-e08ab0902540.gif)\r\n\r\n### Whole document goto overlay\r\nParse all section commands (with their file and region) and create an overlay, which shows those in Sublime Text.\r\n\r\n![overlay_section](https://cloud.githubusercontent.com/assets/12573621/11513857/c33470fc-9875-11e5-9a12-0337606f9fe7.gif)\r\n\r\n\r\n### Other\r\nthis could also used to\r\n- easily find all generated .aux files for deletion\r\n- retrieve all labels (simplify ref/cite completion)\r\n- IDE like rename of commands\r\n- easy add used packages to cwl completion\r\n- *your idea*\r\n\r\n\r\nAdvantages:\r\n- central parsing engine -> packages with flow behavior (e.g. `subfile`, `ifthen`) can be considered in one file. The changes will propagate to every consuming function\r\n\r\nDisadvantages:\r\n- parsing the information might be slower, because the analysis also contains unused information (nevertheless through caching I would assume it will be faster in critical cases)\r\n\r\n## TODO list/additional feature ideas for future pull requests:\r\n- handle nested commands in the analysis (`\\firstcommand{\\secondcommand{}}`)\r\n- improvements wrt control flow, e.g. check if-conditions\r\n- automatically add the cache folder to the sublime ignored folders?\r\n- an automatic mechanics to clear the cache?\r\n- parse additional information (e.g. magic header)?\r\n\r\nI would be happy to get feedback, improvements and ideas. Especially from @ig0774.\r\nIf someone want to rewrite parts, he can post his branch in the comments and I will mostlikely merge it.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/SublimeText/LaTeXTools/issues/comments/161105985","html_url":"https://github.com/SublimeText/LaTeXTools/pull/611#issuecomment-161105985","issue_url":"https://api.github.com/repos/SublimeText/LaTeXTools/issues/611","id":161105985,"user":{"login":"KapSteR","id":4233736,"avatar_url":"https://avatars.githubusercontent.com/u/4233736?v=3","gravatar_id":"","url":"https://api.github.com/users/KapSteR","html_url":"https://github.com/KapSteR","followers_url":"https://api.github.com/users/KapSteR/followers","following_url":"https://api.github.com/users/KapSteR/following{/other_user}","gists_url":"https://api.github.com/users/KapSteR/gists{/gist_id}","starred_url":"https://api.github.com/users/KapSteR/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/KapSteR/subscriptions","organizations_url":"https://api.github.com/users/KapSteR/orgs","repos_url":"https://api.github.com/users/KapSteR/repos","events_url":"https://api.github.com/users/KapSteR/events{/privacy}","received_events_url":"https://api.github.com/users/KapSteR/received_events","type":"User","site_admin":false},"created_at":"2015-12-01T21:47:53Z","updated_at":"2015-12-01T21:47:53Z","body":"Amazing! This is some serious forward thinking... +1 "}}