{"action":"created","issue":{"url":"https://api.github.com/repos/manastech/crystal/issues/1824","labels_url":"https://api.github.com/repos/manastech/crystal/issues/1824/labels{/name}","comments_url":"https://api.github.com/repos/manastech/crystal/issues/1824/comments","events_url":"https://api.github.com/repos/manastech/crystal/issues/1824/events","html_url":"https://github.com/manastech/crystal/issues/1824","id":113397464,"number":1824,"title":"The next step","user":{"login":"asterite","id":209371,"avatar_url":"https://avatars.githubusercontent.com/u/209371?v=3","gravatar_id":"","url":"https://api.github.com/users/asterite","html_url":"https://github.com/asterite","followers_url":"https://api.github.com/users/asterite/followers","following_url":"https://api.github.com/users/asterite/following{/other_user}","gists_url":"https://api.github.com/users/asterite/gists{/gist_id}","starred_url":"https://api.github.com/users/asterite/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/asterite/subscriptions","organizations_url":"https://api.github.com/users/asterite/orgs","repos_url":"https://api.github.com/users/asterite/repos","events_url":"https://api.github.com/users/asterite/events{/privacy}","received_events_url":"https://api.github.com/users/asterite/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/manastech/crystal/labels/task","name":"task","color":"fbca04"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":24,"created_at":"2015-10-26T16:38:59Z","updated_at":"2015-10-27T13:51:25Z","closed_at":null,"body":"An always present worry that we have with the language is that it won't be useful for large projects. Since the global type inference algorithm always has to start from scratch, it's good if we find a way to do incremental or partial compilation: some way to reuse a previous compilation's result for the next one. That means that the first time you checkout a project it might take a few seconds to compile, but the next compiles should be fast, given small changes are made to the codebase.\r\n\r\nWe sat down and thought how it can be done with the current language, and we think (or at least we are strongly convinced) that there's no efficient way to do this. The language might need a change.\r\n\r\nWe thought of the minimum delta that could make this work. The conclusion is not a happy one, but it's also not the worse one could imagine:\r\n\r\n> When declaring a type (class, struct) you have to specify its instance variables and their types. The same applies to class variables and global variables.\r\n\r\nAfter some more brainstorming we think that the above rule can be relaxed in some cases: the compiler will try, for example, to infer instance variables types from the expressions assigned to them in the `initialize` methods, if these expressions are simple (literals and `new` calls). An example of where this simplification can be applied is the `CSV::Lexer` class from the standard library:\r\n\r\n```crystal\r\n# Current code will compile just fine\r\nclass CSV::Lexer\r\n  def initialize\r\n    @token = Token.new         # inferred to be Token\r\n    @buffer = MemoryIO.new     # inferred to be MemoryIO\r\n    @column_number = 1         # inferred to be Int32\r\n    @line_number = 1           # inferred to be Int32\r\n    @last_empty_column = false # inferred to be Bool\r\n  end\r\nend\r\n```\r\n\r\nAnother rule is defining a type restriciton of an initialize argument. For example:\r\n\r\n```crystal\r\nclass CSV::Builder\r\n  # @io inferred to be an IO\r\n  def initialize(@io : IO)\r\n    @first_cell_in_row = true # inferred to be Bool\r\n  end\r\nend\r\n```\r\n\r\nBut note that if in the above cases an instance variable is assigned another type in other methods, it will immediately be an error: if you want a union, define the instance variable type as such.\r\n\r\nIn other cases you'll inevitably have to write down the types:\r\n\r\n```crystal\r\nclass Markdown::Parser\r\n  # Needed because the expression assigned to it\r\n  # in the initialize is not simple (involves calls but it's not a `new` call)\r\n  @lines :: Array(String)\r\n\r\n  # @renderer is inferred to be a Markdown::Renderer\r\n  def initialize(text, @renderer : Markdown::Renderer)\r\n    @lines = text.lines.map &.chomp\r\n    @line = 0 # inferred to be an Int32\r\n  end\r\nend\r\n```\r\n\r\nThe current compiler works by taking into account all assignments to instance variables in the whole program. This has many (bad) consequences:\r\n\r\n* Because their type might change by mistake, if an errors follows it is hard to digest and hard to track. Also, the rule of \"if it's not assigned in the initialize then it's nilable\" is not very intuitive and, again, can happen by mistake. By having a clear rule of what are the types of instance variables these type-changing errors should happen less or at least will be easier to understood. Also: accessining a non-declared instance variable will immediately give an error, instead of propagating a nil value.\r\n* Because their type might change at any point in the program, they can affect the type of any method that uses them. This means that local type inference for a method can't be done: you always have to take into account the possibility of an instance variable changing after that method has been typed. If the compiler knows the types of all \"global\" data (instance and class variables, and global variables) local type inference becomes possible, leading to a faster compiler but most importantly allowing the possibility of reusing a previous compilation's result (for example by caching the \"dependencies\" (types, other methods) of each method).\r\n\r\nThe second point is the most important. If we want Crystal to be used for real, large projects, it's paramount that the compiler can handle such projects and that you don't have to wait 30 or 60 seconds between compiles. We prefer making this change rather than not doing it now and later get suck with a language that is useful for \"mostly toy\" projects.\r\n\r\nBecause this change has a huge impact on the overall algorithm of the compiler, we decided to take this opportunity to also rewrite the compiler \"from scratch\". This means: we'll rewrite it taking into account the current code and specs, but we won't reuse most of the code we have (well, the lexer and parser will be reused, the formatter, as well as some other bits). The current code, even though it's kind of efficient and more or less organized, has some flaws that are hard to fix without a rewrite (for example, we'd like to make an initial pass to gather all types so that we know which types have subclasses and so are \"virtual\"). And we'll do it taking into account the current bugs (which are about 45 and are more or less related to the same compiler/language flaws).\r\n\r\nThe good news is that we'll make sure that this new compiler is perfectly commented and documented: you'll be able to jump right into its source code and all phases and algorithms will be explained (promise!). Right now this is far from this case :-)\r\n\r\nLet's summarise the cons and pros of this change:\r\n\r\nPros:\r\n* Faster compile times (even without using a previou's compile result), and incremental compilation becomes possible.\r\n* Better error messages when incorrectly assining a wrong type to an instance variable, or when reading a non-existant one.\r\n* No need for a human to infer which instance variables are nilable: if they are not mentioned in the initialize there will be a type annotation saying its type.\r\n* We have some ideas on how to make possible Array(Object), Array(Reference) and basically using any type as a generic argument. You'll be able to have an Array(Object) and invoke `to_s` on each of them, for example. Because the compiler will know all types before the \"main\" code, virtual method lookups (that involve search a type's subclasses recursively) will be faster and can be cached because the type hierarchy will be fixed.\r\n\r\nCons:\r\n* It might be tedious to sometimes write these types. However, when you program, most of the time is spent in defining methods and in using types, not in defining the type of their instance variables. And you always know their type. A downside is that this might not go well with duck typing and you might have to define a module to act as an interface. But you can take that change to your benefit and define abstract methods on that module that tell you what has to be implemented (or you can leave the module empty: both approaches will work). Also: you'll be able to reopen a type and redefine the type of an instance variable, so mocks are still possible.\r\n* The promise of never having to write types down becomes false. However, this **is** already false: you have to do it for generic type arguments. And sometimes you do it in method arguments (although that's mostly for overloads, but it can also lead to better error messages and better documented code). And with the relaxation rules this won't have to be done all the time.\r\n\r\nRemember that types won't be mandatory in method arguments or return types, so duck typing still holds. This method:\r\n\r\n```crystal\r\ndef add(x, y)\r\n  x + y\r\nend\r\n```\r\n\r\nhas no type annotations and works on any type that defines a `#+(other)` method, so we belive Crystal is still unique and doesn't suddendly become an existing language.\r\n\r\nBecause the rewrite will take some time (and also because this next month we'll be busy with some specific work/life matters) you'll have to be patient. In the meantime the standard library can be grown and existing issues will be fixed unless they can only be fixed in this new compiler rewrite (some examples of issues we'll fix in the new compiler are #456, #718, #729, #846, #867, #916, #941, #962, #1346). After the rewrite you can use `crystal tool hierarchy` to port your code to the new version, or we might provide a migration tool."},"comment":{"url":"https://api.github.com/repos/manastech/crystal/issues/comments/151502544","html_url":"https://github.com/manastech/crystal/issues/1824#issuecomment-151502544","issue_url":"https://api.github.com/repos/manastech/crystal/issues/1824","id":151502544,"user":{"login":"ozra","id":563674,"avatar_url":"https://avatars.githubusercontent.com/u/563674?v=3","gravatar_id":"","url":"https://api.github.com/users/ozra","html_url":"https://github.com/ozra","followers_url":"https://api.github.com/users/ozra/followers","following_url":"https://api.github.com/users/ozra/following{/other_user}","gists_url":"https://api.github.com/users/ozra/gists{/gist_id}","starred_url":"https://api.github.com/users/ozra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ozra/subscriptions","organizations_url":"https://api.github.com/users/ozra/orgs","repos_url":"https://api.github.com/users/ozra/repos","events_url":"https://api.github.com/users/ozra/events{/privacy}","received_events_url":"https://api.github.com/users/ozra/received_events","type":"User","site_admin":false},"created_at":"2015-10-27T13:51:25Z","updated_at":"2015-10-27T13:51:25Z","body":"One idea for the inference cases like #1809 is of course that if a type is not instantiated - it could be ditched from all type unions - since it will not affect program, and hence it's \"could-be\" extension of method-types is a non matter. That's just one part of it all though.\r\nTrying to find a way for the \"no types\" crowd :)"}}