{"action":"created","issue":{"url":"https://api.github.com/repos/symphonycms/symphony-2/issues/2369","labels_url":"https://api.github.com/repos/symphonycms/symphony-2/issues/2369/labels{/name}","comments_url":"https://api.github.com/repos/symphonycms/symphony-2/issues/2369/comments","events_url":"https://api.github.com/repos/symphonycms/symphony-2/issues/2369/events","html_url":"https://github.com/symphonycms/symphony-2/issues/2369","id":60568757,"number":2369,"title":"Input field: control characters must be stripped","user":{"login":"michael-e","id":40072,"avatar_url":"https://avatars.githubusercontent.com/u/40072?v=3","gravatar_id":"","url":"https://api.github.com/users/michael-e","html_url":"https://github.com/michael-e","followers_url":"https://api.github.com/users/michael-e/followers","following_url":"https://api.github.com/users/michael-e/following{/other_user}","gists_url":"https://api.github.com/users/michael-e/gists{/gist_id}","starred_url":"https://api.github.com/users/michael-e/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michael-e/subscriptions","organizations_url":"https://api.github.com/users/michael-e/orgs","repos_url":"https://api.github.com/users/michael-e/repos","events_url":"https://api.github.com/users/michael-e/events{/privacy}","received_events_url":"https://api.github.com/users/michael-e/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/symphonycms/symphony-2/milestones/19","html_url":"https://github.com/symphonycms/symphony-2/milestones/3.0.0","labels_url":"https://api.github.com/repos/symphonycms/symphony-2/milestones/19/labels","id":736848,"number":19,"title":"3.0.0","description":"","creator":{"login":"brendo","id":69268,"avatar_url":"https://avatars.githubusercontent.com/u/69268?v=3","gravatar_id":"","url":"https://api.github.com/users/brendo","html_url":"https://github.com/brendo","followers_url":"https://api.github.com/users/brendo/followers","following_url":"https://api.github.com/users/brendo/following{/other_user}","gists_url":"https://api.github.com/users/brendo/gists{/gist_id}","starred_url":"https://api.github.com/users/brendo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brendo/subscriptions","organizations_url":"https://api.github.com/users/brendo/orgs","repos_url":"https://api.github.com/users/brendo/repos","events_url":"https://api.github.com/users/brendo/events{/privacy}","received_events_url":"https://api.github.com/users/brendo/received_events","type":"User","site_admin":false},"open_issues":8,"closed_issues":4,"state":"open","created_at":"2014-07-30T10:51:47Z","updated_at":"2015-03-11T11:31:34Z","due_on":"2015-05-13T14:00:00Z","closed_at":null},"comments":3,"created_at":"2015-03-10T21:01:42Z","updated_at":"2015-03-11T12:03:44Z","closed_at":null,"body":"As mentioned in #2024, control characters in Input fields will currently be saved to the database and also make their way to the handle of the field. Thus certain characters can even break pages.\r\n\r\n**This is Pandora's box.**\r\n\r\nIn XML 1.0, characters under U+0020 — except `TAB`, `CR` and `LF` — must not occur. But as I posted in #2024, there is the `DELETE` character as well (U+007F, 7-Bit-code 1111111, a historical control character). Many people say it's a control character and should be stripped, but at the same time it's a legal character in ISO/IEC 10646 which is part of the XML 1.0 specs. There are even more control codes that are valid in XML, see <http://en.wikipedia.org/wiki/C0_and_C1_control_codes>...\r\n\r\nAs you see, theoretically we must distinguish:\r\n\r\n* \"unwanted\" characters (for a special field)\r\n* illegal characters (in XML)\r\n\r\nThe second is simple, because the [XML specs tell us](http://www.w3.org/TR/REC-xml/#charsets):\r\n\r\n> … Consequently, XML processors must accept any character in the range specified for Char.\r\n>\r\n>\t\tChar ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]\r\n>\r\n>\t/* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */\r\n\r\nGiven this one might discuss if removal of illegal characters should be done immediately before passing characters to the XSLT layer, which means \"save whatever possible, but strip characters for XML purposes when reading data\". However, let's keep things simple. I strongly assume that the above definition of valid characters would be useful for any templating layer, not only XML/XSLT. So our **first decision** might be:\r\n\r\n* Shall we allow all characters from the XML specs? (Although, theoretically, there might still be problems with our current transliterations when building handles to be used in URLs.)\r\n* Or shall we try and strip even more/different characters that we assume to be useless or dangerous?\r\n\r\n### The MySQL issue\r\n\r\nBut: Even with the _character set_ set to UTF-8 (as Symphony does it), MySQL can not save any 4-byte characters. For more information on that matter please read:\r\n\r\n* <https://mathiasbynens.be/notes/mysql-utf8mb4>\r\n* <http://stackoverflow.com/questions/8491431/remove-4-byte-characters-from-a-utf-8-string/16902461>\r\n\r\nSo if we pass through only characters according to the XML specs, things can still break. (This happened to me once when I accidentally imported a 4-byte emoticon character from Facebook. Since then I strip 4-byte characters in the importer, before Symphony attempts to save these characters to the database. Of course, certain emoticons are lost.)\r\n\r\nSo if we talk about \"valid characters\" in Symphony, we come to the **second decision**:\r\n\r\n* Shall we change Symphony's database character set (again) to really work with all \"valid\" characters (in XML)?\r\n* Or shall we strip 4-byte characters?\r\n* Or shall we simply care for control characters and disregard the 4-byte issue?\r\n\r\n\r\n### The field issue\r\n\r\nNot all fields must behave in the same way, For example, a TAB (=control character) is an unwanted character in an Input field, but perfectly fine for a Textarea field. So our **third decision** may be:\r\n\r\n* Shall we build two stripping layers (one on the field level, one in the core routines for saving data)?\r\n* Or shall we do it all in every field?\r\n\r\nAs you see, the distinction at the top of my post actually raises the right questions. Which characters are invalid in XML? Which characters are \"invalid for a field\"? Is the latter the same as \"unwanted characters\"?\r\n\r\n\r\n### How to strip characters\r\n\r\nJust for reference, I am listing some REGEX examples .\r\n\r\nStrip control characters only, excluding DEL:\r\n\r\n```php\r\n$data = preg_replace( '/[\\x00-\\x1F]/', '', $data);\r\n```\r\n\r\nStrip control characters only, including DEL:\r\n\r\n```php\r\n$data = preg_replace('/[\\x00-\\x1F\\x7F]/', '', $data);\r\n```\r\n\r\nStrip 4-byte characters (see <http://stackoverflow.com/a/16902461>):\r\n\r\n```php\r\n$data = preg_replace('/[\\xF0-\\xF4].../s', '', $data);\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/symphonycms/symphony-2/issues/comments/78250556","html_url":"https://github.com/symphonycms/symphony-2/issues/2369#issuecomment-78250556","issue_url":"https://api.github.com/repos/symphonycms/symphony-2/issues/2369","id":78250556,"user":{"login":"michael-e","id":40072,"avatar_url":"https://avatars.githubusercontent.com/u/40072?v=3","gravatar_id":"","url":"https://api.github.com/users/michael-e","html_url":"https://github.com/michael-e","followers_url":"https://api.github.com/users/michael-e/followers","following_url":"https://api.github.com/users/michael-e/following{/other_user}","gists_url":"https://api.github.com/users/michael-e/gists{/gist_id}","starred_url":"https://api.github.com/users/michael-e/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michael-e/subscriptions","organizations_url":"https://api.github.com/users/michael-e/orgs","repos_url":"https://api.github.com/users/michael-e/repos","events_url":"https://api.github.com/users/michael-e/events{/privacy}","received_events_url":"https://api.github.com/users/michael-e/received_events","type":"User","site_admin":false},"created_at":"2015-03-11T12:03:43Z","updated_at":"2015-03-11T12:03:43Z","body":"> Because of that I think we should implement something similar to your second regular expression, all control characters and DEL\r\n\r\nAs I pointed out, this would be impractical for the Textarea field (which needs line breaks and `TAB`m which are control characters). This is why I suggested to do it on the field level—only the field can know what is \"sane\" (also because the field class generates a database table with a certain character set)."}}