{"action":"opened","issue":{"url":"https://api.github.com/repos/juttle/redis-backend/issues/4","labels_url":"https://api.github.com/repos/juttle/redis-backend/issues/4/labels{/name}","comments_url":"https://api.github.com/repos/juttle/redis-backend/issues/4/comments","events_url":"https://api.github.com/repos/juttle/redis-backend/issues/4/events","html_url":"https://github.com/juttle/redis-backend/issues/4","id":124121714,"number":4,"title":"Juttle support for Redis","user":{"login":"dmehra","id":5817365,"avatar_url":"https://avatars.githubusercontent.com/u/5817365?v=3","gravatar_id":"","url":"https://api.github.com/users/dmehra","html_url":"https://github.com/dmehra","followers_url":"https://api.github.com/users/dmehra/followers","following_url":"https://api.github.com/users/dmehra/following{/other_user}","gists_url":"https://api.github.com/users/dmehra/gists{/gist_id}","starred_url":"https://api.github.com/users/dmehra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmehra/subscriptions","organizations_url":"https://api.github.com/users/dmehra/orgs","repos_url":"https://api.github.com/users/dmehra/repos","events_url":"https://api.github.com/users/dmehra/events{/privacy}","received_events_url":"https://api.github.com/users/dmehra/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-12-28T20:38:27Z","updated_at":"2015-12-28T20:38:27Z","closed_at":null,"body":"Writeup by @davidvgalbraith:\r\n\r\n[Redis](http://redis.io/) is a database-type thing that's pretty popular, so we want Juttle users to be able to write and read from it. Its worldview is pretty different from that of any other backend we've implemented support for so far, so let's see how we can reconcile them into something usable.\r\n\r\n## Redis\r\n\r\nRedis is a key-value store. So you give it commands like \"SET X 1\", and then you can do \"GET X\" and it'll give you 1. Actually, it'll give you \"1\", because Redis doesn't know about numbers. Values can be strings, lists of strings, HashMap<String, String>s, sets of strings, sorted sets of strings, bitmap strings, and HyperLogLogs, whatever those are. Each one of these value types has a special Redis method for setting and querying values of that type. Here's an example of a hash map:\r\n\r\n```\r\n> HMSET x a 1 b 2\r\nOK\r\n> HMGET x a\r\n1) \"1\"\r\n```\r\n\r\nSo what I did there was I set the value of the key \"x\" to be the hash map {a: 1, b: 2}, and then I queried the value of a for the hash map stored at x.\r\n\r\nThere is an [alphabet soup](http://redis.io/commands) of other commands you can use to analyze Redis data.\r\n\r\n## Juttle\r\n\r\n### Read\r\n\r\nRedis queries return either numbers or arrays of strings. Juttle needs to transform these into Javascript objects for the flowgraph to analyze. There doesn't seem to be much of a way to get any Juttley grammar-type abstraction out of Redis's query language, so let's just have readx redis take a mandatory \"-raw\" option specifying a Redis query string. We'll execute this query using the rawCallAsync method of a promisified [redis-fast-driver](https://www.npmjs.com/package/redis-fast-driver). Then we'll have to do some post-processing wherein we translate the resulting array of strings into a Javascript object. Let's check out the read-style Redis sources and how we can make sense of their output.\r\n\r\n```\r\nBITCOUNT/BITPOS/BRPOPLPUSH/CONFIG GET/DECR/DECRBY/DEL/ECHO/EXISTS/GETBIT/GETSET/GETRANGE/HDEL/HEXISTS/HGET/HINCRBY/HINCRBYFLOAT/HSETNX/HSTRLEN/INCR/INCRBY/INCRBYFLOAT/LASTSAVE/LINDEX/LINSERT/LLEN/LPOP/LPUSH/LPUSHX/LREM/MSETNX/PFADD/PFCOUNT/RANDOMKEY/RENAMENX/RPOP/RPOPLPUSH/RPUSH/RPUSHX/SADD/SCARD/SDIFFSTORE/SETBIT/SETRANGE/SISMEMBER/SMOVE/SREM/STRLEN/ZADD/ZCARD/ZCOUNT/ZINCRBY/ZINTERSTORE/ZLEXCOUNT/ZRANK/ZREM/ZREMRANGEBYLEX/ZREMRANGEBYRANK/ZREMRANGEBYSCORE/ZREVRANK/ZSCORE/ZUNIONSTORE\r\n```\r\n\r\nThese functions all perform some analysis on the value at a given key or keys and return a single integer or string as their result. Let's emit this result in a one-field object where the field is named \"value\" and the value is the result.\r\n\r\n#### GET\r\n\r\nGET takes a key with a string value and returns the value. We can turn it into a point mapping the key to the value.\r\n\r\n#### BLPOP/BRPOP\r\n\r\nThese guys are variadic functions that take as arguments any number of keys whose values are lists, and from the first such list that is nonempty they remove the last element and return a two-element array whose first element is the key that mapped to the nonempty list and whose second element is the value removed from that list. Let's emit this as a point with a key called \"list\" and a key called \"value\".\r\n\r\n#### CONFIG SET/CONFIG RESETSTAT/LSET/LTRIM/MSET/PFMERGE/RENAME/SELECT/SETEX/\r\n\r\nThese functions return the string \"OK\". Let's just swallow that and emit EOF if there was no error.\r\n\r\n#### HGETALL\r\n\r\nThis takes a key whose value is a hash and returns an array where for n = 0,1,2..., the 2n<sup>th</sup> item is the name of a key in the hash and the 2n+1<sup>th</sup> item is the value of that key. We have the technology to rebuild the hash from this and emit that.\r\n\r\n#### HKEYS/HVALS\r\n\r\nThese take a key whose value is a hash and return an array of the keys or of the values in the hash. It seems like the best way to make this into a Juttle object is to map each item in this array to True.\r\n\r\n#### KEYS\r\n\r\nLike HKEYS, but operates on the global namespace. There is no VALUES for the global namespace.\r\n\r\n#### MGET\r\n\r\nThis takes a list of keys whose values are strings and returns an array containing the values, or null if there is not a string value for a given key. We can map the nth input key to the nth output value to make an object.\r\n\r\n#### SDIFF/SINTER/LRANGE/SMEMBERS/SORT/SPOP/SRANDMEMBER/SUNION/ZRANGE/ZRANGEBYLEX/ZREVRANGEBYLEX/ZRANGEBYSCORE\r\n\r\nThese perform operations on lists or sets and return an array containing the elements of a new list or set. Let's turn this into an array of objects with a key called \"value\" whose value is the corresponding item in the array.\r\n\r\n### Write\r\n\r\nThere are several ways users might want to write data into Redis. Let's survey them.\r\n\r\n#### APPEND\r\n\r\nThe APPEND command takes a key whose value is a string and another string argument, and it concatenates the argument string to the end of the stored string.\r\n\r\n#### SET\r\n\r\nAs above, SET takes a key and a value string and sets the value as the value of the key.\r\n\r\n#### HSET\r\n\r\nHSET takes three strings, and it uses the first string as a key whose value is a hash mapping the second to the third.\r\n\r\n#### HMSET\r\n\r\nHMSET takes an odd number of strings, and it uses the first string as a key whose value is a hash mapping each even-indexed element to the next odd-indexed element, as in the example above.\r\n\r\n#### LINSERT\r\n\r\nAdds a string to a list at a specified position.\r\n\r\n#### LPUSH\r\n\r\nAdds a string to the beginning of a list.\r\n\r\n#### RPUSH\r\n\r\nAdds a string to the end of a list.\r\n\r\n#### MSET\r\n\r\nLike SET, but takes any number of key-value pairs to set.\r\n \r\nSo there are lots of ways to write, and they all depend on pulling strings off the points to form queries. So let's use string interpolation here. The unfortunate thing is that string interpolation in Juttle is somewhat limited, so we can't pass a string to interpolate as an option to the writex proc. Instead, we have to set it as a field on each point. So here's a command that writes the point `{a: 1, time: \"2015-11-25T00:13:08.862Z\"}` as a hash which is the value of the key \"test\":\r\n\r\n```\r\nemit -points [{\"a\":\"1\",\"id\":\"test\",\"time\":\"2015-11-25T00:13:08.862Z\"}] | put redis_write_command = \"HMSET ${id} a ${a} time ${time}\" | writex redis\r\n```\r\n\r\nAnd then to read the point back you'd do\r\n\r\n```\r\nreadx redis -raw \"HMGET test a time\"\r\n```\r\n\r\nAnd here's how one might implement timeseries:\r\n\r\n```\r\nemit -limit 100 | put series = 'my_series', value = count(), redis_write_command = \"APPEND ${series} '${time} ${value} ' | writex redis\r\n```\r\n\r\nand then\r\n\r\n```\r\nreadx redis -raw \"LRANGE my_series 0 100\"\r\n```\r\n\r\nwould give you 100 points like `{value: '2015-11-25T00:46:30.276Z 1'}`, which you could parse into the actual time and value fields. Not ideal, but it's about all I could do with the built-in Redis APIs."}}