{"action":"created","issue":{"url":"https://api.github.com/repos/Magnetme/task-scheduler/issues/1","labels_url":"https://api.github.com/repos/Magnetme/task-scheduler/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/Magnetme/task-scheduler/issues/1/comments","events_url":"https://api.github.com/repos/Magnetme/task-scheduler/issues/1/events","html_url":"https://github.com/Magnetme/task-scheduler/issues/1","id":103926122,"number":1,"title":"Initial Design ","user":{"login":"alexnederlof","id":992615,"avatar_url":"https://avatars.githubusercontent.com/u/992615?v=3","gravatar_id":"","url":"https://api.github.com/users/alexnederlof","html_url":"https://github.com/alexnederlof","followers_url":"https://api.github.com/users/alexnederlof/followers","following_url":"https://api.github.com/users/alexnederlof/following{/other_user}","gists_url":"https://api.github.com/users/alexnederlof/gists{/gist_id}","starred_url":"https://api.github.com/users/alexnederlof/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexnederlof/subscriptions","organizations_url":"https://api.github.com/users/alexnederlof/orgs","repos_url":"https://api.github.com/users/alexnederlof/repos","events_url":"https://api.github.com/users/alexnederlof/events{/privacy}","received_events_url":"https://api.github.com/users/alexnederlof/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-08-30T14:14:30Z","updated_at":"2015-09-07T09:34:25Z","closed_at":null,"body":"We want to schedule jobs periodically and incidentally, on a distributed system, and be fault-tolerant. Googling led to know re-usable existing solutions, so lets build our own, but only build the things we absolutely need, and make it awesome. \r\n\r\n# Prerequisites \r\n\r\n- I'll call the service/node executing a job the *executor*. I'll call this service *the scheduler*. The scheduler consists of redundant nodes.\r\n- I assume Kafka and Zookeeper always available.\r\n- When PostgreSQL goes down this can be seen as \"all schedulers go down\".\r\n\r\n\r\n# Problem:\r\n\r\n1. We want to run some jobs periodically \r\n2. It should not matter which application, node, server, internet-of-things-refrigerator, or whatever executes this job (e.g. location unaware) \r\n3. It should be fault-tolerant in the following scenarios:\r\n    1. The executing node aborts and lets the scheduled know\r\n    2. The executing node disappears and the scheduler needs to find out\r\n    3. The scheduler is taken down\r\n4. It should run on the Magnet.me stack (Too bad, no [Chronos](https://www.youtube.com/watch?v=FLqURrtS8IA) or [Oozie](http://oozie.apache.org/),  for us)\r\n5. It should allow incidental kickoff of jobs (e.g. let a developer run something form a UI) and scheduled repetitive jobs.\r\n\r\n# Design:\r\n\r\nA Java/Jersey/Angular Microservice that uses PostgreSQL as a storage, and Zookeeper for locking. \r\n\r\nThe postgres table might look like this:\r\n\r\nname  | type | goal\r\n-------- | ------ | ------\r\nid | number | Random ID of this scheduled job\r\njob_id | string | Id of the job. For example \"calculate_churn_metrics\"\r\ncreated_at | date | When the job was scheduled (either by a human or because it is recurring)\r\nexecute_at | date | When it should be executed\r\nrequest_started_at | date | The date a request for execution was sent.\r\nstarted_at | date | When it received ACK that the task is being executed\r\nstarted_by | string | The id sent by the node executing the task, to identify itself. \r\ncompleted_at | optional Date | when did it complete\r\nprogress | optional int [0...100] | if the job pings back it's progress, it is saved here\r\nlast_ping | optional date | The last time we heard from the job. Either by ping or by progress report. This is used to determine if a job is lost.\r\ncompletion_status | enum { completed, failed, aborted, lost } | The state at completion, if the job was completed\r\ncompletion_message | text | Arbitrary text field to ~~hack in everything we forgot in the initial design~~ report any message like \"Updated 10 items\". Only meant logging purposes.\r\n\r\nAnd another table for recurring jobs, that configures how often a job should run\r\n\r\nname  | type | goal\r\n-------- | ------ | ------\r\njob_id | string | Id of the job. For example \"calculate_churn_metrics\"\r\ninterval | enum { hourly, daily, weekly } | How often it should run\r\nparallel | boolean | should it await completion of its predecessor? \r\nat | optional int | The hour of the day, or day of the week you want to run it at (given that `at_interval=true` and `interval = {hourly, daily}`\r\nrate | boolean | if true, run jobs at a fixed interval (every hour). If false, run them [interval amount of time] after the last completion.\r\nreschedule_on_abort | boolean | \r\nreschedule_on_failure | boolean | \r\nreschedule_cool_down | int | Amount of time to wait before rescheduling\r\n\r\n# How it works\r\n\r\n### Scheduling jobs:\r\n\r\n1. The service checks Zookeeper (with Curator) to see if it is the master. \r\n2. If it is, it polls for any jobs that should be executed now (e.g. `run_at < NOW()`) \r\n3. For every job that needs to run\r\n    1. Send event over Kafka that requests its execution\r\n    2. Set `request_started_at = NOW()`\r\n4. Goto 1.\r\n\r\n### Polling lost jobs \r\n\r\nThis is not in the MVP, because it won't happen that often but:\r\n\r\n1. Check for jobs that have not pinged back in the last X amount of time\r\n2. Rescheduled them if necessary.\r\n\r\n### The rest of the work is listening to Kafka:\r\n\r\n1. Upon **job_started**\r\n    1. Set `started_at = NOW(); started_by = id` for the Job\r\n1. Upon**job_progress** or **job_ping**: Set `progress = progress` for the Job\r\n1. Upon **job_finished**: Set the completion_status of the finished job\r\n    1. If status `success` then nothing\r\n    1. If other status, then check if the job requires a re-run. If so, re-scheduled the job after cooldown. Otherwise no_op.\r\n\r\n\r\n# Disaster strikes.\r\n\r\n### When the executor goes down.\r\n\r\nThe server receives no pings and progress anymore. After a determined or derived amount of time we can re-schedule the service (this is not in the MVP).\r\n\r\n### When a scheduler node goes down\r\n\r\nAnother scheduler node becomes master and starts dispatching jobs. If all scheduler nodes go down it will simply \"catch up\" when they come back online. \r\n\r\nLater on we might implement a flag that disallows this behavior for certain jobs, in case it is \"time of the day\" sensitive. \r\n\r\n@Magnetme/developers "},"comment":{"url":"https://api.github.com/repos/Magnetme/task-scheduler/issues/comments/138251046","html_url":"https://github.com/Magnetme/task-scheduler/issues/1#issuecomment-138251046","issue_url":"https://api.github.com/repos/Magnetme/task-scheduler/issues/1","id":138251046,"user":{"login":"erwinvaneyk","id":2803726,"avatar_url":"https://avatars.githubusercontent.com/u/2803726?v=3","gravatar_id":"","url":"https://api.github.com/users/erwinvaneyk","html_url":"https://github.com/erwinvaneyk","followers_url":"https://api.github.com/users/erwinvaneyk/followers","following_url":"https://api.github.com/users/erwinvaneyk/following{/other_user}","gists_url":"https://api.github.com/users/erwinvaneyk/gists{/gist_id}","starred_url":"https://api.github.com/users/erwinvaneyk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/erwinvaneyk/subscriptions","organizations_url":"https://api.github.com/users/erwinvaneyk/orgs","repos_url":"https://api.github.com/users/erwinvaneyk/repos","events_url":"https://api.github.com/users/erwinvaneyk/events{/privacy}","received_events_url":"https://api.github.com/users/erwinvaneyk/received_events","type":"User","site_admin":false},"created_at":"2015-09-07T09:34:25Z","updated_at":"2015-09-07T09:34:25Z","body":"Ah okay, I read it as implied that it would run on the same database(server)."}}