{"action":"created","comment":{"url":"https://api.github.com/repos/quattor/quattor.github.com/pulls/comments/27866690","id":27866690,"diff_hunk":"@@ -0,0 +1,908 @@\n+---\n+layout: article\n+title: Bootstrapping and Troubleshooting SCDB\n+category: documentation\n+author: Michel Jouvin\n+---\n+\n+This page contains a step-by-step installation guide for Quattor SCDB, the second generation of Quattor configuration database, \n+and its associated deployment tools. It also contains some instructions to troubleshoot SCDB issues, in particular failure to deploy changes.\n+\n+***SCDB is now considered a legacy configuration database. If you start a new site you are encouraged to look at [Aquilon](/documentation/2012/10/31/install-aquilon.html), the new configuration\n+database, that provides more flexible workflows and an improved scalability.***\n+\n+SCDB relies on several underlying services not specific to Quattor, like Apache, Subversion, \n+DHCP, TFTP. Installing SCDB involves installing these services and configure them for Quattor use. As these services can be used for\n+other purposes than Quattor, the actual details of the service configurations can be potentially site-specific and instructions given here \n+may need to be adapted. For the sake of simplicity, the configuration steps for the underlying services assume that these services are \n+dedicated to Quattor.\n+\n+The instructions in this documentation assume the following conventions: \n+\n+* SVN server is `svn.example.org`. This clearly needs to be updated to reflect your host/domain name.\n+* The URL of the associated SVN repository is `http://svn.example.org/svn/quattor`: may be changed to suit your needs.\n+* TFTP server root directory is the default one, `/tftpboot`.\n+ \n+Having these services dedicated to Quattor is not at all a requirement. If you want to use an already existing instances of these services, \n+you'll need to tweak the proposed configuration to fit in your existing configuration. This may require information not present in this guide: \n+in this case refer to the service documentation. If you have no specific constraint and you aren't familiar with these services, \n+you are probably better to stick with a configuration as close as possible with the one used in examples.\n+\n+*Note: this documentation tries to be as generic as possible as far as OS versions are concerned, but the commands used in examples have \n+been tested only on Scientific Linux 6. \n+They may need to be modified when underlying services like SVN or DHCP are hosted on another platform.*\n+\n+**Important: when you will have succeeded in installing and configuring Quattor, it is recommended to add a profile for your deployment server so \n+that it will be managed by Quattor! This will help to keep it consistent and up-to-date.**\n+\n+\n+## OS Installation\n+\n+To proceed with the next steps, you need to install a basic RHEL or derivative \n+server configuration by any means appropriate to you (CD-Rom, Kickstart, imaging...).\n+Configure the network according to your site requirements and ensure that you have\n+a network connection with the outside world.\n+\n+### YUM Repositories\n+\n+After doing the base OS installation, configure the two following YUM repositories by adding\n+the following files to `/etc/yum.repos.d`:\n+\n+* EPEL\n+\n+  ```bash\n+  # Adapt to your OS version if not using EL6\n+  # You may also want to use another EPEL mirror closer to you\n+  cat >/etc/yum.repos.d/epel.repo <<EOF\n+  [sl6_epel]\n+  name=sl6_epel\n+  baseurl=http://mirrors.ircam.fr/pub/fedora/epel/6/x86_64/enabled=1\n+  gpgcheck=0\n+  EOF\n+  ```\n+\n+* Quattor\n+\n+  ```bash\n+  cat >/etc/yum.repos.d/quattor.repo <<EOF\n+  [quattor_release]\n+  name=Quattor current release\n+  baseurl=    http://yum.quattor.org/current\n+  gpgcheck=0\n+  EOF\n+  ```\n+\n+### Quattor RPMs\n+\n+Use YUM to install the RPM required on a Quattor deployment server:\n+\n+```bash\n+yum install aii-pxelinux ncm-ncd ncm-ccm\n+```\n+\n+## Web Server Installation\n+\n+The Quattor server needs to run a Web server to serve profiles,\n+kickstart configuration files and execute the CGI script at end of\n+a machine installation to change PXE boot to local disk. In addition, \n+you may want to use this Web server for serving YUM repositories if you\n+don't already have one for this purpose.\n+\n+Web server installation requires nothing specific, just the\n+configuration of a document root with enough space if you plan to\n+serve RPMs and the configuration of the CGI script. This Web server can be\n+shared with other usages and you can use an Apache virtual host.\n+\n+Apache is the recommended Web server (installation instructions in this documentation\n+refer to Apache) but nothing prevent you from using another one if you prefer and if\n+you know how to achieve the same configuration.\n+\n+### Apache Recommended Settings\n+\n+SCDB has no strong requirement concerning Apache configuration. It\n+needs two distinct URLs for two different purposes :\n+\n+ * Profiles: machine profiles are all stored under the same parent URL, for\n+   example `/var/www/quattor/profiles`. \n+   The files in this area are produced by the pan\n+   compiler when executing `ant deploy`.\n+ * Kickstart configuration files: this URL is used to store the\n+   Kickstart configuration file for each machine, for example `/var/www/quattor/ks`. These files are\n+   produced by `aii-shellfe --configure`.\n+   \n+In addition, you may want to configure the area (*directory*) used for serving\n+YUM repositories if you use the same server for this.\n+\n+Recommended setting for these areas are :\n+\n+ * Restrict access to profile and Kickstart configuration to IP\n+   adresses (or subnets) matching Quattor-managed machines, as these files may\n+   contain sensitive information like encrypted passwords or MySQL\n+   passwords (cleartext).   \n+ * Configure all these areas to ignore any `index.html` file and\n+   auto-indexing.\n+\n+Configuration for these areas is normally done by creating a file\n+`/etc/httpd/conf.d/quattor.conf` with directives like the following\n+ones (adjust to the path you are using):\n+\n+```\n+<Directory /var/www/quattor/profiles>\n+    Options Indexes\n+    DirectoryIndex VeryUnlikelyDirectoryIndex.none\n+    AllowOverride None\n+</Directory>\n+\n+<Directory /var/www/quattor/ks>\n+    Options Indexes\n+    DirectoryIndex VeryUnlikelyDirectoryIndex.none\n+    AllowOverride None\n+</Directory>\n+```\n+\n+*Note: if you are installing a new Apache server, don't forget to\n+edit `DocumentRoot` in `/etc/httpd/conf/httpd.conf` to reflect your\n+local configuration.*\n+\n+*Note: even though it is generally easy to rebuild it, it is recommended to backup\n+`quattor.conf` file.*\n+\n+## Subversion Server\n+\n+There is no need for a Subversion server dedicated to Quattor. SCDB is\n+just one repository from the Subversion point of view. If you already\n+run a Subversion server, you can skip the installation part and go\n+directly to the configuration part.\n+\n+### Subversion Installation and Configuration\n+\n+There are many possible installation options for a Subversion\n+server. Our recommendation is to install it as an Apache module. There is\n+no requirement for the Subversion server to run on a Linux machine,\n+even if it is the installation option documented here. You can even\n+choose to use a Subversion server outside of your site, if you think\n+the network connection is good enough.\n+\n+*Note: Quattor has no requirement regarding the name of the directory where SVN repository is located and \n+the URL associated with the repository. Even a repository non dedicated to SCDB (with already existing contents) can be used. \n+The examples in this page are based on a repository created as part of the SCDB installation and associated with URL \n+`http://svn.example.org/svn/quattor`. If you are not familiar with SVN and SVN management, you are probably better to use a configuration \n+as close as possible to the examples.*\n+\n+If you need to install a Subversion server, use YUM. Don't forget to install the Apache module which is\n+in a separate RPM. A typical SVN installation is done with the following command:\n+\n+```bash\n+yum install subversion mod_dav_svn\n+```\n+\n+After installing, you have to configure the Subversion server. Refer to Subversion [web site](https://subversion.apache.org) \n+for details. Configuration the SVN server typically involves:\n+* Creation of directory which will be the parent for SVN repositories (this example uses `/var/svn`):\n+ \n+  ```bash\n+  mkdir -p /var/svn\n+  ```\n+\n+* Create Subversion repository that will be used for the Quattor SCDB (don't forget to **backup this directory**):\n+  ```bash\n+  svnadmin create /var/svn/quattor\n+  # Repository must be owned by Apache account\n+  chown -R apache:apache /var/svn/quattor\n+  ```\n+\n+Apache SVN module configuration (`/etc/httpd/conf.d/subversion.conf`)\n+must be edited to configure URL used by SVN (examples in this page assume this is `/svn`)  \n+and bind it to the directory created at the previous step. A typical example, based on previously created repository \n+(adjust paths to reflect your configuration) is:\n+\n+```\n+<Location /svn>\n+   DAV svn\n+   SVNParentPath /var/svn\n+\n+   AuthzSVNAccessFile security/svn-repositories-access\n+\n+   AuthType        Basic\n+   AuthUserFile    security/passwd\n+   AuthGroupFile   security/group\n+   AuthName        \"SCDB SVN server\"\n+\n+   # Limit write permission to list of valid users.\n+   <LimitExcept GET PROPFIND OPTIONS REPORT>\n+      # Require SSL connection for password protection.\n+      # SSLRequireSSL\n+\n+      Require valid-user\n+   </LimitExcept>\n+</Location>\n+```\n+\n+To configure SVN authentication for the SCDB repository, you need to\n+create one or more accounts in `/etc/httpd/security/passwd`. You can\n+use `htpasswd` or `openssl passwd -apr1` to generate an encrypted\n+password.\n+\n+You also need to define SVN ACLs in\n+`/etc/httpd/security/svn-repositories-access`. To start, a typical file\n+is (it assumes the account you created is called `quattormgr`):\n+\n+```\n+[groups]\n+quattor-mgrs = quattormgr\n+\n+[/]\n+* = r\n+@quattor-mgrs = rw\n+```\n+\n+*Note: even though it is easily rebuilt, it is better to backup `subversion.conf` file and files in `/etc/httpd/security`.*\n+\n+### Subversion repository configuration\n+\n+For Quattor, you need to create a repository with the usual structure inside it (or inside a branch) :\n+\n+ * `trunk`: where you make the changes to your running configuration\n+ * `tags`: used by SCDB administration tool to do deployment\n+ * `branches` (optional): for alternative developments\n+\n+Assuming the repository has already been created (see previous step) and that its URL is `http://svn.example.org/quattor`, \n+one of the possible methods to initialise this structure is:\n+\n+```bash\n+mkdir /tmp/scdb\n+cd /tmp/scdb\n+mkdir trunk\n+mkdir tags\n+mkdir branches\n+svn import . http://svn.example.org/svn/quattor --message 'Quattor repository layout initialised'\n+```\n+\n+*Note: as stated previously, there is no need to dedicate a repository to SCDB and the same commands can be used to import a specific branch\n+in an existing repository.*\n+\n+### SCDB Initialisation\n+\n+If you followed this documentation and executed the step about [Subversion initialisation](#subversion-initialisation-and-configuration), you should have a Subversion \n+repository ready for use by SCDB. Be sure to use http/https to access the repository as  the standalone access won't work (this is a limitation of the build\n+script). \n+\n+To create your initial SCDB with the standard templates and the associated examples, follow the following steps:\n+\n+* In another directory, for example `scdb_src`, clone the `scdb` Git repository from [GitHub](http://github.com/quattor/scdb):\n+\n+  ```bash\n+  git clone https://github.com/quattor/scdb.git scdb_src\n+  ```\n+  \n+* Go into the directory containing the clone and execute the command below to create your initial configuration database with SCDB tools, \n+the template library and the site template examples. The configuration database will be createad in directory `scdb` (you may use another name\n+but the next steps in documentation assumes this name). The directory must not exist (use option `-F` to remove it if it exists):\n+\n+  ```bash\n+  cd scdb_src\n+  utils/scdb/create-vanilla-SCDB.sh -d ../scdb\n+  ```\n+\n+* Checkout the empty SCDB trunk created during [Subversion configuration](#subversion-repository-configuration) in the same directory:\n+\n+  ```bash\n+  cd ../scdb\n+  svn co http://svn.example.org/svn/quattor/trunk .\n+  ```\n+\n+* Configure the repository to ignore some files produced when compiling, using the following command :\n+\n+  ```bash\n+  cat > /tmp/ignore <<EOF\n+  .settings\n+  build\n+  build.saved\n+  deploy\n+  .project\n+  EOF\n+  svn propset svn:ignore -F /tmp/ignore .\n+  svn ci -m 'Add a .svnignore file appropriate for Quattor'\n+  ```\n+\n+* Add everything to your repository with command :\n+\n+  ```bash\n+  svn add *\n+  ```\n+\n+* Commit your vanilla SDCB with :\n+\n+  ```bash\n+  svn ci -m 'Create initial SCDB'\n+  ```\n+\n+\n+## SCDB Deployment Service Configuration\n+\n+This section explains how to install and configure the scripts\n+use by SCDB deployment service. These 2 scripts communicate using either SSH \n+if they are executed on\n+different machines or sudo if both run on the same machine. Depending on\n+the method used, you will need to configure one or the other.\n+\n+### Installation of deployment scripts\n+\n+SCDB deployment uses a pair of cooperating scripts:\n+\n+* SVN post-commit hook script: this script is launched when `ant deploy` creates a new deployment tag. This scripts run on the SVN server.\n+* Deployment script: this script is launched by the post-commit hook script to do the real work. It runs on the Quattor deployment server (which \n+may be different from the SVN server).\n+\n+Both scripts are distributed as part of [SCDB](https://github.com/quattor/scdb) (directory `src/hooks`). They need to be copied at the appropriate \n+location on the appropriate server:\n+\n+* The SVN hook script [src/hooks/post-commit.py](https://github.com/quattor/scdb/blob/master/src/hooks/post-commit.py) must be copied to the `hook` \n+directory of the SVN repository on the SVN server (using the naming convention of this guide, it would be `/var/svn/quattor/hooks`)\n+and given executable permission for Apache user. \n+* The other script, [src/hooks/build-tag.py](https://github.com/quattor/scdb/blob/master/src/hooks/build-tag.py) must be installed (root executable) \n+in `/root/quattor/scripts` on the Quattor server.\n+\n+Both scripts use the same configuration file, `/etc/quattor-deploy.conf`, see SCDB \n+[server-side customisations](#scdb-server-side-scripts) for details. The configuration file is **required** (one must exist on each machine \n+if you run SVN server and Quattor deployment server on different machines). \n+If you use a configuration based on suggested defaults and a SVN server and Quattor deployment server co-located on the same machine, a typical \n+configuration file is:\n+\n+```cfg\n+[scdb]\n+# Update to reflect your configuration\n+repository_url=http://svn.example.org/svn/quattor\n+\n+[build-tag]\n+# If not defined, defaults to parent of this build-tag script directory.\n+# If not starting with /, relative to parent of this build-tag script directory.\n+# Ensure the path exists.\n+svn_cache: /scratch/quattor-deployment/svncache\n+```\n+\n+*Note: there used to be a Shell/Perl version of these scripts. These scripts are now **obsolete**. The configuration file format of the \n+new scripts is **different**, even though the name may be the same. Please upgrade to the last version if you are still using the old scripts.*\n+\n+`build-tag.py` also requires the file `quattor.build.properties` to be created in the parent of the directory specified in `svn_cache` paramater \n+of the above configuration file. \n+A [template](https://github.com/quattor/scdb/blob/master/src/hooks/quattor.build.properties) of this file is available in SCDB distribution, in `src/hooks` directory. \n+It must be edited to reflect your local configuration.\n+\n+\n+### Creation of SSH Keys\n+\n+*Note: this step is necessary only if the SVN server and the Quattor deployment server are not the same machine. \n+Otherwise, it is recommended to use [#sudoConfiguration sudo] instead.* \n+\n+When `ssh` is used between the SVN server and the Quattor server, it is necessary to configure SSH keys \n+to allow a password-less ssh connection (there is no way to enter a password in the script).\n+To configure these keys, you need to:\n+\n+* Log under the deployment account on the Quattor deployment server, as specified in `/etc/quattor-deploy.conf` (`root` by default), and run the command:\n+\n+  ```bash\n+  ssh -b 2048\n+  ```\n+\n+* Log in to the Subversion server as the same user as your SVN server (generally Apache account), copy the file with extension `.pub` \n+generated in the previous step (to the directory `~/.ssh`)\n+* Try to ssh to the Quattor deployment server from the SVN server using the same account as \n+the one used by the deployment script. If it doesn't successfully to login without password, check your SSH configuration.\n+ \n+### sudo configuration\n+\n+*Note: this step is necessary only if you use `sudo` for the communication between the SVN server and the Quattor deployment server. \n+This requires both services to run on the same machine and is the recommended communication method in this situation. If both services \n+run on different machines, you need to use and configure [ssh](#creation-of-ssh-keys) instead.* \n+\n+To configure `sudo` for SCDB, use the `visudo` utility and enter the following lines in the `sudo ` configuration (be sure to \n+use the appropriate path and account in your configuration):\n+\n+```\n+Cmnd_Alias   QUATTORDEPLOY=/root/quattor/scripts/build-tag.py *\n+apache ALL = NOPASSWD: QUATTORDEPLOY\n+Defaults:apache !requiretty\n+```\n+\n+*Note: the last line is required:  if it is not given, `sudo` will silently fail.*\n+\n+## Site Configuration\n+\n+The parameters common to all machines of several clusters are defined in a `site`. The examples used\n+to initialise SCDB previously will have created a site called `example`. To create your own site, the easiest method\n+is to copy the `example` site and customise a few files, as described below.\n+\n+### RPM Repositories\n+\n+Pan templates are used to define the YUM repositories that will be used install packages. You can find in the \n+[cfg/sites/examples/repository](https://github.com/quattor/template-library-examples/tree/master/sites/example/repository)\n+URLs of YUM repositories: they should provide a good starting poing for upstream repositories (and they should have\n+been copied to your site if you created it from the examples) but feel free to update the URLs if you know sources\n+more appropriate to your site. You should keep the same template name as they are the name expected by default\n+by other parts of the template library.\n+ \n+When you have successfully \n+configured your first node, you may want to look at how to use \n+[YUM repository snapshots](/documentation/2014/03/24/spma-yum-migration.html#managing-yum-snapshots) with Quattor.\n+\n+*Note: if you use YUM snapshots that are not under the same path, be sure to add the required Apache configuration to serve them.*\n+\n+### Basic System Configuration\n+\n+Basic system configuration (network parameters, DNS servers, ...) are\n+grouped in templates `config.pan` and `global_variables.pan`  in the `site` subdirectory for\n+your site. Look at comments to understand what you need to modify.\n+\n+### Middleware Configuration\n+\n+If you wish to use Quattor to manage machines running the EMI/UMD middleware, you need to define your\n+grid site configuration. This is done in the template\n+`site/glite/config.pan` found in your site directory. Look at\n+comments to understand what you need to modify.\n+\n+*Note: do not be afraid of putting incorrect values in your gLite parameters, this can easily be refined later. As a general rule, keep the example values \n+if you don't understand how to change them.*\n+\n+\n+## Cluster Configuration\n+\n+After creating your site, you need to create your first cluster. You\n+can do this by copying the `clusters/grid/umd-3` directory (or the cluster examples relevant to your use case), removing all profiles from `clusters/grid/umd-3/profiles` and customising a\n+few templates.\n+\n+### Customise the associated site name\n+\n+In each cluster, the file `cluster.build.properties` defines the site (or even sites) associated with the cluster.\n+Edit the file coming from the examples and replace site `example` by your site name (this is the name of the directory\n+under `site`).\n+\n+*`cluster.build.properties` file allows to configure the Ant environment (properties) used to compile the cluster. One property that __must__ be\n+defined in this file is `cluster.pan.includes` which set the path used by the panc compiler to locate the templates.*\n+\n+### Hardware description\n+\n+You need to create a template describing the hardware configuration of\n+your machines. This is generally placed in the `hardware` sub-directory of\n+the site directory. Look at the examples, copy one with a configuration close to yours as a starting point. The name of the hardware template \n+is is internal and merely for convenience, it will be associated with a host name later. If your site has a convention for naming or tagging pieces of \n+hardware, it would make sense to use that here.\n+\n+*Note: our recommendation is to derive the names of machines from their location rather than their hostnames as hostnames may be migrated \n+to different hardware later on.*\n+\n+### Adding a machine to site/databases.pan\n+\n+Before being able to configure the machine, you need to create an\n+entry for the machine name in both variables of\n+[databases.pan](https://github.com/quattor/template-library-examples/blob/master/sites/example/site/databases.pan). First variable defines the address associated\n+with the machine name, second variable defines the hardware template\n+associated with the machine.\n+\n+In both variables, the key is the escaped host name. In the first case the value is an IP address, in the second it is the name of the template (relative to your site \n+name in sites directory) that you created in the previous step.\n+\n+### Creating a Machine Profile\n+\n+Copy an existing profile from examples corresponding to the machine type\n+you want to create.\n+\n+*Note: when you have successfully configured a site BDII, the next recommended steps are a CE and a single WN.*\n+\n+### Compiling and deploying the first profile\n+\n+At this point you should be able to compile and deploy your first profile! With SCDB, this is \n+done with the command `ant deploy`. To execute successfully this command, you must first commit\n+your changes to Subversion. Good practice is to take the following steps to avoid committing\n+broken changes:\n+\n+```bash\n+external/ant/bin/ant\n+# If/when compilation succeeded\n+svn commit -m 'Initial site configuration + first profile added'\n+external/ant/bin/ant deploy\n+```\n+\n+*Note: you can also use `ant deploy` for the first step. Deployment will be blocked until the changes are committed.*\n+\n+If the `ant deploy` succeeded, you should find a profile file created on the Quattor deployment server for the host\n+in the directory that you configured to store the profiles during Apache configuration (`/var/www/quattor/profiles` \n+based on the example provided). The profile name starts with the hostname, followed by the extension `.json`\n+(or `.xml` if you chose to use XML profiles). If you don't find a profile in this directory after a successful \n+execution of `ant deploy`, follow the instructions at the end of this page to troubleshoot the deployment\n+scripts.\n+\n+\n+## Configuration of Installation Service\n+\n+AII, the Quattor service for automated installations, has a front-end, `aii-shellfe`, which\n+is responsible for configuring:\n+\n+* DHCP and TFTP for PXE boot\n+* Kickstart for driving the installation process\n+\n+In addition, it uses a CGI script during the machine installation to change the boot from\n+the installer to the local disk.\n+\n+\n+### DHCP configuraton\n+\n+Install the DHCP server with the following YUM command:\n+\n+```bash\n+yum install dhcp\n+```\n+\n+If the DHCP server will be used for Quattor-managed nodes only, a basic DHCP configuration (`/etc/dhcpd.conf`) is \n+(update hosts/domain name/addresses appropriately):\n+\n+```\n+# DHCP server configuration\n+\n+authoritative;\n+allow bootp;\n+ddns-update-style none;\n+\n+# Edit to reflect your DNS domain name and name servers (a comma-separated list is allowed)\n+option domain-name \"example.org\";\n+option domain-name-servers dns.example.org;\n+option netbios-node-type 2;\n+\n+# Update to reflect your IP subnet\n+subnet 123.456.789.0 netmask 255.255.255.0 {\n+\n+  # Parameters for the installation via PXE using pxelinux\n+  filename                           \"quattor/pxelinux.0\";\n+  option vendor-encapsulated-options 01:04:00:00:00:00:ff;\n+\n+  # This is required on RHEL/SL/SLC/CentOS 5.X\n+  next-server  tftp.server.example.org;\n+\n+  # This is now a required line in DHCP configuration.  This\n+  # option gives the behavior of the previous versions.\n+  ddns-update-style ad-hoc;\n+\n+  option routers 123.456.789.1;\n+}\n+```\n+\n+If you want to share the DHCP server between Quattor and non Quattor usage, it's\n+recommended to move the last part (`subnet...`) into a separate\n+file, for example `/etc/dhcpd/quattor.conf`, and to include it in the main\n+configuration file by:\n+\n+```\n+include \"/etc/dhcpd/quattor.conf\";\n+```\n+\n+See `man dhcpd` and `man dhcpd.conf` for details about DHCP server\n+configuration, in particular to support multiple subnets and other\n+advanced features.\n+\n+### TFTPD configuration\n+\n+Install the TFTP server with the following YUM command:\n+\n+```bash\n+yum install tftp-server\n+```\n+\n+The TFTP server is run by `xinetd`. In the default configuration provided by the OS installation, it is generally\n+disabled. Enable it by editing `/etc/xinetd.d/tftp`, modifying `disable` parameter from `yes` to `no`. \n+\n+Option `server_args` allows to define the TFTP root directory. By default, it is `/tftpboot`. If you'd like to use another location you need \n+to update this option. In this case, you'll also need to define the variable `AII_NBP_DIR` to this directory during next step.\n+\n+\n+### aii-shellfe configuration\n+\n+This involves 2 separate steps :\n+\n+* customisation of `/etc/aii/aii*.conf` files\n+* customisation of AII-related variables in templates\n+\n+There are 2 AII configuration files that need to be customised to reflect your site configuration:\n+\n+* `/etc/aii/aii-shellfe.conf`: a typical file is as follows, with paths edited to reflect your configuration.\n+\n+  ```\n+  # URL corresponding to compiled profiles generated by ant deploy\n+  cdburl=http://quattor.example.org/profiles\n+  # use_fqdn must be set to true\n+  use_fqdn=true\n+  # Directory where to write Kickstart configuration files produced by aii-shellfe.\n+  # Must match the directory served by URL defined in template variable QUATTOR_PROFILE_URL.\n+  osinstalldir = /var/www/quattor/ks\n+  # Directory where pxelinux.cfg is installed. Default is the recommended locaiton.\n+  nbpdir = /tftpboot/quattor/pxelinux.cfg\n+  ```\n+\n+* `/etc/aii/aii-dhcp.conf`: check that `dhcpconf` and `restartcmd` commands match your configuration and edit them if necessary. \n+The file referred by `dhcpconf` must be writable from the AII server. `restartcmd` may execute a command on a remote machine \n+through appropriate means.\n+\n+In addition, there are a few variables to customise in site templates to reflect\n+your Quattor and AII configuration, mainly :\n+\n+* `QUATTOR_PROFILE_URL` : URL to use to download machine profiles.\n+* `AII_OSINSTALL_SRV` : Name of the Web server serving OS rpms for the initial install (can be same as main repository server).\n+* `AII_KS_SRV` : Name of the Web server serving kickstart files. Defaults to `AII_OSINSTALL_SRV`\n+* `AII_ACKSRV` : Name of the Web server to use for the\n+   post-installation CGI. Defaults to `AII_OSINSTALL_SRV`\n+* `AII_ACKCGI` : post-installation CGI URL. Defaults to\n+   `/cgi-bin/aii-installack.cgi`.\n+* `AII_NBP_DIR` : path of the directory containing the PXE configuration for each machine. Defaults: `/tftpboot/quattor/pxelinux.cfg`.\n+\n+These variables are generally defined site-wide, in the template `global_variables.pan` located in site directory. Look at\n+provided examples, in SCDB distribution.\n+\n+### Downloading the OS distribution's images\n+\n+For every OS version you plan to deploy with Quattor, you should mirror the upstream YUM repository with appropriate tools if you don't have it already at your\n+site. Note that an OS-related YUM repository\n+is not made of single directory, so `wget` will generally hardly make a usable mirror. `reposync` is probably a better option. You may also look at \n+[sync_yum_repos](https://github.com/quattor/scdb/blob/master/utils/yum/sync_yum_repos) in SCDB which is a wrapper above `reposync` that helps to automatize\n+repository mirroring.\n+\n+Once you have the OS repository mirror available, you need to add the boot files used for the PXE installation to the AII server. They consist in 2 files (`vmlinuz`and `initrd.img`) \n+generally located in `images/pxeboot` directory of the distribution. You need to copy these files in a directory specific to the OS version/architecture that must be created in the \n+same directory as the `pxelinux.cfg` directory under the TFTP server root (see variable `AII_NBP_DIR` configured in  previous step for the exact path).\n+The directory name must have the usual Quattor format for OS version/architecture with all `-` replaced by `_` (e.g. sl530_x86_64).\n+\n+### Post-installation CGI Script\n+\n+At the end of a machine installation, as part of the Kickstart\n+post-intallation script, a CGI script is executed on the Quattor\n+server to update its PXE configuration in order to boot\n+from local disk at the next reboot. \n+\n+*Note: with Quattor, it is recommended to define the PXE-enabled network interface as the first boot\n+device in the BIOS. Quattor will properly configure PXE to boot either from the local disk or from the installer\n+depending of what has been requested with `aii-shellfe`command.*\n+\n+This script, [aii-installack.cgi](https://github.com/quattor/scdb/blob/master/src/cgis/aii-installack.cgi), \n+can be found in SCDB directory `src/cgis`. It must be placed on the Web server running on the Quattor\n+server, in the directory for CGIs.\n+\n+The apache server must be able to run that script as root. The recommended configuration is to\n+have `sudo` installed and use `visudo` to add the following to\n+`/etc/sudoers`:\n+ \n+```\n+Cmnd_Alias   AIIACKCGI=/usr/sbin/aii-shellfe\n+apache ALL = NOPASSWD: AIIACKCGI\n+```\n+\n+\n+### Testing the installation service\n+\n+On the AII server, run the following commands to test the installation service with the profile built\n+at the end of the [cluster configuration](#compiling-and-deploying-the-first-profile):\n+\n+* Creation of Kickstart configuration file for the machine: after a successful execution of the command, you should\n+find a file whose name is the hostname and whose extension is `.ks` in the directory referred to by `osinstalldir` in\n+`/etc/aii/aii-shellfe.conf`.\n+\n+  ```bash\n+  aii-shellfe --configure your.machine.domain\n+  ```\n+\n+* Update of DHCP and PXE to have the host installed at next boot. Then reboot your machine and check\n+that it installs properly.\n+\n+  ```bash\n+  aii-shellfe --install your.machine.domain\n+  ```\n+\n+*Note: you can ignore the warning returned by `aii-shellfe` about `base_url` is not defined in configuration` if you get it.*\n+\n+If the installation of your first machine completes successfully, you are done with Quattor SCDB installation!\n+\n+## SCDB Server-Side Scripts\n+\n+This section describes the two scripts used by SCDB to trig a Quattor deployment from the SVN server, how to customize them and how to\n+troubleshoot them. Refer to the previous sections for information on how to install them. These scripts are executed as a result of\n+entering command `ant deploy`.\n+\n+### Subversion Hook Script\n+\n+This script is triggered at the end of every commit: it is called a *post-commit* hook. It does nothing, except if called as a result of the command\n+`ant deploy`, when creating the deployment tag. Note that, as it is a *post-commit* hook, the commit succeeds even though the script encounters\n+an error. The main role of this script to launch the [deployment script](#deployment-script) after doing a few checks.\n+\n+It must be installed on the Subversion server, inside the `hooks` directory of the SCDB repository (the repository itself, not a working copy). \n+For this, you need a write access to the directory containing the repository on the Subversion server.\n+This script *must be* named `post-commit` and be executable by the account the SVN server is running under (typically the Apache account). \n+This script is released as part of [SCDB](https://github.com/quattor/scdb/blob/master/src/hooks/post-commit.py).\n+\n+This script requires a configuration file, `/etc/quattor-deploy.conf` (common with the [deployment script](#deployment-script) when running on the same machine).\n+All the configuration options supported and their default values are defined at the beginning of the script. The configuration file has several sections. \n+One of them, `[scdb]`, is common to the hook script and the deployment script. The main options available and their default values are:\n+\n+```\n+[post-commit]\n+# Script launched by the script to actually do the deployment\n+deploy_script : /root/quattor/scripts/build-tag.py\n+# Name of the deployment server where to run the deploy_script. Used only with ssh.\n+# This can be a space-separated list (not yet implemented, see https://trac.lal.in2p3.fr/LCGQWG/ticket/46).\n+#deploy_server : quattorsrv.example.org\n+# Userid to use to run deploy_script\n+deploy_user : root\n+# notify_xxx are used to configure email notification in case of errors.\n+# If notif_from or notif_to is undefined, email notification is disabled\n+notif_mailer : localhost\n+#notif_from=Quattor Deployment <noreply@lal.in2p3.fr>\n+#notif_to=jouvin@lal.in2p3.fr\n+notif_subject_prefix : [Quattor-Deploy]\n+notif_subject : Failed to deploy revision %s of SCDB configuration\n+# Default should be appropriate. Set to false if your client doesn't handle properly returned output.\n+# When set to false, no message is printed on stdout, except if verbose is > 0.\n+report_error_to_svn: yes\n+# When false, ssh is used instead. This requires deploy_server to be defined too.\n+use_sudo : yes\n+# Log operations in /tmp/quattor-post-commit.log\n+verbose: 0\n+\n+[ssh]\n+cmd: /usr/bin/ssh\n+options: -o PasswordAuthentication=no\n+\n+[sudo]\n+cmd: /usr/bin/sudo\n+options: -H\n+\n+[scdb]\n+# URL associated with the repository root\n+#repository_url: http://svn.example.com/scdb\n+# Branch where to create SCDB deployment tags\n+# Keep consistent with quattor.build.properties if not using default values.\n+tags_branch: /tags\n+# Branch corresponding to SCDB trunk (only branch allowed to deploy)\n+# Keep consistent with quattor.build.properties if not using default values.\n+trunk_branch: /trunk\n+```\n+\n+The only required options for which no default value is provided is `repository_url` in `[scdb]` section. It must match the root URL \n+of your SVN repository (as returned by `svn info`).\n+\n+*Note on using `sudo`: recent versions of `sudo` disable the use of `sudo` without a tty by default. This **must** be change in order to use `sudo` \n+in the context of the deployement script. To to this, run `visudo` and comment out the line `Defaults    requiretty`.*\n+\n+### Deployment Script\n+\n+This script is launched by the [hook script](#subversion-hook-script) and does the real work by updating the working copy of the SCDB repository on the Quattor server to the tag to created \n+by `ant deploy` and by calling the appropriate `ant` tasks.\n+\n+It needs to be installed on the Quattor server and will be called through sudo or ssh by the hook script (you need to configure a SSH key without password between \n+your Subversion server and your Quattor server if ssh is used). The default location to install the script is `/root/quattor/scripts` but you can put it wherever you want \n+as long as you update the hook script configuration accordingly (see previous section). The source of this script is in [SCDB](https://github.com/quattor/scdb/blob/master/src/hooks/build-tag.py).\n+\n+This script requires a configuration file, `/etc/quattor-deploy.conf` (common with the [hook script](#subversion-hook-script) when running on the same machine). \n+All the configuration options supported and their default values are defined at the beginning of the script. The configuration file has several sections. One of them, `[scdb]`, \n+is common to the hook script and the deployment script. The main options available and their default values are:\n+\n+```\n+[build-tag]\n+# If not starting with /, relative to directory specified by option svn_cache\n+ant_cmd: external/ant/bin/ant\n+# ant options (passed through env variable ANT_OPTS)\n+#ant_opts: -Xmx2048M\n+# ant stdout: allow to redirect ant output to a file for debugging purposes (default is PIPE)\n+# Default changed to a file because of problem in subprocess module if the \n+# output is very large (Python 2.4) leading to the parent/child communication\n+# to hang.\n+ant_stdout: /tmp/ant-deploy-notify.log\n+#ant_stdout: PIPE\n+# ant target to do the deployment. Default should be appropriate.\n+ant_target: deploy.and.notify\n+# If not starting with /, relative to /usr/java.\n+java_version: latest\n+# If not starting with /, relative to parent of this directory script\n+svn_cache: svncache\n+# Number of retries for SVN switch to new tag in case of error\n+switch_retry_count: 1\n+# Verbosity level\n+verbose: 0\n+\n+[scdb]\n+# URL associated with the repository root. Required parameter without default value.\n+#repository_url: http://svn.example.com/scdb\n+# Branch where to create SCDB deployment tags\n+# Keep consistent with quattor.build.properties if not using default values.\n+tags_branch: /tags\n+# Branch corresponding to SCDB trunk (only branch allowed to deploy)\n+# Keep consistent with quattor.build.properties if not using default values.\n+trunk_branch: /trunk\n+```\n+\n+The only required options for which no default value is provided is `repository_url` in `[scdb]` section. It must match the root URL \n+of your SVN repository (as returned by `svn info`).\n+\n+This script needs to find a file [quattor.build.properties](https://github.com/quattor/scdb/blob/master/src/hooks/quattor.build.properties) in the parent directory of\n+the SCDB local cache (by default, the parent directory for this script location, explicitly defined by `svn_cache` in the script configuration file).\n+\n+\n+## Troubleshooting Server-side Scripts\n+\n+In addition to the specific problems mentionned below, you may look at the [SCDB FAQ](https://trac.lal.in2p3.fr/Quattor/wiki/Doc/SCDB/FAQ).\n+\n+If there is a problem during deployment, after entering command `ant deploy`, the command should return an explicit error message. In addition,\n+it is possible to optionally define a list of people who will receive an email when an error occurs. This is done with\n+`notify_to` configuration option of the [hook script](#subversion-hook-script).\n+If the `notify_to` people don't receive an email about the deployment failure, \n+check your SMTP configuration and your notification parameters. \n+\n+*Note : a common error message is about the deployment script already running: as stated in the message, it is necessary to retry later\n+as only one deployment can occur at any time.*\n+\n+To troubleshoot server side configuration problems, it is necessary to execute manually the server scripts. When doing this, start by\n+testing `build-tag.py` and then the hook script, as explained in the following sections.\n+\n+### Troubleshooting deployment script\n+\n+**To troubleshoot the deployment script, `build-tag.py`, you must log on the Quattor server as the user configured to run deployment script (generally `root`).**\n+\n+The step-by-step procedure to troubleshoot the deployment script is (commands given assume that the script is installed in `/root/quattor/scripts`):\n+\n+1. Look in `/etc/quattor-deploy.conf` the location of the SVN cache on the Quattor server. If it is not defined, the default location is `/root/quattor/svncache`. \n+In the following steps, replace this default location by whatever is appropriate at your site.\n+1. Check the last tag checked out with the following command:\n+\n+  ```\n+  svn info /root/quattor/svncache\n+  ---> look at the current tag name, everything after /tags/ in the URL (without the leading /)\n+  ```\n+\n+1. Check the tag has been successfully checked out. If the following command updates anything or returns an error, it means the script attempt to switch to this tag \n+was unsuccessful. If the command fails again, retry it until it is successful. Failure to switch to a tag is generally related to a Subversion server problem, \n+not to Quattor itself : check Subversion server logs for more information.\n+\n+  ```\n+  svn update\n+  ```\n+\n+1. Try to redeploy the same tag, enabling verbose output, with the following command:\n+\n+  ```\n+  /root/quattor/scripts/build-tag.py --debug TAG-VALUE\n+  ```\n+\n+If the SVN cache is empty (not recognized as a valid SVN working copy by `svn info`), you need to identify last tag in `tags` branch of the Quattor SCDB repository using:\n+\n+```\n+# Replace REPOSITORY_URL by `repository_url` value in the configuration file.\n+# YEAR is the four digit year number and month the two digit month number.\n+svn ls REPOSITORY_URL/tags/YEAR/MONTH\n+```\n+\n+*Note: if you are still using the obsolete deployment script, `build-tag.pl`, these troubleshooting steps don't apply. You need to upgrade to current scripts first.*\n+\n+### Troubleshooting SVN hook script\n+\n+**Most of the deployment problems, after initial configuration, are related to [deployment script](#troubleshooting-deployment-script ). As the hook \n+script mainly launches the deployment script, it is important to complete checks described in previous section before troubleshooting the SVN hook script.**\n+\n+The main cause of problems in the hook script is related to launching the deployment script from the hook script. The reason for the problems and the troubleshooting \n+steps are different whether you use `sudo` or `ssh` to run the deployment script on the Quattor deployment server.\n+ * `ssh`: the main reason is invalid SSH keys for communication between the SVN server and the Quattor server or an invalid configuration. ","path":"_posts/2015-03-25-SCDB-installation.md","position":887,"original_position":887,"commit_id":"38edbba8c4a372e48d65b67b3954a4e59a2ec4e8","original_commit_id":"38edbba8c4a372e48d65b67b3954a4e59a2ec4e8","user":{"login":"jrha","id":323309,"avatar_url":"https://avatars.githubusercontent.com/u/323309?v=3","gravatar_id":"","url":"https://api.github.com/users/jrha","html_url":"https://github.com/jrha","followers_url":"https://api.github.com/users/jrha/followers","following_url":"https://api.github.com/users/jrha/following{/other_user}","gists_url":"https://api.github.com/users/jrha/gists{/gist_id}","starred_url":"https://api.github.com/users/jrha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jrha/subscriptions","organizations_url":"https://api.github.com/users/jrha/orgs","repos_url":"https://api.github.com/users/jrha/repos","events_url":"https://api.github.com/users/jrha/events{/privacy}","received_events_url":"https://api.github.com/users/jrha/received_events","type":"User","site_admin":false},"body":"the **most common** reason","created_at":"2015-04-07T10:30:33Z","updated_at":"2015-04-07T10:30:33Z","html_url":"https://github.com/quattor/quattor.github.com/pull/137#discussion_r27866690","pull_request_url":"https://api.github.com/repos/quattor/quattor.github.com/pulls/137","_links":{"self":{"href":"https://api.github.com/repos/quattor/quattor.github.com/pulls/comments/27866690"},"html":{"href":"https://github.com/quattor/quattor.github.com/pull/137#discussion_r27866690"},"pull_request":{"href":"https://api.github.com/repos/quattor/quattor.github.com/pulls/137"}}},"pull_request":{"url":"https://api.github.com/repos/quattor/quattor.github.com/pulls/137","id":31979225,"html_url":"https://github.com/quattor/quattor.github.com/pull/137","diff_url":"https://github.com/quattor/quattor.github.com/pull/137.diff","patch_url":"https://github.com/quattor/quattor.github.com/pull/137.patch","issue_url":"https://api.github.com/repos/quattor/quattor.github.com/issues/137","number":137,"state":"open","locked":false,"title":"Migrate SCDB installation doc from LAL wiki","user":{"login":"jouvin","id":1511937,"avatar_url":"https://avatars.githubusercontent.com/u/1511937?v=3","gravatar_id":"","url":"https://api.github.com/users/jouvin","html_url":"https://github.com/jouvin","followers_url":"https://api.github.com/users/jouvin/followers","following_url":"https://api.github.com/users/jouvin/following{/other_user}","gists_url":"https://api.github.com/users/jouvin/gists{/gist_id}","starred_url":"https://api.github.com/users/jouvin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jouvin/subscriptions","organizations_url":"https://api.github.com/users/jouvin/orgs","repos_url":"https://api.github.com/users/jouvin/repos","events_url":"https://api.github.com/users/jouvin/events{/privacy}","received_events_url":"https://api.github.com/users/jouvin/received_events","type":"User","site_admin":false},"body":"Work in progress. Reviewed until `Quattor Server` heading.\r\n\r\nFixes #129.","created_at":"2015-03-26T00:19:54Z","updated_at":"2015-04-07T10:30:33Z","closed_at":null,"merged_at":null,"merge_commit_sha":"27103e288dc56c98bc3b2b1f625abae7bef10eba","assignee":{"login":"jouvin","id":1511937,"avatar_url":"https://avatars.githubusercontent.com/u/1511937?v=3","gravatar_id":"","url":"https://api.github.com/users/jouvin","html_url":"https://github.com/jouvin","followers_url":"https://api.github.com/users/jouvin/followers","following_url":"https://api.github.com/users/jouvin/following{/other_user}","gists_url":"https://api.github.com/users/jouvin/gists{/gist_id}","starred_url":"https://api.github.com/users/jouvin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jouvin/subscriptions","organizations_url":"https://api.github.com/users/jouvin/orgs","repos_url":"https://api.github.com/users/jouvin/repos","events_url":"https://api.github.com/users/jouvin/events{/privacy}","received_events_url":"https://api.github.com/users/jouvin/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/quattor/quattor.github.com/pulls/137/commits","review_comments_url":"https://api.github.com/repos/quattor/quattor.github.com/pulls/137/comments","review_comment_url":"https://api.github.com/repos/quattor/quattor.github.com/pulls/comments{/number}","comments_url":"https://api.github.com/repos/quattor/quattor.github.com/issues/137/comments","statuses_url":"https://api.github.com/repos/quattor/quattor.github.com/statuses/38edbba8c4a372e48d65b67b3954a4e59a2ec4e8","head":{"label":"jouvin:scdb-install","ref":"scdb-install","sha":"38edbba8c4a372e48d65b67b3954a4e59a2ec4e8","user":{"login":"jouvin","id":1511937,"avatar_url":"https://avatars.githubusercontent.com/u/1511937?v=3","gravatar_id":"","url":"https://api.github.com/users/jouvin","html_url":"https://github.com/jouvin","followers_url":"https://api.github.com/users/jouvin/followers","following_url":"https://api.github.com/users/jouvin/following{/other_user}","gists_url":"https://api.github.com/users/jouvin/gists{/gist_id}","starred_url":"https://api.github.com/users/jouvin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jouvin/subscriptions","organizations_url":"https://api.github.com/users/jouvin/orgs","repos_url":"https://api.github.com/users/jouvin/repos","events_url":"https://api.github.com/users/jouvin/events{/privacy}","received_events_url":"https://api.github.com/users/jouvin/received_events","type":"User","site_admin":false},"repo":{"id":18037113,"name":"quattor.github.com","full_name":"jouvin/quattor.github.com","owner":{"login":"jouvin","id":1511937,"avatar_url":"https://avatars.githubusercontent.com/u/1511937?v=3","gravatar_id":"","url":"https://api.github.com/users/jouvin","html_url":"https://github.com/jouvin","followers_url":"https://api.github.com/users/jouvin/followers","following_url":"https://api.github.com/users/jouvin/following{/other_user}","gists_url":"https://api.github.com/users/jouvin/gists{/gist_id}","starred_url":"https://api.github.com/users/jouvin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jouvin/subscriptions","organizations_url":"https://api.github.com/users/jouvin/orgs","repos_url":"https://api.github.com/users/jouvin/repos","events_url":"https://api.github.com/users/jouvin/events{/privacy}","received_events_url":"https://api.github.com/users/jouvin/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/jouvin/quattor.github.com","description":"External Website and Project Documentation","fork":true,"url":"https://api.github.com/repos/jouvin/quattor.github.com","forks_url":"https://api.github.com/repos/jouvin/quattor.github.com/forks","keys_url":"https://api.github.com/repos/jouvin/quattor.github.com/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jouvin/quattor.github.com/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jouvin/quattor.github.com/teams","hooks_url":"https://api.github.com/repos/jouvin/quattor.github.com/hooks","issue_events_url":"https://api.github.com/repos/jouvin/quattor.github.com/issues/events{/number}","events_url":"https://api.github.com/repos/jouvin/quattor.github.com/events","assignees_url":"https://api.github.com/repos/jouvin/quattor.github.com/assignees{/user}","branches_url":"https://api.github.com/repos/jouvin/quattor.github.com/branches{/branch}","tags_url":"https://api.github.com/repos/jouvin/quattor.github.com/tags","blobs_url":"https://api.github.com/repos/jouvin/quattor.github.com/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jouvin/quattor.github.com/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jouvin/quattor.github.com/git/refs{/sha}","trees_url":"https://api.github.com/repos/jouvin/quattor.github.com/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jouvin/quattor.github.com/statuses/{sha}","languages_url":"https://api.github.com/repos/jouvin/quattor.github.com/languages","stargazers_url":"https://api.github.com/repos/jouvin/quattor.github.com/stargazers","contributors_url":"https://api.github.com/repos/jouvin/quattor.github.com/contributors","subscribers_url":"https://api.github.com/repos/jouvin/quattor.github.com/subscribers","subscription_url":"https://api.github.com/repos/jouvin/quattor.github.com/subscription","commits_url":"https://api.github.com/repos/jouvin/quattor.github.com/commits{/sha}","git_commits_url":"https://api.github.com/repos/jouvin/quattor.github.com/git/commits{/sha}","comments_url":"https://api.github.com/repos/jouvin/quattor.github.com/comments{/number}","issue_comment_url":"https://api.github.com/repos/jouvin/quattor.github.com/issues/comments{/number}","contents_url":"https://api.github.com/repos/jouvin/quattor.github.com/contents/{+path}","compare_url":"https://api.github.com/repos/jouvin/quattor.github.com/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jouvin/quattor.github.com/merges","archive_url":"https://api.github.com/repos/jouvin/quattor.github.com/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jouvin/quattor.github.com/downloads","issues_url":"https://api.github.com/repos/jouvin/quattor.github.com/issues{/number}","pulls_url":"https://api.github.com/repos/jouvin/quattor.github.com/pulls{/number}","milestones_url":"https://api.github.com/repos/jouvin/quattor.github.com/milestones{/number}","notifications_url":"https://api.github.com/repos/jouvin/quattor.github.com/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jouvin/quattor.github.com/labels{/name}","releases_url":"https://api.github.com/repos/jouvin/quattor.github.com/releases{/id}","created_at":"2014-03-23T16:05:06Z","updated_at":"2014-06-24T10:55:30Z","pushed_at":"2015-04-02T21:16:40Z","git_url":"git://github.com/jouvin/quattor.github.com.git","ssh_url":"git@github.com:jouvin/quattor.github.com.git","clone_url":"https://github.com/jouvin/quattor.github.com.git","svn_url":"https://github.com/jouvin/quattor.github.com","homepage":"www.quattor.org","size":1958,"stargazers_count":0,"watchers_count":0,"language":"CSS","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":0,"forks":1,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"quattor:master","ref":"master","sha":"62d93c663a12c2d233e5960a2d2e2d1b4467ea26","user":{"login":"quattor","id":1557169,"avatar_url":"https://avatars.githubusercontent.com/u/1557169?v=3","gravatar_id":"","url":"https://api.github.com/users/quattor","html_url":"https://github.com/quattor","followers_url":"https://api.github.com/users/quattor/followers","following_url":"https://api.github.com/users/quattor/following{/other_user}","gists_url":"https://api.github.com/users/quattor/gists{/gist_id}","starred_url":"https://api.github.com/users/quattor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/quattor/subscriptions","organizations_url":"https://api.github.com/users/quattor/orgs","repos_url":"https://api.github.com/users/quattor/repos","events_url":"https://api.github.com/users/quattor/events{/privacy}","received_events_url":"https://api.github.com/users/quattor/received_events","type":"Organization","site_admin":false},"repo":{"id":3776935,"name":"quattor.github.com","full_name":"quattor/quattor.github.com","owner":{"login":"quattor","id":1557169,"avatar_url":"https://avatars.githubusercontent.com/u/1557169?v=3","gravatar_id":"","url":"https://api.github.com/users/quattor","html_url":"https://github.com/quattor","followers_url":"https://api.github.com/users/quattor/followers","following_url":"https://api.github.com/users/quattor/following{/other_user}","gists_url":"https://api.github.com/users/quattor/gists{/gist_id}","starred_url":"https://api.github.com/users/quattor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/quattor/subscriptions","organizations_url":"https://api.github.com/users/quattor/orgs","repos_url":"https://api.github.com/users/quattor/repos","events_url":"https://api.github.com/users/quattor/events{/privacy}","received_events_url":"https://api.github.com/users/quattor/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/quattor/quattor.github.com","description":"External Website and Project Documentation","fork":false,"url":"https://api.github.com/repos/quattor/quattor.github.com","forks_url":"https://api.github.com/repos/quattor/quattor.github.com/forks","keys_url":"https://api.github.com/repos/quattor/quattor.github.com/keys{/key_id}","collaborators_url":"https://api.github.com/repos/quattor/quattor.github.com/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/quattor/quattor.github.com/teams","hooks_url":"https://api.github.com/repos/quattor/quattor.github.com/hooks","issue_events_url":"https://api.github.com/repos/quattor/quattor.github.com/issues/events{/number}","events_url":"https://api.github.com/repos/quattor/quattor.github.com/events","assignees_url":"https://api.github.com/repos/quattor/quattor.github.com/assignees{/user}","branches_url":"https://api.github.com/repos/quattor/quattor.github.com/branches{/branch}","tags_url":"https://api.github.com/repos/quattor/quattor.github.com/tags","blobs_url":"https://api.github.com/repos/quattor/quattor.github.com/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/quattor/quattor.github.com/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/quattor/quattor.github.com/git/refs{/sha}","trees_url":"https://api.github.com/repos/quattor/quattor.github.com/git/trees{/sha}","statuses_url":"https://api.github.com/repos/quattor/quattor.github.com/statuses/{sha}","languages_url":"https://api.github.com/repos/quattor/quattor.github.com/languages","stargazers_url":"https://api.github.com/repos/quattor/quattor.github.com/stargazers","contributors_url":"https://api.github.com/repos/quattor/quattor.github.com/contributors","subscribers_url":"https://api.github.com/repos/quattor/quattor.github.com/subscribers","subscription_url":"https://api.github.com/repos/quattor/quattor.github.com/subscription","commits_url":"https://api.github.com/repos/quattor/quattor.github.com/commits{/sha}","git_commits_url":"https://api.github.com/repos/quattor/quattor.github.com/git/commits{/sha}","comments_url":"https://api.github.com/repos/quattor/quattor.github.com/comments{/number}","issue_comment_url":"https://api.github.com/repos/quattor/quattor.github.com/issues/comments{/number}","contents_url":"https://api.github.com/repos/quattor/quattor.github.com/contents/{+path}","compare_url":"https://api.github.com/repos/quattor/quattor.github.com/compare/{base}...{head}","merges_url":"https://api.github.com/repos/quattor/quattor.github.com/merges","archive_url":"https://api.github.com/repos/quattor/quattor.github.com/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/quattor/quattor.github.com/downloads","issues_url":"https://api.github.com/repos/quattor/quattor.github.com/issues{/number}","pulls_url":"https://api.github.com/repos/quattor/quattor.github.com/pulls{/number}","milestones_url":"https://api.github.com/repos/quattor/quattor.github.com/milestones{/number}","notifications_url":"https://api.github.com/repos/quattor/quattor.github.com/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/quattor/quattor.github.com/labels{/name}","releases_url":"https://api.github.com/repos/quattor/quattor.github.com/releases{/id}","created_at":"2012-03-20T15:50:40Z","updated_at":"2015-03-26T12:04:18Z","pushed_at":"2015-04-02T21:16:41Z","git_url":"git://github.com/quattor/quattor.github.com.git","ssh_url":"git@github.com:quattor/quattor.github.com.git","clone_url":"https://github.com/quattor/quattor.github.com.git","svn_url":"https://github.com/quattor/quattor.github.com","homepage":"www.quattor.org","size":2390,"stargazers_count":8,"watchers_count":8,"language":"CSS","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":10,"mirror_url":null,"open_issues_count":6,"forks":10,"open_issues":6,"watchers":8,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/quattor/quattor.github.com/pulls/137"},"html":{"href":"https://github.com/quattor/quattor.github.com/pull/137"},"issue":{"href":"https://api.github.com/repos/quattor/quattor.github.com/issues/137"},"comments":{"href":"https://api.github.com/repos/quattor/quattor.github.com/issues/137/comments"},"review_comments":{"href":"https://api.github.com/repos/quattor/quattor.github.com/pulls/137/comments"},"review_comment":{"href":"https://api.github.com/repos/quattor/quattor.github.com/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/quattor/quattor.github.com/pulls/137/commits"},"statuses":{"href":"https://api.github.com/repos/quattor/quattor.github.com/statuses/38edbba8c4a372e48d65b67b3954a4e59a2ec4e8"}}}}