Puppet agent unable to retrieve catalog from master when using ENC (ERROR 400 on Server)
<p>I have a rather bizarre issue that I am unable to resolve. I have a Puppet environment that uses Foreman as an ENC via the /etc/puppet/node.rb script. The master is servicing roughly one hundred hosts. All but three hosts are working perfectly. </p>

<p>The error I get when triggering a catalog run on the three affected hosts is as follows:</p>

<pre><code>Error: Could not retrieve catalog from remote server: Error 400 on
SERVER: Failed when searching for node myhost.mydomain.foo.bar: Failed to find myhost.mydomain.foo.bar via exec: Execution of '/etc/puppet/node.rb myhost.mydomain.foo.bar' returned : 
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
</code></pre>

<p>Manually running /etc/puppet/node.rb myhost.mydomain.foo.bar returns:</p>

<pre><code>---
classes:
  bash: 
  common_packages: 
  cron: 
  customfacts: 
  fail2ban: 
  ntp: 
  puppet: 
  resolvconf: 
  ssh: 
  sysctl: 
  syslog_ng: 
  yum_repos: 
  zabbix: 
parameters:
  puppetmaster: mymaster.mydomain.foo.bar
  hostgroup: Base/App servers
  root_pw: 
  foreman_env: production
  owner_name: Admin User
  owner_email: devops@mydomain.foo.bar
  foreman_subnets: []
  foreman_interfaces:
  - mac: 
    ip: 
    type: Interface
    name: myhost.mydomain.foo.bar
    attrs: {}
    virtual: false
    link: true
    identifier: ''
    managed: true
    primary: true
    provision: true
    subnet: 
environment: production
</code></pre>

<p>Using curl to grab the node definition from the master works fine:</p>

<pre><code>curl -k -H 'Accept: pson' --cert /var/lib/puppet/ssl/certs/$(hostname -f).pem --key /var/lib/puppet/ssl/private_keys/$(hostname -f).pem "https://mymaster.mydomain.foo.bar:8140/production/node/$(hostname -f)?transaction_uuid=701fa6a0-3240-42f6-8ea1-209b86535dec&amp;fail_on_404=true"
</code></pre>

<p>Using curl to grab the catalog from the master also works just fine (this returns a massive, unformatted catalog):</p>

<pre><code>curl -k -H 'Accept: pson' --cert /var/lib/puppet/ssl/certs/$(hostname -f).pem --key /var/lib/puppet/ssl/private_keys/$(hostname -f).pem "https://mymaster.mydomain.foo.bar:8140/production/catalog/$(hostname -f)"
</code></pre>

<p>Manually compiling a catalog on the master by doing</p>

<pre><code>puppet master --compile myhost.mydomain.foo.bar &gt; /tmp/myhost.mydomain.foo.bar.json
</code></pre>

<p>Copying the json file to the following location on the agent:</p>

<pre><code>/var/lib/puppet/client_data/catalog/
</code></pre>

<p>And then manually applying it like so:</p>

<pre><code>puppet catalog apply --server=mymaster.mydomain.foo.bar --verbose --terminus=json
</code></pre>

<p>Triggers a proper catalog run.</p>

<p>My agent's config looks as follows (this is the same for all hosts):</p>

<pre><code># This file is managed by Puppet
# Please do not edit this file

[main]

logdir = /var/log/puppet
rundir = /var/run/puppet
ssldir = $vardir/ssl

[agent]

classfile   = $vardir/classes.txt
localconfig = $vardir/localconfig

ca_server   = myforeman.mydomain.foo.bar
server      = mymaster.mydomain.foo.bar
environment = production

report            = true
runinterval       = 600
usecacheonfailure = true
</code></pre>

<p>My master's config is as follows:</p>

<pre><code>[main]
    logdir           = /var/log/puppet
    rundir           = /var/run/puppet
    ssldir           = $vardir/ssl
    privatekeydir    = $ssldir/private_keys { group = service }
    hostprivkey      = $privatekeydir/$certname.pem { mode = 640 }
    autosign         = $confdir/autosign.conf { mode = 664 }
    show_diff        = false
    hiera_config     = $confdir/hiera.yaml
    environmentpath  = /etc/puppet/environments
    basemodulepath   = /etc/puppet/environments/common:/etc/puppet/modules:/usr/share/puppet/modules

[agent]
    classfile         = $vardir/classes.txt
    localconfig       = $vardir/localconfig
    default_schedules = false
    report            = true
    pluginsync        = true
    masterport        = 8140
    environment       = production
    certname          = mymaster.mydomain.foo.bar
    server            = myforeman.mydomain.foo.bar
    listen            = false
    splay             = false
    splaylimit        = 1800
    runinterval       = 600
    noop              = false
    configtimeout     = 120
    usecacheonfailure = true

[master]
    autosign         = $confdir/autosign.conf { mode = 664 }
    reports          = log, foreman
    external_nodes   = /etc/puppet/node.rb
    node_terminus    = exec
    ca               = false
    ssldir           = /var/lib/puppet/ssl
    certname         = mymaster.mydomain.foo.bar
    parser           = current
    strict_variables = false
</code></pre>

<p>/etc/puppet/foreman.yaml is configured as follows:</p>

<pre><code>---
:url: "https://myforeman.mydomain.foo.bar"
:ssl_ca: "/var/lib/puppet/ssl/certs/ca.pem"
:ssl_cert: "/var/lib/puppet/ssl/certs/mymaster.mydomain.foo.bar.pem"
:ssl_key: "/var/lib/puppet/ssl/private_keys/myforeman.mydomain.foo.bar.pem"
:user: "admin"
:password: "mypassword"
:puppetdir: "/var/lib/puppet"
:puppetuser: "puppet"
:facts: true
:timeout: 10
:threads: null
</code></pre>

<p>Any advice or assistance would be highly appreciated!</p>

<p>UPDATE: I've included some info from Foreman's production.log below.</p>

<pre><code>Started GET "/node/myhost.mydomain.foo.bar?format=yml" for 172.16.2.109 at 2016-07-14 07:43:44 +0200
2016-07-14 07:43:44 [I] Processing by HostsController#externalNodes as YML
2016-07-14 07:43:44 [I]   Parameters: {"name"=&gt;"myhost.mydomain.foo.bar"}
2016-07-14 07:43:44 [I] Completed 201 Created in 283ms (Views: 11.0ms | ActiveRecord: 0.0ms)
2016-07-14 07:43:44 [I] 

Started POST "/api/hosts/facts" for 172.16.2.109 at 2016-07-14 07:43:47 +0200
2016-07-14 07:43:47 [I] Processing by Api::V2::HostsController#facts as JSON
2016-07-14 07:43:47 [I]   Parameters: {"certname"=&gt;"myhost.mydomain.foo.bar", "name"=&gt;"myhost.mydomain.foo.bar", "facts"=&gt;"[FILTERED]", "apiv"=&gt;"v2", :host=&gt;{"name"=&gt;"myhost.mydomain.foo.bar", "certname"=&gt;"myhost.mydomain.foo.bar"}}
2016-07-14 07:43:47 [I] Import facts for 'myhost.mydomain.foo.bar' completed. Added: 1, Updated: 6, Deleted 0 facts
2016-07-14 07:43:49 [I] 

Started POST "/api/reports" for 172.16.2.109 at 2016-07-14 07:44:48 +0200
2016-07-14 07:44:48 [I] Processing by Api::V2::ReportsController#create as JSON
2016-07-14 07:44:48 [I]   Parameters: {"report"=&gt;"[FILTERED]", "apiv"=&gt;"v2"}
2016-07-14 07:44:48 [I] processing report for myhost.mydomain.foo.bar
2016-07-14 07:44:48 [I]   Rendered dashboard/_reports_widget.html.erb (80.0ms)
2016-07-14 07:44:48 [I]   Rendered dashboard/_distribution_widget.html.erb (38.4ms)
2016-07-14 07:44:48 [I]   Rendered dashboard/index.html.erb within layouts/application (125.4ms)
2016-07-14 07:44:48 [I]   Rendered common/_searchbar.html.erb (2.9ms)
2016-07-14 07:44:48 [I]   Rendered home/_user_dropdown.html.erb (1.0ms)
2016-07-14 07:44:48 [I] Read fragment views/tabs_and_title_records-3 (0.1ms)
2016-07-14 07:44:48 [I]   Rendered home/_topbar.html.erb (1.8ms)
2016-07-14 07:44:48 [I]   Rendered layouts/base.html.erb (2.8ms)
2016-07-14 07:44:48 [I] Completed 200 OK in 185ms (Views: 41.9ms | ActiveRecord: 117.8ms)
2016-07-14 07:44:48 [I] Imported report for myhost.mydomain.foo.bar in 0.12 seconds
2016-07-14 07:44:48 [I] Completed 201 Created in 135ms (Views: 1.0ms | ActiveRecord: 0.0ms)
2016-07-14 07:44:51 [I] 
</code></pre>