Openstack: Horizon - "Unable to retrieve usage information" error after VM Shutdown
<p>I have setup Openstack (Newton) with Nova (Compute) on Ubuntu 16.04.1 LTS Desktop as a virtual machine guest (Oracle VM VirtualBox ver.5.1.8) on Windows 10 host.</p>

<p>The installation was successful and could access the Horizon after the login process via <a href="http://192.168.56.102/dashboard/" rel="nofollow noreferrer">http://192.168.56.102/dashboard/</a> (from the Host) as shown in the figure below. (Can also access via <a href="http://10.0.2.15/dashboard/" rel="nofollow noreferrer">http://10.0.2.15/dashboard/</a> from the Guest but not shown here)</p>

<p><a href="https://i.stack.imgur.com/Tynd4.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Tynd4.png" alt="enter image description here"></a></p>

<p>When the VM was shut down and later switched on again, the access to Openstack using Horizon’s dashboard was lost. “Error: Unable to retrieve usage information” is shown on the dashboard (as in the top right position of the figure below). This issue is a hard problem and could be replicated all the time.</p>

<p><a href="https://i.stack.imgur.com/uaAo8.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/uaAo8.png" alt="enter image description here"></a></p>

<p>The below is the snipped of error shown in /var/log/apache2/horizon_error.log file when Computer->Overview option was selected from the Horizon’s dashboard after the login process (that produces Figure 2 above).</p>

<pre><code>2016-12-02 13:03:08.786092 Recoverable error: HTTPConnectionPool(host='10.0.2.15', port=8774): Max retries exceeded with url: /v2.1/extensions (Caused by NewConnectionError('&lt;requests.packages.urllib3.connection.HTTPConnection object at 0x7f3436ebeb10&gt;: Failed to establish a new connection: [Errno 111] Connection refused',))
2016-12-02 13:03:08.804729 DEBUG:keystoneauth.session:REQ: curl -g -i -X GET http://10.0.2.15/identity/v3/users/9af3721f2d8e41cc9342b9941bafc5d1/projects -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}67ffd6b14a7e8891ae7cc8b29a639b81822c7b45"
2016-12-02 13:03:08.935723 DEBUG:keystoneauth.session:RESP: [200] Date: Fri, 02 Dec 2016 13:03:08 GMT Server: Apache/2.4.18 (Ubuntu) Vary: X-Auth-Token x-openstack-request-id: req-fa398fe1-c5c9-4ce6-986b-cec266278f04 Content-Length: 966 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: application/json 
2016-12-02 13:03:08.935791 RESP BODY: {"links": {"self": "http://10.0.2.15/identity/v3/users/9af3721f2d8e41cc9342b9941bafc5d1/projects", "previous": null, "next": null}, "projects": [{"is_domain": false, "description": "Bootstrap project for initializing the cloud.", "links": {"self": "http://10.0.2.15/identity/v3/projects/4350966521b54001aeb666b6e55b7deb"}, "enabled": true, "id": "4350966521b54001aeb666b6e55b7deb", "parent_id": "default", "domain_id": "default", "name": "admin"}, {"is_domain": false, "description": "", "links": {"self": "http://10.0.2.15/identity/v3/projects/43dba94541ba4e8e83d5528038346ea9"}, "enabled": true, "id": "43dba94541ba4e8e83d5528038346ea9", "parent_id": "default", "domain_id": "default", "name": "demo"}, {"is_domain": false, "description": "", "links": {"self": "http://10.0.2.15/identity/v3/projects/be3ee2f8cb074628ac24c8be6e4c7160"}, "enabled": true, "id": "be3ee2f8cb074628ac24c8be6e4c7160", "parent_id": "default", "domain_id": "default", "name": "alt_demo"}]}
2016-12-02 13:03:08.935809 
2016-12-02 13:03:09.018515 Call to list enabled services failed. This is likely due to a problem communicating with the Neutron endpoint. Load Balancers panel will not be displayed.
2016-12-02 13:03:09.024775 Call to list enabled services failed. This is likely due to a problem communicating with the Neutron endpoint. Firewalls panel will not be displayed.
2016-12-02 13:03:09.029002 Call to list enabled services failed. This is likely due to a problem communicating with the Neutron endpoint. VPN panel will not be displayed.
2016-12-02 13:03:09.035255 Call to list supported extensions failed. This is likely due to a problem communicating with the Nova endpoint. Host Aggregates panel will not be displayed
</code></pre>

<p>As the error about indicated problem in communicating with the endpoint. Verified if the endpoint for the services is ok in the VM, unfortunately it is not.</p>

<pre><code>svanan@svanan:~/devstack$ openstack endpoint list
Password: 
Missing parameter(s): 
Set a username with --os-username, OS_USERNAME, or auth.username
Set an authentication URL, with --os-auth-url, OS_AUTH_URL or auth.auth_url
</code></pre>

<p>The below is the original output in the VM when before it was shutdown.</p>

<pre><code>svanan@svanan:~/devstack$ openstack endpoint list
+----------------------------------+-----------+--------------+----------------+---------+-----------+-----------------------------------------+
| ID                               | Region    | Service Name | Service Type   | Enabled | Interface | URL                                     |
+----------------------------------+-----------+--------------+----------------+---------+-----------+-----------------------------------------+
| 07480eca2071416c8dec30bafa95ff8f | RegionOne | neutron      | network        | True    | admin     | http://10.0.2.15:9696/                  |
| 0aa32dc14c194561816b6d1b60d04404 | RegionOne | glance       | image          | True    | admin     | http://10.0.2.15:9292                   |
| 0e5365a8ab3744c780feb25ecffc2c93 | RegionOne | keystone     | identity       | True    | internal  | http://10.0.2.15/identity               |
| 15d8608b021f42e2a3b0402889a9064f | RegionOne | neutron      | network        | True    | public    | http://10.0.2.15:9696/                  |
| 289a99bb83e846b2bfc5eabfb800d86d | RegionOne | glance       | image          | True    | internal  | http://10.0.2.15:9292                   |
| 41397886443d44b7b6d857fccb34d55d | RegionOne | neutron      | network        | True    | internal  | http://10.0.2.15:9696/                  |
| 4458eed550ae44479feca25f9bec96ce | RegionOne | nova_legacy  | compute_legacy | True    | public    | http://10.0.2.15:8774/v2/$(project_id)s |
| 51be7c90e6c4437fbb75c3a40857774f | RegionOne | keystone     | identity       | True    | public    | http://10.0.2.15/identity               |
| 7526c5a7e5084a07b1d1c7affe261ae4 | RegionOne | nova_legacy  | compute_legacy | True    | admin     | http://10.0.2.15:8774/v2/$(project_id)s |
| 7c1e026c1c2d4f0b9dfdfe15860d6c5b | RegionOne | keystone     | identity       | True    | admin     | http://10.0.2.15/identity_v2_admin      |
| 8641d88381fc499688162e8bcde5456c | RegionOne | cinderv2     | volumev2       | True    | internal  | http://10.0.2.15:8776/v2/$(project_id)s |
| 8a194a2954a943568ccaf8bce5eb9400 | RegionOne | cinder       | volume         | True    | internal  | http://10.0.2.15:8776/v1/$(project_id)s |
| 8c0b4d932da34c5990e43c0877146819 | RegionOne | nova         | compute        | True    | admin     | http://10.0.2.15:8774/v2.1              |
| 94e8cae799954ba09abcd3b31b1c926e | RegionOne | nova         | compute        | True    | public    | http://10.0.2.15:8774/v2.1              |
| a1527968e91d4df9b98fe09039e80bed | RegionOne | cinderv2     | volumev2       | True    | public    | http://10.0.2.15:8776/v2/$(project_id)s |
| a3e47c8f82d546c8917a8223b8b81a74 | RegionOne | cinderv3     | volumev3       | True    | internal  | http://10.0.2.15:8776/v3/$(project_id)s |
| b39a19918016462db99cf1de151fc870 | RegionOne | cinderv3     | volumev3       | True    | public    | http://10.0.2.15:8776/v3/$(project_id)s |
| d17514aa0eec4752848df9599dd8cd00 | RegionOne | cinder       | volume         | True    | admin     | http://10.0.2.15:8776/v1/$(project_id)s |
| db570fe7343b461a95cdd4179e1b28d9 | RegionOne | glance       | image          | True    | public    | http://10.0.2.15:9292                   |
| eb4aa71c03b0418eadfed00715744b07 | RegionOne | cinderv3     | volumev3       | True    | admin     | http://10.0.2.15:8776/v3/$(project_id)s |
| ee474205e562411abc1842dc9bccbb5d | RegionOne | nova_legacy  | compute_legacy | True    | internal  | http://10.0.2.15:8774/v2/$(project_id)s |
| f383e0696ee647f79f6fd3a3d7ac5dff | RegionOne | cinder       | volume         | True    | public    | http://10.0.2.15:8776/v1/$(project_id)s |
| f4239a1f86144d8dab3ed94237d7c10f | RegionOne | cinderv2     | volumev2       | True    | admin     | http://10.0.2.15:8776/v2/$(project_id)s |
| fe679fcbc79e4664b7897a381e90dd78 | RegionOne | nova         | compute        | True    | internal  | http://10.0.2.15:8774/v2.1              |
+----------------------------------+-----------+--------------+----------------+---------+-----------+-----------------------------------------+
</code></pre>

<p>Verified ifconfig at VM, it has the enp0s3, enp0s8, lo, virbr0  interfaces configured except the br-ex which was there before the shutdown as below:</p>

<pre><code>br-ex     Link encap:Ethernet  HWaddr 1a:ba:3a:4a:d5:4f  
          inet addr:172.24.4.1  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::18ba:3aff:fe4a:d54f/64 Scope:Link
          inet6 addr: 2001:db8::2/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:15 errors:0 dropped:0 overruns:0 frame:0
          TX packets:79 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:776 (776.0 B)  TX bytes:10328 (10.3 KB)
</code></pre>

<p>Not sure if the absense of this bridge (br-ex) could be the cause Horizon (from the Host) could not connect with the services in the Guest. Tried to configure statically in etc/network/interfaces as below to get the same above br-ex output.</p>

<pre><code>auto br-ex
  iface br-ex inet static
  hwaddress 1a:ba:3a:4a:d5:4f
  address 172.24.4.1
  broadcast 0.0.0.0
  netmask 255.255.255.0

  iface br-ex inet6 static
  pre-up modprobe ipv6
  address 2001:db8::2
  netmask 64
</code></pre>

<p>The br-ex output I got after </p>

<pre><code>svanan@svanan:~$ sudo systemctl restart networking
svanan@svanan:~$ ifconfig

br-ex     Link encap:Ethernet  HWaddr 7a:07:1d:1d:1a:40  
          inet addr:172.24.4.1  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::18ba:3aff:fe4a:d54f/64 Scope:Link
          inet6 addr: 2001:db8::2/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:126 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:0 (0.0 B)  TX bytes:19334 (19.3 KB)
</code></pre>

<p>As can be seen above I am unable to configure HWaddr to 1a:ba:3a:4a:d5:4f instead 7a:07:1d:1d:1a:40 showing up.</p>

<p><strong>Questions:</strong></p>

<ol>
<li>Could the absense of br-ex after the VM shutdown and switched on sequence could be the reason Horizon’s failure to retrieve the information from its services.</li>
<li>How to statically set the HWaddr to 1a:ba:3a:4a:d5:4f for br-ex? The syntax hwaddress 1a:ba:3a:4a:d5:4f does not seem to work.</li>
<li>If you faces similar problem, can you share how did you solve it.</li>
<li><p>Is it true, one need to run ./unstack.sh and ./stack.sh ( ./rejoin-stack.sh has been depreciated in Newton) everytime a VM is restarted as mentioned in <a href="https://stackoverflow.com/questions/36268822/no-rejoin-stack-sh-script-in-my-setup">No rejoin-stack.sh script in my setup</a> .This approach takes very long time as shown below,</p>

<p>=========================</p>

<p>DevStack Component Timing</p>

<p>=========================</p>

<p>Total runtime         1777</p></li>
</ol>

<p>Hope you can assist to guide me in this. Thanks.</p>

<p>Saravanan Kandasamy</p>