Paste #541592

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
I followed the below reference links.
http://www.ubuntu.com/download/cloud/install-openstack-with-autopilot

I am using MAAS Version 1.9.3+bzr4577-0ubuntu1 (wily1).

1) I setup a MAAS server and an empty node to PXEboot.
2) Registered node to MAAS.
3)Started commission
4)The empty node PXE booted and Configured OS.
5)Once the OS is configured, tried to SSH from MAAS controller to node.
6)SSH not working [giving permission denied public key issue]. But Node IP and Hostname pinging from MAAS controller.
7)The node has internet access through maas as gateway and using maas DNS services
8)But while running cloud-init* script suddenly the node losing its network communication from MAAS controller. 
9)Due to this, commissioning script call back to MAAS controller via API request as commission failed.

vedams@maas-node:~$ dpkg -l '*maas*'|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                           Version                      Architecture Description
+++-==============================-============================-============-======================================
ii  maas                           1.9.3+bzr4577-0ubuntu1~wily1 all          MAAS server all-in-one metapackage
ii  maas-cli                       1.9.3+bzr4577-0ubuntu1~wily1 all          MAAS command line API tool
ii  maas-cluster-controller        1.9.3+bzr4577-0ubuntu1~wily1 all          MAAS server cluster controller
ii  maas-common                    1.9.3+bzr4577-0ubuntu1~wily1 all          MAAS server common files
ii  maas-dhcp                      1.9.3+bzr4577-0ubuntu1~wily1 all          MAAS DHCP server
ii  maas-dns                       1.9.3+bzr4577-0ubuntu1~wily1 all          MAAS DNS server
ii  maas-proxy                     1.9.3+bzr4577-0ubuntu1~wily1 all          MAAS Caching Proxy
ii  maas-region-controller         1.9.3+bzr4577-0ubuntu1~wily1 all          MAAS server complete region controller
ii  maas-region-controller-min     1.9.3+bzr4577-0ubuntu1~wily1 all          MAAS Server minimum region controller
ii  python-django-maas             1.9.3+bzr4577-0ubuntu1~wily1 all          MAAS server Django web framework
ii  python-maas-client             1.9.3+bzr4577-0ubuntu1~wily1 all          MAAS python API client
ii  python-maas-provisioningserver 1.9.3+bzr4577-0ubuntu1~wily1 all          MAAS server provisioning libraries
vedams@maas-node:~$ zip -r ~/maas.zip /var/log/maas/*
  adding: var/log/maas/apache2/ (stored 0%)
  adding: var/log/maas/apache2/error.log (deflated 77%)
  adding: var/log/maas/apache2/access.log (deflated 95%)
  adding: var/log/maas/apache2/other_vhosts_access.log (stored 0%)
  adding: var/log/maas/clusterd.log (deflated 94%)
  adding: var/log/maas/maas.log (deflated 91%)
  adding: var/log/maas/proxy/ (stored 0%)
  adding: var/log/maas/regiond.log (deflated 95%)
  adding: var/log/maas/rsyslog/ (stored 0%)
vedams@maas-node:~$ ls -al /var/log/maas
total 568
drwxr-xr-x  4 root   root     4096 Jul 20 20:49 .
drwxrwxr-x 14 root   syslog   4096 Jul 20 20:47 ..
lrwxrwxrwx  1 root   root       16 Jul 20 20:49 apache2 -> /var/log/apache2
-rw-r--r--  1 maas   maas    84353 Jul 21 00:17 clusterd.log
-rw-r--r--  1 syslog syslog  20489 Jul 21 01:44 maas.log
drwxr-x---  2 proxy  proxy    4096 Jul 20 20:49 proxy
-rw-r--r--  1 maas   maas   442324 Jul 21 01:50 regiond.log
drwxr-xr-x  2 syslog syslog   4096 Jul 20 20:49 rsyslog

 
maas.log
======

Jul 20 21:51:09 maas-node maas.bootresources: [INFO] Started importing of boot images from 1 source(s).
Jul 20 21:51:25 maas-node maas.bootresources: [INFO] Importing images from source: http://maas.ubuntu.com/images/ephemeral-v2/releases/
Jul 20 21:51:39 maas-node maas.bootresources: [INFO] Finished importing of boot images from 1 source(s).
Jul 20 21:51:39 maas-node maas.import-images: [INFO] Started importing boot images.
Jul 20 21:51:39 maas-node maas.import-images: [INFO] Finished importing boot images, the region does not have any new images.
Jul 20 22:05:45 maas-node maas.node: [INFO] slave1: Status transition from NEW to COMMISSIONING
Jul 20 22:05:45 maas-node maas.dns: [INFO] Generating new DNS zone file for maas
Jul 20 22:05:45 maas-node maas.dns: [INFO] Generating new DNS zone file for 10.168.192.in-addr.arpa
Jul 20 22:05:45 maas-node maas.power: [INFO] Changing power state (on) of node: slave1 (node-00f4f76c-4e98-11e6-b4a4-0800272733e3)
Jul 20 22:05:45 maas-node maas.node: [INFO] slave1: Commissioning started
Jul 20 22:05:47 maas-node maas.power: [INFO] Changed power state (on) of node: slave1 (node-00f4f76c-4e98-11e6-b4a4-0800272733e3)
Jul 20 22:08:37 maas-node maas.lease_upload_service: [INFO] Uploading 1 DHCP leases to region controller.
Jul 20 22:10:37 maas-node maas.import-images: [INFO] Started importing boot images.
Jul 20 22:10:38 maas-node maas.import-images: [INFO] Writing boot image metadata and iSCSI targets.
Jul 20 22:10:38 maas-node maas.import-images: [INFO] Installing boot images snapshot /var/lib/maas/boot-resources/snapshot-20160720-164038
Jul 20 22:10:51 maas-node maas.import-images: [INFO] Updating boot image iSCSI targets.
Jul 20 22:10:52 maas-node maas.import-images: [INFO] Cleaning up old snapshots and cache.
Jul 20 22:10:52 maas-node maas.import-images: [INFO] Finished importing boot images.
Jul 20 22:25:45 maas-node maas.node: [INFO] slave1: Status transition from COMMISSIONING to FAILED_COMMISSIONING
Jul 20 22:25:45 maas-node maas.node: [ERROR] slave1: Marking node failed: Node operation 'Commissioning' timed out after 0:20:00.
Jul 20 22:26:05 maas-node maas.node_query: [INFO] slave1: Power is on.
Jul 20 22:30:37 maas-node maas.import-images: [INFO] Started importing boot images.
Jul 20 22:30:38 maas-node maas.import-images: [INFO] Finished importing boot images, the region does not have any new images.

clustered.log
========

2016-07-21 00:17:18+0530 [TFTP (UDP)] Datagram received from ('192.168.10.4', 49162): <RRQDatagram(filename=ubuntu/amd64/hwe-t/trusty/release/boot-initrd, mode=octet, options={'tsize': '0', 'blksize': '1408'})>
2016-07-21 00:17:18+0530 [-] RemoteOriginReadSession starting on 36546
2016-07-21 00:17:18+0530 [-] Starting protocol <tftp.bootstrap.RemoteOriginReadSession instance at 0x7f3018109a70>
2016-07-21 00:17:21+0530 [-] Timed during option negotiation process
2016-07-21 00:17:21+0530 [-] (UDP Port 40162 Closed)
2016-07-21 00:17:21+0530 [-] Stopping protocol <tftp.bootstrap.RemoteOriginReadSession instance at 0x7f30180cd248>
2016-07-21 00:17:32+0530 [RemoteOriginReadSession (UDP)] Final ACK received, transfer successful
2016-07-21 00:17:32+0530 [-] (UDP Port 36546 Closed)
2016-07-21 00:17:32+0530 [-] Stopping protocol <tftp.bootstrap.RemoteOriginReadSession instance at 0x7f3018109a70>

region.log
========
2016-07-21 02:35:30 [-] 192.168.10.2 - - [20/Jul/2016:21:05:30 +0000] "GET /MAAS/rpc/ HTTP/1.0" 200 296 "-" "provisioningserver.rpc.clusterservice.ClusterClientService"
2016-07-21 02:36:00 [-] 192.168.10.2 - - [20/Jul/2016:21:06:00 +0000] "GET /MAAS/rpc/ HTTP/1.0" 200 296 "-" "provisioningserver.rpc.clusterservice.ClusterClientService"
2016-07-21 02:36:30 [-] 192.168.10.2 - - [20/Jul/2016:21:06:29 +0000] "GET /MAAS/rpc/ HTTP/1.0" 200 296 "-" "provisioningserver.rpc.clusterservice.ClusterClientService"
2016-07-21 02:37:00 [-] 192.168.10.2 - - [20/Jul/2016:21:06:59 +0000] "GET /MAAS/rpc/ HTTP/1.0" 200 296 "-" "provisioningserver.rpc.clusterservice.ClusterClientService"
2016-07-21 02:37:30 [-] 192.168.10.2 - - [20/Jul/2016:21:07:29 +0000] "GET /MAAS/rpc/ HTTP/1.0" 200 296 "-" "provisioningserver.rpc.clusterservice.ClusterClientService"
2016-07-21 02:38:00 [-] 192.168.10.2 - - [20/Jul/2016:21:07:59 +0000] "GET /MAAS/rpc/ HTTP/1.0" 200 296 "-" "provisioningserver.rpc.clusterservice.ClusterClientService"
2016-07-21 02:38:30 [-] 192.168.10.2 - - [20/Jul/2016:21:08:29 +0000] "GET /MAAS/rpc/ HTTP/1.0" 200 296 "-" "provisioningserver.rpc.clusterservice.ClusterClientService"
2016-07-21 02:39:00 [-] 192.168.10.2 - - [20/Jul/2016:21:08:59 +0000] "GET /MAAS/rpc/ HTTP/1.0" 200 296 "-" "provisioningserver.rpc.clusterservice.ClusterClientService"
2016-07-21 02:39:30 [-] 192.168.10.2 - - [20/Jul/2016:21:09:29 +0000] "GET /MAAS/rpc/ HTTP/1.0" 200 296 "-" "provisioningserver.rpc.clusterservice.ClusterClientService"
2016-07-21 02:40:00 [-] 192.168.10.2 - - [20/Jul/2016:21:09:59 +0000] "GET /MAAS/rpc/ HTTP/1.0" 200 296 "-" "provisioningserver.rpc.clusterservice.ClusterClientService"


Which leads me to believe the machine, after set to COMMISSIONING, it tried to PXE boot, however, the commissioning process timeout out after 20 mins.

And I am not working behind a Proxy or behind a firewall.