puppet modules problem with liberty_multinodes scenario

[kartik894]

Hi,

I am getting the following error upon executing the liberty_multinodes scenario:

--> Execute task puppet:hiera:generate
mkdir -p scenarios/liberty_multinodes/hiera/generated/nodes
cp scenarios/liberty_multinodes/hiera/templates/nodes/puppetserver.yaml scenarios/liberty_multinodes/hiera/generated/nodes/stremi-11.reims.grid5000.fr.yaml
cp scenarios/liberty_multinodes/hiera/templates/nodes/controller.yaml scenarios/liberty_multinodes/hiera/generated/nodes/stremi-20.reims.grid5000.fr.yaml
cp scenarios/liberty_multinodes/hiera/templates/nodes/storage.yaml scenarios/liberty_multinodes/hiera/generated/nodes/stremi-21.reims.grid5000.fr.yaml
cp scenarios/liberty_multinodes/hiera/templates/nodes/network.yaml scenarios/liberty_multinodes/hiera/generated/nodes/stremi-24.reims.grid5000.fr.yaml
cp scenarios/liberty_multinodes/hiera/templates/nodes/compute.yaml scenarios/liberty_multinodes/hiera/generated/nodes/stremi-27.reims.grid5000.fr.yaml
cp scenarios/liberty_multinodes/hiera/templates/nodes/compute.yaml scenarios/liberty_multinodes/hiera/generated/nodes/stremi-35.reims.grid5000.fr.yaml
cp scenarios/liberty_multinodes/hiera/templates/nodes/compute.yaml scenarios/liberty_multinodes/hiera/generated/nodes/stremi-36.reims.grid5000.fr.yaml
cp scenarios/liberty_multinodes/hiera/templates/common.yaml scenarios/liberty_multinodes/hiera/generated/
--> Execute task puppet:server:bootstrap
rm -rf /tmp/xp5k_$USER/xp5k-openstack/bootstrap
mkdir -p /tmp/xp5k_$USER/xp5k-openstack/bootstrap/{hieradata,modules}
rsync -a --delete provision/puppet/modules/ /tmp/xp5k_$USER/xp5k-openstack/bootstrap/modules 
rsync -a --delete scenarios/liberty_multinodes/hiera/generated/ /tmp/xp5k_$USER/xp5k-openstack/bootstrap/hieradata 
cd /tmp/xp5k_$USER/xp5k-openstack && tar -cf - bootstrap/ | ssh root@stremi-11.reims.grid5000.fr 'cd /etc/puppetlabs/code/environments && tar xf -'
Connected to stremi-11.reims.grid5000.fr...
[command][stremi-11.reims.grid5000.fr] eatmydata /opt/puppetlabs/bin/puppet apply --environment bootstrap -e 'include xp,xp::locales,xp::puppet::server'
[stdout][stremi-11.reims.grid5000.fr] Notice: Compiled catalog for stremi-11.reims.grid5000.fr in environment bootstrap in 2.24 seconds
[stdout][stremi-11.reims.grid5000.fr] Notice: /Stage[main]/Locales/File[/etc/default/locale]/content: content changed '{md5}ec5364242858c6bb38b023522e4d45b4' to '{md5}9b5e64b0b8a2be2f6ff9bbfa3cc060fb'
[stdout][stremi-11.reims.grid5000.fr] Notice: /Stage[main]/Puppet::Server/Package[puppetserver]/ensure: created
[stdout][stremi-11.reims.grid5000.fr] Notice: /Stage[main]/Puppet::Server/Package[puppetdb-termini]/ensure: created
[stdout][stremi-11.reims.grid5000.fr] Notice: /Stage[main]/Puppet::Server/Augeas[puppetdb_terminus]/returns: executed successfully
[stdout][stremi-11.reims.grid5000.fr] Notice: /Stage[main]/Puppet::Server/Augeas[puppet_reports]/returns: executed successfully
[stdout][stremi-11.reims.grid5000.fr] Notice: /Stage[main]/Puppet::Server/File[/etc/puppetlabs/puppet/puppetdb.conf]/ensure: created
[stdout][stremi-11.reims.grid5000.fr] Notice: /Stage[main]/Puppet::Server/Augeas[puppetdb_conf]/returns: executed successfully
[stdout][stremi-11.reims.grid5000.fr] Notice: /Stage[main]/Puppet::Server/File[/etc/puppetlabs/puppet/autosign.conf]/ensure: defined content as '{md5}381ea75a0211f81b38ca81daf442f3da'
[stdout][stremi-11.reims.grid5000.fr] Notice: /Stage[main]/Puppet::Server/File[/etc/puppetlabs/puppet/routes.yaml]/ensure: defined content as '{md5}9f4da661e316690b5599ec2f2c9d60a4'
[stdout][stremi-11.reims.grid5000.fr] Notice: /Stage[main]/Puppet::Server/Service[puppetserver]/ensure: ensure changed 'stopped' to 'running'
[stdout][stremi-11.reims.grid5000.fr] Notice: /Stage[main]/Puppet::Puppetdb/Package[puppetdb]/ensure: created
[stdout][stremi-11.reims.grid5000.fr] Notice: /Stage[main]/Puppet::Puppetdb/Service[puppetdb]/ensure: ensure changed 'stopped' to 'running'
[stdout][stremi-11.reims.grid5000.fr] Notice: /Stage[main]/Xp::Puppet::Server/File[/etc/puppetlabs/code/environments/production/modules-openstack]/ensure: created
[stdout][stremi-11.reims.grid5000.fr] Notice: /Stage[main]/Xp::Puppet::Server/File[/etc/puppetlabs/code/environments/production/modules-scenario]/ensure: created
[stdout][stremi-11.reims.grid5000.fr] Notice: /Stage[main]/Xp::Puppet::Server/File[/etc/puppetlabs/code/environments/production/manifests/site.pp]/ensure: defined content as '{md5}c67bcd23d76cce84112decfdc3ef7bae'
[stdout][stremi-11.reims.grid5000.fr] Notice: /Stage[main]/Xp::Puppet::Server/Puppet::Server::Environment[production]/File[/etc/puppetlabs/code/environments/production/environment.conf]/content: content changed '{md5}2ea922d00a1446132be79b24cb15acf1' to '{md5}2317ffee9ed3837100f918551ae69e65'
[stdout][stremi-11.reims.grid5000.fr] Notice: Applied catalog in 245.64 seconds
[command][stremi-11.reims.grid5000.fr] sync
---- stremi-11.reims.grid5000.fr -----------------------------------------------

Closing ssh connections...
--> Execute task puppet:modules:get
r10k puppetfile install -v
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/aodh
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/ceilometer
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/cinder
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/designate
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/glance
ERROR    -> Command exited with non-zero exit code:
Command: git clone https://git.openstack.org/openstack/puppet-glance scenarios/liberty_multinodes/puppet/modules-openstack/glance --reference /home/ksathyanarayanan/.r10k/git/https---git.openstack.org-openstack-puppet-glance
Stderr:
Cloning into 'scenarios/liberty_multinodes/puppet/modules-openstack/glance'...
fatal: unable to access 'https://git.openstack.org/openstack/puppet-glance/': GnuTLS recv error (-110): The TLS connection was non-properly terminated.
Exit code: 128
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/gnocchi
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/heat
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/horizon
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/ironic
ERROR    -> Command exited with non-zero exit code:
Command: git --git-dir /home/ksathyanarayanan/.r10k/git/https---git.openstack.org-openstack-puppet-ironic fetch --prune
Stderr:
From https://git.openstack.org/openstack/puppet-ironic
 x [deleted]         (none)     -> stable/kilo
error: RPC failed; result=35, HTTP code = 0
fatal: The remote end hung up unexpectedly
Exit code: 128
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/keystone
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/manila
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/murano
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/neutron
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/nova
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/openstack_extras
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/openstacklib
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/sahara
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/swift
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/tempest
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/trove
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/tuskar
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/vswitch
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/apache
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/apt
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/concat
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/corosync
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/dns
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/firewall
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/inifile
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/memcached
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/mongodb
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/mysql
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/postgresql
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/powerdns
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/python
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/qpid
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/rabbitmq
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/rsync
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/staging
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/stdlib
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/sysctl
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/vcsrepo
INFO     -> Updating module scenarios/liberty_multinodes/puppet/modules-openstack/xinetd
rake aborted!
Command failed with status (1): [r10k puppetfile install -v...]
tasks/puppet.rb:74:in `block (3 levels) in <top (required)>'
/home/ksathyanarayanan/xp5k-openstack/.gem/gems/xp5k-0.0.14/lib/xp5k/rake/task.rb:9:in `execute'
/home/ksathyanarayanan/xp5k-openstack/Rakefile:102:in `block (2 levels) in <top (required)>'
/home/ksathyanarayanan/xp5k-openstack/Rakefile:101:in `each'
/home/ksathyanarayanan/xp5k-openstack/Rakefile:101:in `block in <top (required)>'
/home/ksathyanarayanan/xp5k-openstack/.gem/gems/xp5k-0.0.14/lib/xp5k/rake/task.rb:9:in `execute'
Tasks: TOP => run
(See full trace by running task with --trace)

Is there a problem with the puppet server?

[msimonin]

A very first guess is that you have trouble with github servers:

error: RPC failed; result=35, HTTP code = 0
fatal: The remote end hung up unexpectedly
Exit code: 128

Maybe it will succeed if you try again.

[kartik894]

I am getting the same error since yesterday. Have tried it on different sites too. Liberty_starter_kit is working well though.

[kartik894]

Even with grid5000:clean command I get the following error:

$ rake grid5000:clean
--> Execute task grid5000:clean
Clean all Grid'5000 running jobs...
I, [2016-07-19T18:22:04.128258 #5969]  INFO -- : Job #192650 deleted !
Removing interface cache file...
rm scenarios/liberty_multinodes/.interfaces.yml
rake aborted!
Errno::ENOENT: No such file or directory @ unlink_internal - scenarios/liberty_multinodes/.interfaces.yml
scenarios/liberty_multinodes/tasks/interfaces.rb:138:in `block (2 levels) in <top (required)>'
/home/ksathyanarayanan/xp5k-openstack/.gem/gems/xp5k-0.0.14/lib/xp5k/rake/task.rb:9:in `execute'
Tasks: TOP => grid5000:clean

[msimonin]

liberty_starter_kit and liberty_multinodes use the same Puppetfile.
I guess the reason why it's working with starter_kit is that you already have all the puppet modules downloaded so they aren't retrieved from github.
As a workaround you can copy the puppet modules from liberty_starter_kit to liberty_multinodes.

I'm still thinking it's a problem with github.

Concerning the error shown in grid5000:clean, this is uncorrelated and has to be fixed in the future.

[kartik894]

copying the puppet modules helped. Thanks!