Heiko's Admin Blog

I’m using Swift Grizzly Edition on Ubuntu 12.04 LTS.
To offer customers swift you need quotas and/or metric to check which resources your customers used last month. In this article I’ll show you how to setup the quota feature on the Grizzly version of Swift.

I’m assume that you have a running Swift cluster WITHOUT any errors .

Step1: Change your proxy-server.conf

You need to expand your pipeline with the container and account quota filter.

[pipeline:main]
pipeline = catch_errors healthcheck proxy-logging cache slo tempurl ratelimit authtoken keystone s3token container-quotas account-quotas proxy-logging proxy-server

Now you need to define this both filters at the end of your file

[filter:container-quotas]
use = egg:swift#container_quotas

[filter:account-quotas]
use = egg:swift#account_quotas

”’Attention”’
You need to put both filters after auth in the pipeline!

Add to your auth roles the ResellerAdmin role

[filter:keystone]
use = egg:swift#keystoneauth
operator_roles = admin, swiftoperator, ResellerAdmin

Finally swift-init proxy restart

Step2: Create Role

Only an user with the ResellerAdmin role can change the quotas of an account so you need to create a role with keystone and add this to your admin user

keystone role-create –name ResellerAdmin
keystone user-role-add –user admin –role ResellerAdmin –tenant [Customer Tenant]

Step3: Set the quotas

Now you must set the account quotas with the swift cli (or with http curl POST):

swift -V 2 -A https://KEYSTONE.URL:5000/v2.0 -U [CUSTOMER-TENANT]:admin -K [ADMIN-PW] post -m quota-bytes:100000

This are 97.6KB

Step4: Test your quotas

Create two testfiles to check if the quotas are set up correctly

dd if=/dev/zero of=test1.img count=10000 bs=700
10000+0 records in
10000+0 records out
7000000 bytes (7.0 MB) copied, 0.0181041 s, 387 MB/s

dd if=/dev/zero of=test1.img count=100 bs=700
100+0 records in
100+0 records out
70000 bytes (70 kB) copied, 0.000393225 s, 178 MB/s

Now try to upload these files

swift -V 2 -A https://KEYSTONE.URL:5000/v2.0 -U [CUSTOMER_TENANT]:[CUSTOMER-USER] -K [PW] upload test test1.img
Object PUT failed: https://xxxx/test/test2.img 413 Request Entity Too Large [first 60 chars of response] ….

swift -V 2 -A https://KEYSTONE.URL:5000/v2.0 -U [CUSTOMER_TENANT]:[CUSTOMER-USER] -K [PW] upload test test2.img
test2.img

swift -V 2 -A https://KEYSTONE.URL:5000/v2.0 -U [CUSTOMER_TENANT]:[CUSTOMER-USER] -K [PW] list test
test2.img

Congrats!

I’ve tried to enable backends in Cinder Grizzly Release. The creation of a volume has worked but not the attachement.

I got everything the same error on my storage nodes where cinder-volume runs.

ERROR [cinder.openstack.common.rpc.common] Returning exception no such option in group storage1: iscsi_helper to caller

It was a bug in the driver.py library. So after the changes by John it’s running normal without any errors.

Cheers

OpenStack have now a new stable release named Grizzly

You need to add the stable repository in Ubuntu 12.04! It’s probably not working on 12.10.

Add key

apt-key adv –recv-keys –keyserver keyserver.ubuntu.com 5EDB1B62EC4926EA

and create a new file

echo ‘deb http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/grizzly main’ > /etc/apt/sources.list.d/openstack-grizzly-stable.list
apt-get update

If you need more informations you can read the ubuntu cloud archives wiki

I’m running OpenStack Folsom release on Ubuntu 12.04 with Glance in combination of Swift.

I ran in a strange issue if I create a snapshot of a running instance but Nova (compute, api) seems to be works correctly.
I don’t get any errors in Keystone, Nova, Compute, Glance, Swift or any other service but I don’t see this snapshots on Horizon or via glance-client CLI. I can create and delete “normal” image types and see them on Horizon and glance-client.

To solve this behavior you should take a look into your glance configs! In both configs you will find this section

vim /etc/glance/glance-api.conf

[paste_deploy]
# Name of the paste configuration file that defines the available pipelines
# config_file = glance-api-paste.ini

# Partial name of a pipeline in your paste configuration file with the
# service name removed. For example, if your paste section name is
# [pipeline:glance-api-keystone], you would configure the flavor below
# as ‘keystone’.
flavor=keystone+cachemanagement

You will see i have enabled pipeline with keystone in combination with cachemanagement. You must enable keystone to see all snapshots on Horizon.

In addition you need to configure glance-registry too.

vim /etc/glance/glance-registry.conf

[paste_deploy]
# Name of the paste configuration file that defines the available pipelines
# config_file = glance-api-paste.ini

# Partial name of a pipeline in your paste configuration file with the
# service name removed. For example, if your paste section name is
# [pipeline:glance-api-keystone], you would configure the flavor below
# as ‘keystone’.
flavor=keystone

Note

Glance-registry doesn’t have cacheanagement pipeline. So you can only use Keystone

I got an error if I try to compile REE on Fedora17.

Error description

This error appears in google perf tools and it’s locate in source/distro/google-perftools-1.7/src/tcmalloc.cc
The error looks like

/bin/sh ./libtool –tag=CXX –mode=compile g++ -DHAVE_CONFIG_H -I. -I. -I./src -I./src -DNO_TCMALLOC_SAMPLES -pthread -DNDEBUG -Wall -Wwrite-strings -Woverloaded-virtual -Wno-sign-compare -fno-builtin-malloc -fno-builtin-free -fno-builtin-realloc -fno-builtin-calloc -fno-builtin-cfree -fno-builtin-memalign -fno-builtin-posix_memalign -fno-builtin-valloc -fno-builtin-pvalloc -DNO_FRAME_POINTER -march=x86-64 -mtune=generic -O2 -pipe -c -o libtcmalloc_minimal_la-tcmalloc.lo `test -f ‘src/tcmalloc.cc’ || echo ‘./’`src/tcmalloc.cc
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I. -I./src -I./src -DNO_TCMALLOC_SAMPLES -pthread -DNDEBUG -Wall -Wwrite-strings -Woverloaded-virtual -Wno-sign-compare -fno-builtin-malloc -fno-builtin-free -fno-builtin-realloc -fno-builtin-calloc -fno-builtin-cfree -fno-builtin-memalign -fno-builtin-posix_memalign -fno-builtin-valloc -fno-builtin-pvalloc -DNO_FRAME_POINTER -march=x86-64 -mtune=generic -O2 -pipe -c src/tcmalloc.cc -fPIC -DPIC -o .libs/libtcmalloc_minimal_la-tcmalloc.o
src/tcmalloc.cc:1672:54: error: conflicting declaration ‘void* (* __memalign_hook)(size_t, size_t, const void*)’
/usr/include/malloc.h:183:39: error: ‘__memalign_hook’ has a previous declaration as ‘void* (* volatile __memalign_hook)(size_t, size_t, const void*)’
make: *** [libtcmalloc_minimal_la-tcmalloc.lo] Error 1

I’ve found a solution on google groups.
Just add the ‘volotile’ hint on line 1672

vim source/distro/google-perftools-1.7/src/tcmalloc.cc
void *(* volatile __memalign_hook)(size_t, size_t, const void *) = MemalignOverride;

Now can you compile without errors.

Update

I got an error again

/usr/bin/gcc -I/opt/local/include -I. -I/opt/local/include -I../.. -I../../. -I../.././ext/dl -DHAVE_DLFCN_H -DHAVE_DLOPEN -DHAVE_DLCLOSE -DHAVE_DLSYM -DHAVE_DLERROR -I. -fPIC -g -O2 -fno-defer-pop -fno-omit-frame-pointer -c dl.c
In file included from dl.c:104:0:
callback.func:1:6: Fehler: expected »)« before »,« token
In file included from dl.c:104:0:
callback.func:78:33: Fehler: expected »)« before »(« token
callback.func:79:3: Warnung: Datendefinition hat keinen Typ oder Speicherklasse [standardmäßig aktiviert]
callback.func:79:24: Fehler: »proc« ist hier nicht deklariert (nicht in einer Funktion)
callback.func:79:39: Fehler: »argc« ist hier nicht deklariert (nicht in einer Funktion)
callback.func:79:45: Fehler: »argv« ist hier nicht deklariert (nicht in einer Funktion)
callback.func:82:1: Fehler: expected identifier or »(« before »}« token
dl.c:106:1: Fehler: expected »;«, »,« or »)« before »static«
make[1]: *** [dl.o] Fehler 1
make[1]: Leaving directory `/home/heiko/ruby-enterprise-1.8.7-2012.02/source/ext/dl’
make: *** [all] Fehler 1

You can execute the following commands to fix that issue

~> cd source/ext/dl
~> rm callback.func
~> touch callback.func
~> ruby mkcallback.rb >> callback.func
~> rm cbtable.func
~> touch cbtable.func
~> ruby mkcbtable.rb >> cbtable.func

I’ve a task to move all data from one to an other brick. So I’ll move all data from one to an other server but thats not possible with adding a new brick to each volume and rebalancing.

Used system

* 2 active servers with 10+ volumes
* replicated volumes
* Ubuntu 12.04 LTS on each system
* GlusterFS 3.2.5 (Clients 3.2.7)

Problem description

If you will change a complete host but you can’t adding simply a brick. You have set the number of replicas as you created on each volume. So you can’t simply change this but glusterfs has a good reason for this task.

Replace brick

You can replace on each volume the brick.

But first step by step. You need first add the new server to your trusted pool

~> gluster peer probe [new-server]

now you can replace the brick of your volume

~> gluster volume replace-brick [volname] [old-server]:/[path] [new-server]:/[path] start
replace-brick started successfully

you can check now the current status of the data transfer to the new host.

~> gluster volume replace-brick [volname] [old-server]:/[path] [new-server]:/[path] status
Number of files migrated = 1680 Migration complete

yeah migration are complete!
Now the last step

You need to tell all clients the new settings/changes in the vol-file

~> gluster volume replace-brick [volname] [old-server]:/[path] [new-server]:/[path] commit
replace-brick commit successful

Check DNS entry

If you’re using a DNS server you need to check the A-Record of the new server. The failover of glusterfs will check the bricks (DNS lookup) if no IP is present this brick will marked as offline.
Otherwise put your domain of the new server and the IP on each client in /etc/hosts!

Self heal

If you have big data sets and many writes you need trigger a self heal to sync your data on the new brick correctly

find [gluster-mount] -noleaf -print0 | xargs –null stat >/dev/null 2>/var/log/gluster/[volume]-selfheal.log

I had last week a stupid issue. I got errors like this

Client:remote operation failed: No such file or directory
… split brain ….

I’m using glusterFS 3.2.7 Server and the native clients.
I saw my brick didn’t have enough disk space the night before. So on the first brick it was enough space on the second not.
Now I got everytime on 2 or 3 different files errors like that above. But whats wrong ? ….

First)
Check the gfid and afr

root@storage2:~# getfattr -d -m . /var/exports/test/assets/9/tempfile.tmp
getfattr: Removing leading ‘/’ from absolute path names
# file: var/exports/test/assets/9/tempfile.tmp
trusted.afr.test-client-0=0sAAAAAQAAAAkAAAAA
trusted.afr.test-client-1=0sAAAAAAAAAAAAAAAA
trusted.gfid=0sVi+MZv2gQJOpfLUlVDl9CQ==
trusted.glusterfs.quota.4d7a53e2-f51b-40dc-afb1-72ccf38dbe5d.contri=0sAAAAAAAAEAA=

You see the file on brick 0 are broken. Check this on the second, third … too! The gfid must be equal on all other bricks!

second)
To fix you should delete the corrupted file on that brick and start a self heal!

root@storage2:~#rm /var/exports/test/assets/9/tempfile.tmp
# Start self heal
root@storage2:~#find -noleaf -print0 | xargs –null stat >/dev/null 2>/var/log/gluster/-selfheal.log

after that you should check the afr again and you will see all files are very well .

Note!!!
You should have Backups bevfre you play with your production storage

I’ve searched a long time to find a good workaround how I can use bridges with floating IP.

System

I’m using OpenStack Essex on Ubuntu 12.04 on a 3 host test system. One node is the controller node and other nodes are compute and network nodes.
On this compute nodes I’m running each instance with kvm.

At the moment T’m using FlatDHCP network-manager but this is not the best choice only the simpelst. On productive systems you should use vlan manager because you can allocate your networks to a project. Therefore the networks will not be shared by all projects.

But what’s my intention?

I’m using a layer 3 loadbalancer (keepalived) to balance all http/https requests or whatever. It doesn’t works with floating ip and routing (snat & dnat) so you’ll need a bridge with a unique MAC on each instance.

How does it works?

I’ve tested a small workaround to attach floating IP’s as a second NIC and a unique MAC to my instances.

First create a second network

nova-manage network create –label=public –fixed_range_v4=123.123.123.128/26 –multi_host=T –project_id= –bridge=br-ext –bridge_interface=eth0

Note!!! The fixed_range_v4 should represent your floating net

After that you should create your external bridge on your compute nodes

iface br-ext inet static
bridge_ports eth0
address 123.123.123.130
netmask 255.255.255.192
gateway 123.123.123.129

Enable your new network settings

/etc/init.d/networking reload

Now if you starts an instance you will see it gets directly 2 IP’s. One fixed and one floating. In real the floating ip are a second fixed and thats the small workaround.
For big stacks you should first play with that!!

Note!!!
It’s a workaround for OpenStack without Quantum

Hope someone helps!

Openstack can work with different virtualization methods like KVM, Xen, VMWare or LXC. The default virtualization method in Openstack is KVM and – at the moment – most features are implemented for it.
You can find a full list of suppored features here but pay attention there is no guarantee that this matrix is up-to-date.

Why LXC?

The key benefit of LXC is its small resource overhead. So if you spawn an instance with KVM you’ll end up with a RAM overhead because the KVM hypervisor simulates a bare metal machine with all needed things.
LXC is based on cgroups which are part of the Linux kernel since version 2.6.24! It’s a container system and in contrast to KVM or XEN not a full or paravirtualization system.
You will see the spawn time of LXC is awesome. An LXC OpenStack instance with a clean UEC 12.04 image was spawned in 6 secs!

Test Environment

I’m using for my test cases a three server setup. One server as controller node which includes nova-api, nova-volume, keystone, glance and nova-scheduler. On the other both servers are installed nova-compute (LXC) and nova-network.

The network service (nova-network) runs in multi_host=T mode for a better adept between production and testing environment.

I’m using on all servers Ubuntu 12.04 LTS and Openstack Essex (2012.3).

Installing

I’ll explain only the LXC steps to configure your existing OpenStack installation.

Let’s begin:
First you need to install the nova-package for LXC

apt-get install nova-compute-lxc lxctl

Note: lxctl are not required for running openstack compute node with lxc but it’s useful to debug some lxc things.

Check if your libvirt type is set correctly

cat /etc/nova/nova-compute.conf
–libvirt_type=lxc

Note: You can set the type in your nova.conf, too.
Nova uses the libvirt API to control the LXC container.

Image Types

I’ve tested a lot of image types and RAW disk format turned out to be the best option. As far as I know, Nova will convert a qcow2 image to raw format which didn’t work out for me due to errors while booting the instances.

You will find this kind of errors if you’re running your compute node in debug mode (–verbose && –debug). My instance boot failed after trying to mount the network block device of qemu.
In your compute log you will see a set of commands

qemu-nbd -c /dev/nbd15 /var/lib/nova/instances/instance-00000007/disk (makes image to block device)
mount /dev/nbd15 /var/lib/nova/instances/instance-00000007/rootfs (mount block dev as root fs for the container)

If you’re using qcow2 images you will get an error when mounting the image because qcow2 images don’t have a correct MBR to mount it, directly.
This might look somehow like this:

mount: you must specify the filesystem type

So if you encounter this problem, you can use RAW image types to work around it

Networking

Networking is the same as by KVM => Internal (fiexd_ips) will be create a bridge and public (floating) will be routed.
I’m using for my test cases FlatDHCP network manager.

After an instance boot, you see ‘veth0′ which is the instance NIC mapped to your bridge.

root@compute1 ~ # brctl show
bridge name bridge id STP enabled interfaces
br100 8000.1cbdb9da40e5 no eth1
veth0

Disable useless things

I saw on self made system compiled LXC packages (Ubuntu too) and LXC will make, similar to XEN, a bridge lxcbr0. You shouldn’t have trouble with this but I prefer to disable this stuff, for reasons of clearity

vim /etc/default/lxc

USE_LXC_BRIDGE=”false”

Let’s take a loot how you can improve your performnce, exactly your memory management of your ruby stack. Ruby Enterprise Editon have modified the default settings of thr normal ruby interpreter but you can overwrite this settings too.

You will need to play with settings for your own optimum.

Create a wrapper file with 755 permissions and root owner.

sudo touch /opt/ruby_wrapper
#!/bin/sh
export RUBY_HEAP_MIN_SLOTS=1500000
export RUBY_HEAP_SLOTS_INCREMENT=100000
export RUBY_HEAP_SLOTS_GROWTH_FACTOR=1
export RUBY_GC_MALLOC_LIMIT=59000000
export RUBY_HEAP_FREE_MIN=30000

exec “/usr/local/bin/ruby” “$@”

Il spare myself to explain all ENV variables because it’s good documented. You should reed first here before you change your settings!!!

All right now install passenger

sudo gem install passenger

Now change the Ruby interpreter path in (if you use apache2 mod_passenger module)

#~>vi /etc/apache2/mods-available/passenger.load
LoadModule passenger_module /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.12/ext/apache2/mod_passenger.so
PassengerRoot /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.12
PassengerRuby /opt/ruby_wrapper

#~>service restart apache2

Et voilá Passenger will use the new Settings all the time.

For futher informtaions about the Ruby heap stack you can read this site.

Note
Since Ruby 1.9 you don’t need different heap, exactly GC, settings because the GC in ruby1.9 are much more faster and really stable.