Cloud Foundry: Push Fails, Cannot Allocate memory!

We have an in-house Cloud Foundry setup on Openstack for our developments teams. Last week on the our environment we had application pushes failing strangely with the following errors.

Done uploading               
FAILED
Error uploading application.
An unknown error occurred.

And nothing else to start our investigation with, but luckily we looked at the Cloud Controller logs first, and found this.

Job VCAP::CloudController::Jobs::ExceptionCatchingJob (id=346) FAILED (6 prior attempts) with Errno::ENOMEM: Cannot allocate memory - zip -q -r --symlinks /var/vcap/data/cloud_controller_ng/tmp/safezipper20150724-19425-ub32ql/package.zip

Cannot allocate memory is a Ruby error to denote there is not enough RAM. A quick at the free showed us this.

(root)$ free
  total       used       free     shared    buffers     cached
Mem:       2056468    1960548      95920          0       1148      89396
-/+ buffers/cache:    1870004     186464
Swap:            0          0          0

Now that was not looking good with close 95% of the system memory occupied. Usually the following commands should work to free up some RAM.

(root)$ sync
(root)$ # To free pagecache
(root)$ echo 1 > /proc/sys/vm/drop_caches
(root)$ # To free dentries and inodes
(root)$ echo 2 > /proc/sys/vm/drop_caches
(root)$ # To free pagecache, dentries and inodes
(root)$ echo 3 > /proc/sys/vm/drop_caches

Now running free for some happy news, did not show any significant change.

(root)$ free
           total       used       free     shared    buffers     cached
Mem:       2056468    1877436     179032          0        128       7564
-/+ buffers/cache:    1869744     186724
Swap:            0          0          0

Though there was a drop in usage, it was not significant enough to run away with.

Swap to the Rescue

Noticed that instance Flavors in the Openstack setup had no Swap provide to the Flavor type. Though this is the norm, it would have saved a lot work if it had been setup during flavor definition.

So now, we had to create the Swap manually. First create the file to be used as swap.

(root)$ dd if=/dev/zero of=/swapfile bs=1024 count=256k
262144+0 records in
262144+0 records out
268435456 bytes (268 MB) copied, 4.74066 s, 56.6 MB/s

This creates a NULL, /dev/zero, ridden file, /swapfile, of size approximately 256MB. Not much, but enough. Then we formatted it as swap.

(root)$ mkswap /swapfile
mkswap: /swapfile: warning: don't erase bootbits sectors
        on whole disk. Use -f to force.
Setting up swapspace version 1, size = 262140 KiB
no label, UUID=xxxxx-xxxx-...

Forget about the warning. To turn on the swap, issued the following the following command.

(root)$ swapon /swapfile

For the protection of this location, we make it write-only-by-root.

(root)$ chown root:root /swapfile
(root)$ chmod 0600 /swapfile

It would be better to define the swapiness to enhance performance, we did this with following commands.

(root)$ echo 10 | tee /proc/sys/vm/swappiness
10
(root)$ echo vm.swappiness = 10 | tee -a /etc/sysctl.conf
vm.swappiness = 10

Adding swap did help with successful push commands, and having our applications successfully deployed.

Takeaways

  • The Cloud Controller Machine should be given enough memory.
  • Add swap to every Flavor of the Openstack instances, so even if the RAM overflows, Swap is there to rescue.
Show Comments