[Ltsp-discuss] Is it possible to run ltsp-pnp in an openvz container?

[Robert Mavrinac]

I have an existing LTSP cluster running on 4 Ubuntu 12.04 openvz
containers, one for root/load-balancing/dhcp, the other three for app
servers.

Generally, I have had no issues overcoming some of the limitations of
the container (for example, indirectly supporting an nfs4 home directory
by using a simfs mount, or nfs-kernel-server by having a dummy mount on
CT0 and starting exportfs,rpcbind and nfs-kernel-server in the
container), and I have had this cluster working for years.

Now I'm looking at upgrading to Ubuntu 14.04. Everyone seems to say that
ltsp-pnp is simpler and that ltsp-cluster is no longer supported.

The load-balancing feature is important to our site, as is running the
servers in containers.

I quickly followed the steps outlined in a recent message 41981
(https://www.mail-archive.com/ltsp-***@lists.sourceforge.net/msg41981.html).
But clearly this fails on "tsp-update-image -c /" precisely because the
server is a container.

Any suggestions?

[Ben Green]

Post by Robert Mavrinac
I quickly followed the steps outlined in a recent message 41981
But clearly this fails on "tsp-update-image -c /" precisely because the
server is a container.
Any suggestions?

We run LTSP in Linux-Vserver containers. You need to set a few options
from the host, and NFS is more tricky, we stick to NBD, but basically
it's totally possible.

What problem do you get with OpenVZ?

Cheers,
Ben

[Robert Mavrinac]

Post by Ben Green

Post by Robert Mavrinac
I quickly followed the steps outlined in a recent message 41981
But clearly this fails on "tsp-update-image -c /" precisely because the
server is a container.
Any suggestions?

We run LTSP in Linux-Vserver containers. You need to set a few options
from the host, and NFS is more tricky, we stick to NBD, but basically
it's totally possible.
What problem do you get with OpenVZ?

Are you using chroot images? That I can do. The problem with OpenVZ is
OS-level virtualization, just the same way it might be for VServer.
Obviously from the output, I'm running this on a Proxmox server.

***@server:/# ltsp-update-image -c /
modprobe: ERROR: ../libkmod/libkmod.c:507
kmod_lookup_alias_from_builtin_file() could not open builtin file
'/lib/modules/2.6.32-31-pve/modules.builtin.bin'
modprobe: FATAL: Module overlayfs not found.
modprobe: ERROR: ../libkmod/libkmod.c:507
kmod_lookup_alias_from_builtin_file() could not open builtin file
'/lib/modules/2.6.32-31-pve/modules.builtin.bin'
modprobe: FATAL: Module aufs not found.
No overlayfs or aufs support detected

If you don't have this problem, I need to look at VServer.

[Vagrant Cascadian]

Post by Robert Mavrinac
Are you using chroot images? That I can do. The problem with OpenVZ is
OS-level virtualization, just the same way it might be for VServer.
Obviously from the output, I'm running this on a Proxmox server.
modprobe: ERROR: ../libkmod/libkmod.c:507
kmod_lookup_alias_from_builtin_file() could not open builtin file
'/lib/modules/2.6.32-31-pve/modules.builtin.bin'
modprobe: FATAL: Module overlayfs not found.
modprobe: ERROR: ../libkmod/libkmod.c:507
kmod_lookup_alias_from_builtin_file() could not open builtin file
'/lib/modules/2.6.32-31-pve/modules.builtin.bin'
modprobe: FATAL: Module aufs not found.
No overlayfs or aufs support detected
If you don't have this problem, I need to look at VServer.

Looks like you need to load the overlayfs or aufs modules. It's not
uncommon for container technologies to disallow loading modules by
default, as that can compromise the security of the host machine. If you
manually load overlayfs or aufs from the host server, it might work.

live well,
vagrant

[Robert Mavrinac]

Post by Vagrant Cascadian

Post by Robert Mavrinac
Are you using chroot images? That I can do. The problem with OpenVZ is
OS-level virtualization, just the same way it might be for VServer.
Obviously from the output, I'm running this on a Proxmox server.
modprobe: ERROR: ../libkmod/libkmod.c:507
kmod_lookup_alias_from_builtin_file() could not open builtin file
'/lib/modules/2.6.32-31-pve/modules.builtin.bin'
modprobe: FATAL: Module overlayfs not found.
modprobe: ERROR: ../libkmod/libkmod.c:507
kmod_lookup_alias_from_builtin_file() could not open builtin file
'/lib/modules/2.6.32-31-pve/modules.builtin.bin'
modprobe: FATAL: Module aufs not found.
No overlayfs or aufs support detected
If you don't have this problem, I need to look at VServer.

Looks like you need to load the overlayfs or aufs modules. It's not
uncommon for container technologies to disallow loading modules by
default, as that can compromise the security of the host machine. If you
manually load overlayfs or aufs from the host server, it might work.
live well,
vagrant

Well, I'm not going to rebuild the Proxmox VE kernel to include aufs
support. I'm going test this with a Linux-VServer KVM. It looks like I
can get that to work, so I may be migrating all of my openvz containers
very soon.

- Rob

[Robert Mavrinac]

Post by Vagrant Cascadian

Post by Robert Mavrinac
Are you using chroot images? That I can do. The problem with OpenVZ is
OS-level virtualization, just the same way it might be for VServer.
Obviously from the output, I'm running this on a Proxmox server.
modprobe: ERROR: ../libkmod/libkmod.c:507
kmod_lookup_alias_from_builtin_file() could not open builtin file
'/lib/modules/2.6.32-31-pve/modules.builtin.bin'
modprobe: FATAL: Module overlayfs not found.
modprobe: ERROR: ../libkmod/libkmod.c:507
kmod_lookup_alias_from_builtin_file() could not open builtin file
'/lib/modules/2.6.32-31-pve/modules.builtin.bin'
modprobe: FATAL: Module aufs not found.
No overlayfs or aufs support detected
If you don't have this problem, I need to look at VServer.

Looks like you need to load the overlayfs or aufs modules. It's not
uncommon for container technologies to disallow loading modules by
default, as that can compromise the security of the host machine. If you
manually load overlayfs or aufs from the host server, it might work.

I've opted to recompile the Proxmox kernel to include aufs support.
Currently, I've patched the Proxmox sources to have aufs compile into
the kernel using the aufs2-standalone source for the 2.6.32 kernel. I'm
still working on getting it to build a kernel module.

Are the default settings for aufs in the kernel config sufficient, or
should some of them be changed?

Post by Vagrant Cascadian
live well,
vagrant
------------------------------------------------------------------------------
Slashdot TV.
Video for Nerds. Stuff that matters.
http://tv.slashdot.org/
_____________________________________________________________________
https://lists.sourceforge.net/lists/listinfo/ltsp-discuss
For additional LTSP help, try #ltsp channel on irc.freenode.net

- Rob

[Robert Mavrinac]

Post by Vagrant Cascadian
Looks like you need to load the overlayfs or aufs modules. It's not
uncommon for container technologies to disallow loading modules by
default, as that can compromise the security of the host machine. If you
manually load overlayfs or aufs from the host server, it might work.

I still haven't gotten any response on what the preferred kernel options
for AUFS should be for LTSP.

I have built AUFS as a module, because I don't necessarily want it
available to all containers as a feature. If anyone would kindly provide
their suggestions for the following options, I'd appreciate it. I made
the following choices, but I may not need all of them:

Aufs (Advanced multi layered unification filesystem) support
(AUFS_FS) [N/m/y/?] (NEW) m
Maximum number of branches

Post by Vagrant Cascadian
1. 127 (AUFS_BRANCH_MAX_127) (NEW)

2. 511 (AUFS_BRANCH_MAX_511) (NEW)
3. 1023 (AUFS_BRANCH_MAX_1023) (NEW)
4. 32767 (AUFS_BRANCH_MAX_32767) (NEW)
choice[1-4?]: 1
Detect direct branch access (bypassing aufs) (AUFS_HNOTIFY) [N/y/?]
(NEW) y
method

Post by Vagrant Cascadian
1. fsnotify (AUFS_HFSNOTIFY) (NEW)

2. inotify (DEPRECATED) (AUFS_HINOTIFY) (NEW)
choice[1-2]: 1
NFS-exportable aufs (AUFS_EXPORT) [N/y/?] (NEW) y
Readdir in userspace (AUFS_RDU) [N/y/?] (NEW)
support for /proc/maps and lsof(1) (AUFS_PROC_MAP) [N/y/?] (NEW)
Respect the attributes (mtime/ctime mainly) of special files
(AUFS_SP_IATTR) [N/y/?] (NEW)
Show whiteouts (AUFS_SHWH) [N/y/?] (NEW)
Ramfs (initramfs/rootfs) as an aufs branch (AUFS_BR_RAMFS) [N/y/?]
(NEW) y
Fuse fs as an aufs branch (AUFS_BR_FUSE) [N/y/?] (NEW) y
Hfsplus as an aufs branch (AUFS_BR_HFSPLUS) [Y/n/?] (NEW)
Debug aufs (AUFS_DEBUG) [N/y/?] (NEW)


Summary of what I have done so far:

The latest LTSP servers can be run as OpenVZ containers, but obviously
require AUFS support in the hardware node (HN or CT0) kernel as pointed
out above. I don't really want to give up PVE. The Proxmox Virtual
Environment provides complete server virtualization management for
OpenVZ containers that we've adapted to; its kernel currently uses the
OpenVZ Kernel from sources listed below.

The Proxmox kernel sources are available from:

https://github.com/proxmox/pve-kernel-2.6.32

pve-kernel-2.6.36-32

which includes sources and patches from

http://download.openvz.org/kernel/branches/rhel6-2.6.32/042stab093.4/

vzkernel-2.6.32-042stab093.4.src.rpm
patch-042stab093


The AUFS 2.1-32 sources are available from:

http://sourceforge.net/p/aufs/aufs2-standalone/ci/aufs2.1-32/tree/

git clone git://git.code.sf.net/p/aufs/aufs2-standalone
aufs2-standalone.git
cd aufs2-standalone.git
git checkout aufs2.1-32


Here is a summary of what I've done to replace the Proxmox kernel with a
custom one.

Both vzkernel-2.6.32-042stab093.4.src.rpm and patch-042stab093 patch
linux-2.6.32, change symbol names and relocate functions. This, of
course, requires modifying the distributed patches for AUFS 2.1 to
successfully rebuild kernel. I've created a complete aufs2.1-32 patch
that has been created specifically to rebuild the Proxmox 2.6.32 kernel
packages. Note that this custom patch includes all files required to
either compile aufs2.1 into the kernel or as a module. All of
aufs2-base.patch, aufs2-kbuild.patch, aufs2-standalone.patch,
proc_map.patch and loopback.patch have been included into this patch. I
will continue to post new patches as the kernel gets updated

My current pve-kernel-2.6.32+aufs2.1-32.patch can be found on the
owncloud link below, but will be updated to reflect the suggestions
offered for the question at the beginning of this message.

https://owncloud.cs.uwindsor.ca/public.php?service=files&t=83cdb7f8067adaa046c3b3579cc47fdd

To rebuild the kernel:

cd /your/src
apt-get install kernel-package lintian
git clone https://github.com/proxmox/pve-kernel-2.6.32
pve-kernel-2.6.32.git
cd pve-kernel-2.6.32.git
patch -p1 < /your/downloads/pve-kernel-2.6.32+aufs2.1-32.patch

The root changelog.Debian is patched with an arbitrarily large version
epoch so this kernel doesn't get overwritten by an update. You should
customize this changelog.Debian for your site before rebuilding the kernel.

Finally,

make

Install the kernel and headers packages created.

[Vagrant Cascadian]

Post by Robert Mavrinac

Post by Vagrant Cascadian
Looks like you need to load the overlayfs or aufs modules. It's not
uncommon for container technologies to disallow loading modules by
default, as that can compromise the security of the host machine. If you
manually load overlayfs or aufs from the host server, it might work.

I still haven't gotten any response on what the preferred kernel options
for AUFS should be for LTSP.

Not sure which options are needed, but it's worked fine with standard
kernels built in Debian for at least the several years. Here's a grep of
the kernel config options:

$ grep AUFS /boot/config-3.14-2-amd64
CONFIG_AUFS_FS=m
CONFIG_AUFS_BRANCH_MAX_127=y
# CONFIG_AUFS_BRANCH_MAX_511 is not set
# CONFIG_AUFS_BRANCH_MAX_1023 is not set
# CONFIG_AUFS_BRANCH_MAX_32767 is not set
CONFIG_AUFS_SBILIST=y
# CONFIG_AUFS_HNOTIFY is not set
CONFIG_AUFS_EXPORT=y
CONFIG_AUFS_INO_T_64=y
# CONFIG_AUFS_RDU is not set
# CONFIG_AUFS_SP_IATTR is not set
# CONFIG_AUFS_SHWH is not set
# CONFIG_AUFS_BR_RAMFS is not set
# CONFIG_AUFS_BR_FUSE is not set
CONFIG_AUFS_BR_HFSPLUS=y
CONFIG_AUFS_BDEV_LOOP=y
# CONFIG_AUFS_DEBUG is not set


Hope that's helpful.


live well,
vagrant

[Robert Mavrinac]

Post by Vagrant Cascadian

Post by Robert Mavrinac

Post by Vagrant Cascadian
Looks like you need to load the overlayfs or aufs modules. It's not
uncommon for container technologies to disallow loading modules by
default, as that can compromise the security of the host machine. If you
manually load overlayfs or aufs from the host server, it might work.

I still haven't gotten any response on what the preferred kernel options
for AUFS should be for LTSP.

Not sure which options are needed, but it's worked fine with standard
kernels built in Debian for at least the several years. Here's a grep of
$ grep AUFS /boot/config-3.14-2-amd64
CONFIG_AUFS_FS=m
CONFIG_AUFS_BRANCH_MAX_127=y
# CONFIG_AUFS_BRANCH_MAX_511 is not set
# CONFIG_AUFS_BRANCH_MAX_1023 is not set
# CONFIG_AUFS_BRANCH_MAX_32767 is not set
CONFIG_AUFS_SBILIST=y
# CONFIG_AUFS_HNOTIFY is not set
CONFIG_AUFS_EXPORT=y
CONFIG_AUFS_INO_T_64=y
# CONFIG_AUFS_RDU is not set
# CONFIG_AUFS_SP_IATTR is not set
# CONFIG_AUFS_SHWH is not set
# CONFIG_AUFS_BR_RAMFS is not set
# CONFIG_AUFS_BR_FUSE is not set
CONFIG_AUFS_BR_HFSPLUS=y
CONFIG_AUFS_BDEV_LOOP=y
# CONFIG_AUFS_DEBUG is not set

This is exactly what I needed. Thanks.

I'm adding a config diff to the patch to reflect these kernel config
options.

Setting the epoch=99 was surely overkill - that value can just be a
small non-zero integer. I suppose it may have just been sufficient to
hold the package to prevent premature updates. I didn't want to change
the package version or debian revision, but did want the changelog to
document the addition of AUFS.

Post by Vagrant Cascadian
Hope that's helpful.
live well,
vagrant
------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce.
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_____________________________________________________________________
https://lists.sourceforge.net/lists/listinfo/ltsp-discuss
For additional LTSP help, try #ltsp channel on irc.freenode.net

[Robert Mavrinac]

For anyone else following this, I neglected to mention that you do need
to run 'make install_headers" from the aufs2-standalone.git directory in
order to build aufs2-util.

git clone git://git.code.sf.net/p/aufs/aufs2-standalone
aufs2-standalone.git
cd aufs2-standalone.git
git checkout aufs2.1-32
make install_headers

git clone git://git.code.sf.net/p/aufs/aufs2-util aufs2-util.git
cd aufs2-util.git
git checkout aufs2.1
make install

Although I've managed to get this part done, I haven't been able to get
a container to use aufs yet.

Post by Robert Mavrinac

Post by Vagrant Cascadian
Looks like you need to load the overlayfs or aufs modules. It's not
uncommon for container technologies to disallow loading modules by
default, as that can compromise the security of the host machine. If you
manually load overlayfs or aufs from the host server, it might work.

I still haven't gotten any response on what the preferred kernel
options for AUFS should be for LTSP.
I have built AUFS as a module, because I don't necessarily want it
available to all containers as a feature. If anyone would kindly
provide their suggestions for the following options, I'd appreciate
Aufs (Advanced multi layered unification filesystem) support
(AUFS_FS) [N/m/y/?] (NEW) m
Maximum number of branches

Post by Vagrant Cascadian
1. 127 (AUFS_BRANCH_MAX_127) (NEW)

2. 511 (AUFS_BRANCH_MAX_511) (NEW)
3. 1023 (AUFS_BRANCH_MAX_1023) (NEW)
4. 32767 (AUFS_BRANCH_MAX_32767) (NEW)
choice[1-4?]: 1
Detect direct branch access (bypassing aufs) (AUFS_HNOTIFY)
[N/y/?] (NEW) y
method

Post by Vagrant Cascadian
1. fsnotify (AUFS_HFSNOTIFY) (NEW)

2. inotify (DEPRECATED) (AUFS_HINOTIFY) (NEW)
choice[1-2]: 1
NFS-exportable aufs (AUFS_EXPORT) [N/y/?] (NEW) y
Readdir in userspace (AUFS_RDU) [N/y/?] (NEW)
support for /proc/maps and lsof(1) (AUFS_PROC_MAP) [N/y/?] (NEW)
Respect the attributes (mtime/ctime mainly) of special files
(AUFS_SP_IATTR) [N/y/?] (NEW)
Show whiteouts (AUFS_SHWH) [N/y/?] (NEW)
Ramfs (initramfs/rootfs) as an aufs branch (AUFS_BR_RAMFS) [N/y/?]
(NEW) y
Fuse fs as an aufs branch (AUFS_BR_FUSE) [N/y/?] (NEW) y
Hfsplus as an aufs branch (AUFS_BR_HFSPLUS) [Y/n/?] (NEW)
Debug aufs (AUFS_DEBUG) [N/y/?] (NEW)
The latest LTSP servers can be run as OpenVZ containers, but obviously
require AUFS support in the hardware node (HN or CT0) kernel as
pointed out above. I don't really want to give up PVE. The Proxmox
Virtual Environment provides complete server virtualization management
for OpenVZ containers that we've adapted to; its kernel currently uses
the OpenVZ Kernel from sources listed below.
https://github.com/proxmox/pve-kernel-2.6.32
pve-kernel-2.6.36-32
which includes sources and patches from
http://download.openvz.org/kernel/branches/rhel6-2.6.32/042stab093.4/
vzkernel-2.6.32-042stab093.4.src.rpm
patch-042stab093
http://sourceforge.net/p/aufs/aufs2-standalone/ci/aufs2.1-32/tree/
git clone git://git.code.sf.net/p/aufs/aufs2-standalone
aufs2-standalone.git
cd aufs2-standalone.git
git checkout aufs2.1-32
Here is a summary of what I've done to replace the Proxmox kernel with
a custom one.
Both vzkernel-2.6.32-042stab093.4.src.rpm and patch-042stab093 patch
linux-2.6.32, change symbol names and relocate functions. This, of
course, requires modifying the distributed patches for AUFS 2.1 to
successfully rebuild kernel. I've created a complete aufs2.1-32 patch
that has been created specifically to rebuild the Proxmox 2.6.32
kernel packages. Note that this custom patch includes all files
required to either compile aufs2.1 into the kernel or as a module. All
of aufs2-base.patch, aufs2-kbuild.patch, aufs2-standalone.patch,
proc_map.patch and loopback.patch have been included into this patch.
I will continue to post new patches as the kernel gets updated
My current pve-kernel-2.6.32+aufs2.1-32.patch can be found on the
owncloud link below, but will be updated to reflect the suggestions
offered for the question at the beginning of this message.
https://owncloud.cs.uwindsor.ca/public.php?service=files&t=83cdb7f8067adaa046c3b3579cc47fdd
cd /your/src
apt-get install kernel-package lintian
git clone https://github.com/proxmox/pve-kernel-2.6.32
pve-kernel-2.6.32.git
cd pve-kernel-2.6.32.git
patch -p1 < /your/downloads/pve-kernel-2.6.32+aufs2.1-32.patch
The root changelog.Debian is patched with an arbitrarily large version
epoch so this kernel doesn't get overwritten by an update. You should
customize this changelog.Debian for your site before rebuilding the kernel.
Finally,
make
Install the kernel and headers packages created.