Skip to content

Tag: Docker

What is d_type and why Docker overlayfs need it

In my previous post I’ve mentioned a strange problem that occurs on Discourse running in Docker. Today I’m going to explain this further as this problem could potentially impact any Docker setup uses overlayfs storage driver. Practically, CentOS 7 with all default setup during installation is 100% affected. Docker on Ubuntu uses AUFS so is not affected.

What is d_type

d_type is the term used in Linux kernel that stands for “directory entry type”. Directory entry is a data structure that Linux kernel used to describe the information of a directory on the filesystem. d_type is a field in that data structure which represents the type of an “file” the directory entry points to. It could a directory, a regular file, some special file such as a pipe, a char device, a socket file etc.

d_type information was added to Linux kernel version 2.6.4. Since then Linux filesystems started to implement it over time. However still some filesystem don’t implement yet, some implement it in a optional way, i.e. it could be enabled or disabled depends on how the user creates the filesystem.

Why it is important to Docker

Overlay and Overlay2 are the two supported storage driver of Docker. Both of them depends on the overlayfs filesystem. Below is a picture from Docker document shows how Docker uses overlayfs for its image storage.

Overlay storage driver in Docker

In the overlayfs code(it’s part of the Linux kernel), this d_type information is accessed and used to make sure some file operations are correctly handled. There is code in overlayfs to specifically check for existence of the d_type feature, and print warning message if it does not exist on the underlying filesystem.

Docker, when running on overlay/overlay2 storage driver, requires the d_type feature to functioning correctly. A check was added to Docker 1.13. By running docker info command now you can tell whether your backing filesystems supports it or not. The plan is to issue an error message in Docker 1.16 if d_type is not enabled.

When d_type is no supported on the backing filesystem of overlayfs, containers running on Docker would run into some strange errors doing file operation. Chown error during Discourse bootstrap or rebuild is one common error. There are some other examples you can find in Docker issues on GitHub, I’ve take some for example as below.

Randomly cannot start Containers with “Clean up Error! Cannot destroy container” “mkdir …-init/merged/dev/shm: invalid argument” #22937

Centos 7 fails to remove files or directories which are created on build of the image using overlay or overlay2. #27358

docker run fails with “invalid argument” when using overlay driver on top of xfs #10294

Check whether your system is affected

TL;DR: Ext4? Good. XFS on RHEL/CentOS 7? High chance bad, use xfs_info to confirm

As mentioned above, d_type support is optional for some filesystem. This includes XFS, the default filesystem in Red Hat Enterprise Linux 7, which is the upstream base of CentOS 7. Unfortunately, the Red Hat /CentOS installer and mkfs.xfs command all by default create XFS filesystem without d_type feature turned on…… What a mess!

As a quick rule, if you are using RHEL 7 or CentOS 7, and your filesystem is created by default without specifying an parameter, you can almost be 100% sure that d_type is not turned on on your filesystem. To check for sure, follow below steps.

First you need to find out what filesystem you are currently using. Although XFS is the default during installation, some people or the hosting provider may choose to use Ext4. If that’s the case, then relax, d_type is supported.

If you are on XFS, then you need to run xfs_info command against the filesystem you need to check. Below is an example from my system.

$ xfs_info /
meta-data=/dev/sda1              isize=256    agcount=4, agsize=3276736 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=13106944, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=6399, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Pay attention to the last column of 6th line of the command output. You can see ftype=1. That’s a good news. It means my XFS was created with the correct parameter, ftype=1, thus d_type was turned on. If you see a ftype=0 there, that means d_type is off.

How to solve the problem

Another bad news is this problem can only be fixed by recreate the filesystem. It cannot be change on an existing filesystem! Basically the steps are:

  1. Backup your data
  2. Recreate the filesystem with correct parameter for XFS, or just create an Ext4 filesystem instead.
  3. Restore your data back.

Let’s focus on step #2. DON’T try any of below command on your server before you fully understand them and have backup secured!

If you chose Ext4 filesystem, then it’s easy, just run mkfs.ext4 /path/to/your/device and that’s it.

If you chose XFS filesystem, the correct command is:

mkfs.xfs -n ftype=1 /path/to/your/device

The -n ftype=1 parameter tells mkfs.xfs program to create a XFS with d_type feature turned on.

Take actions

It is a good idea to check your system asap to see if this d_type problem affects your RHEL/CentOS 7 installation. The sooner you fix the problem the better.

4 Comments

Fix strange chown error during Discourse bootstrap or rebuild

TL;DR

Discourse is an forum software runs in Docker. When using overlayfs/overlayfs2 storage driver, Docker requires the backing filesystem supports d_type. Or else some strange error will just pop up during some very basic file operations, such as chown command.

The symptom

When bootstrapping or rebuilding Discourse on CentOS, the process fails with chown command related errors.

# ./launcher bootstrap app
FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && chown -R discourse /var/www/discourse failed with return # Location of failure: /pups/lib/pups/exec_command.rb:108:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"web", "cmd"=>["gem update bundler", "chown -R discourse $home"]}
d5f9b0e0bb45020d8b9e1f25cc8b96091eba69a505a60b4178e641530e4ac159
** FAILED TO BOOTSTRAP **

How to confirm whether you are hit by this problem.

If you are running CentOS 7 and the filesystem was created all by default, you get it. CentoOS installer and the mkfs.xfs command both by default create XFS with ftype=0, which does not meet Docker requirement for filesystem d_type support.

Check the xfs_info command output, mind the ftype=0 thing.

# xfs_info /
......
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
......

Then run docker info command to see if Docker pointed it out. You will have to use a new enough Docker version, older ones don’t report d_type information. Below is the output from Docker 1.13.

......
Storage Driver: overlay
Backing Filesystem: extfs
Supports d_type: false
......

If you see above in your system, then you are in trouble now. Not only Discourse but other container apps may run into strange problem when they do file operations on the overlayfs. Fix it ASAP!

The solution

The key is to get your Docker a filesystem with d_type support. Unfortunately this option can only be set while creating the filesytem. Once filesystem creation is done, the only way to change it is to:

  1. Backup data
  2. Recreate filesystem
  3. Restore data.

Step #1 & 3 is out of the scope of this post. Let’s focus on step #2, how to create the filesystem in the correct way. Two options exist, use XFS or Ext4.

If you prefer XFS

When you run mkfs.xfs command to create XFS on your partition/volume, make sure you pass the -n ftype=1 parameter. The command line looks like below

mkfs.xfs -n ftype=1 /path/to/your/device

If you prefer Ext4 FS

Ext4 filesystem created with default option supports d_type, so there is no special parameter to use when you create Ext4 filesystem on your partition/volume. Easy!

Tips for Docker and Discourse

Since Docker puts its files under /var/lib/docker directory, you only need to make sure d_type is supported for this specific directory. So if you have free space on your disk, you don’t have to touch your whole root filesystem. Just allocate some space, create a new filesystem with correct parameter, then mount it under /var/lib/docker and it’s done.

As regards to Discourse, this procedure won’t even hurt your Discourse data. Discourse puts all data under /var/discourse/share directory. When you get a new /var/lib/docker directory, only the container definition is gone. You just need to recreate the container with launcher script, then the site will just back to normal.

With that said, backup data before doing any filesystem or disk related operation is still a good practice!

Reference

My post regarding this issue on Discourse official forum.

Issue: Centos 7 fails to remove files or directories which are created on build of the image using overlay or overlay2.

Issue: overlayfs: Can’t delete file moved from base layer to newly created dir even on ext4

1 Comment