elasticsearch Too Many Open Files

UPDATE: I may lose some info about this issue, so I will write a new post about open file limits. It's here.

UPDATE: There are two levels in OS open files limit: http://www.chengweiyang.cn/2015/11/14/how-to-enlarge-linux-open-files-upper-cell/

As almost posts on Google, you can increase ulimit nofile limits for root or the user you use.

The limits of open files means how many files one process can open, notice one process, not whole system.

You can find how many files opened by process via lsof or list this directory: /proc/<PID>/fd, PID is your process ID.

I use elasticsearch in LXC, there are two scenarios: privileged LXC or unprivileged LXC.

If you use privileged LXC, just modify ulimits.conf or add config file to /etc/security/limits.d/, it's OK.

If you use unprivileged LXC, you must modify your host limits firstly.

When I modified ulimits.conf I found elasticsearch open 4000+ files, original limit is 1024.

If your elasticsearch is managed by supervisor, you should check supervisor process limit and the minfds config, more details here.

Recommended nofile for elasticsearch is 65536.

You can modify ulimits follow http://posidev.com/blog/2009/06/04/set-ulimit-parameters-on-ubuntu/, and this is a PAM tutorial https://www.digitalocean.com/community/tutorials/how-to-use-pam-to-configure-authentication-on-an-ubuntu-12-04-vps
sysctl fs doc: https://www.kernel.org/doc/Documentation/sysctl/fs.txt