UPDATE:If you run
supervisor as root, just set
minfds to right value (32K or 64K),
supervisor will increase hard limit value if current value is not enough and run as root, it’s the simplest way!
Several days ago, my elasticsearch came across Too Many Open files error, the straight forward way is modify ulimit conf, but today I came across this error again, so I walk a wrong way before.
Why I came across that error again?
Elasticsearch ran as root, but it’s managed by
supervisor which runs as
supervisor is a process management program, it must limit its child process resources, so it provides a config argument
minfds, the minimum file description running
supervisor, if you omitted this, default value is 1024, even your user has a high
Supervisor sets its soft and hard open file limit to 1024 and 4096, so the child process can not break the limits!
You can get elasticsearch node process info:
you will find the
max_file_descriptors on each
node, after added
max_file_descriptors was broken!
In Ubuntu(may be almost linux), there are two levels about open file limit, the kernel space and the user space, user space must meet kernel space limit.
/proc/sys/fs/ means the maximum fds per process can allocate, it means the hard limit in
file-max means the maximum number of fds that the Linux kernel will allocate, it included all process in kernel.
file-nr means the number of allocated file handles, the number of allocated but unused file handles, and the maximum number of file handles. Linux 2.6 always reports 0 as the number of free file handles – this is not an error, it just means that the number of allocated file handles exactly matches the number of used file handles.