Blog
Skip to end of metadata
Go to start of metadata

0. Summary.

If you installing NOC on hardened operation systems, you may found some additional troubles, when try to install NOC on special kind of OS: Gentoo Hardened, Debian Hardened (Adamantix), and may be: OpenBSD, Trusted Solaris, and other OS of this class.

This problems depends of "bugs" in NOC dependencies.. That bugs should be fixed in future, but rite now you must use some tricks..

All discussion is about Hardened Gentoo GNU/Linux, but it describe main points, witch can be easy interpreted on other OS....

1. Illegal instruction.

if you see in log lot of such maseges:

Jul 11 17:42:51 noc kernel: [ 5971.747502] python2.7[14212] trap invalid opcode ip:7b14f8f4618c sp:7b885e42a820 error:0 in time.so[7b14f8f44000+4000]
Jul 11 17:42:51 noc kernel: [ 5971.747533] grsec: From 192.168.4.19: Illegal instruction occurred at 00007b14f8f4618c in /usr/bin/python2.7[python2.7:14212] uid/euid:999/999 gid/egid:0/0, parent /usr/bin/python2.7[noc-launcher.py:17878] uid/euid:0/0 gid/egid:0/0

it's not NOC bag, nether security volation. It is problem of bad program architecture and processor instraction. In new NOC installation when you change cflag to match you CPU, and install gentoo with NOC, some packages stay in initial archtecture as in stagetarbol. So look in log file, found wich packages has such problem:

$ equery b time.so
 * Searching for time.so ...
dev-lang/python-2.7.3-r2 (/usr/lib64/python2.7/lib-dynload/time.so)
and remege all such packages:
# emerge -1 =dev-lang/python-2.7.3-r2

alternatively you can rebuild world:

# emerge -et world

2 . Mongo need access to sysfs!

No programs normally should have access to sysfs, and this is big security hole!

If you see in mongo log some think like this:

***** SERVER RESTARTED *****
Sun Feb 5 12:08:30 [initandlisten] MongoDB starting : pid=17636 port=27017 dbpath=/home/db/mongodb 32-bit host=noc
Sun Feb 5 12:08:30 [initandlisten]
Sun Feb 5 12:08:30 [initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
Sun Feb 5 12:08:30 [initandlisten] ** see http://blog.mongodb.org/post/137788967/32-bit-limitations
Sun Feb 5 12:08:30 [initandlisten] ** with --journal, the limit is lower
Sun Feb 5 12:08:30 [initandlisten] exception in initAndListen std::exception: boost::filesystem::exists: Permission denied: "/sys/devices/system/node/node1", terminating
Sun Feb 5 12:08:30 dbexit:
Sun Feb 5 12:08:30 [initandlisten] shutdown: going to close listening sockets...
Sun Feb 5 12:08:30 [initandlisten] shutdown: going to flush diaglog...
Sun Feb 5 12:08:30 [initandlisten] shutdown: going to close sockets...
Sun Feb 5 12:08:30 [initandlisten] shutdown: waiting for fs preallocator...
Sun Feb 5 12:08:30 [initandlisten] shutdown: lock for final commit...
Sun Feb 5 12:08:30 [initandlisten] shutdown: final commit...
Sun Feb 5 12:08:30 [initandlisten] shutdown: closing all files...
Sun Feb 5 12:08:30 [initandlisten] closeAllFiles() finished
Sun Feb 5 12:08:30 dbexit: really exiting now

It is this problem! Because kernel option:

CONFIG_GRKERNSEC_SYSFS_RESTRICT=y

I found that version mongodb-2.2.4 compiled without mms-agent static-libs and v8 work god with CONFIG_GRKERNSEC_SYSFS_RESTRICT=y so recompile properly mongodb-2.2.4 is rite way.

Alternatively, this is  bad way, to garant mongo access to sysfs in Linux kernel you must change configuretion to:

# CONFIG_GRKERNSEC_SYSFS_RESTRICT is not set

like this:

Security options --->
Grecurity --->
Filesystem Protections --->
[ ] Sysfs/debugfs restriction

and recompile kernel.

In other OS same will be when

chmod 700 /sys

or protect /sys in some other way...


3. Security OS decide that all executables should be read only and all writeable should never be executable!

If mmap try receive memory for executable end not map it read-only kernel will kill such process!
So if Linux kernel options is set:

CONFIG_PAX_NOEXEC=y
CONFIG_PAX_PAGEEXEC=y
CONFIG_PAX_SEGMEXEC=y
# CONFIG_PAX_EMUTRAMP is not set
CONFIG_PAX_MPROTECT=y

and in kernel log you see:

Feb 3 16:43:21 noc kernel: [ 2592.126338] grsec: denied RWX mmap of <anonymous mapping> by /usr/bin/mongod[mongod:8866] uid/euid:104/104 gid/egid:101/101, parent /sbin/init[init:1] uid/euid:0/0 gid/egid:0/0
Feb 3 16:43:21 noc kernel: [ 2592.127740] grsec: denied RWX mmap of <anonymous mapping> by /usr/bin/mongod[mongod:8866] uid/euid:104/104 gid/egid:101/101, parent /sbin/init[init:1] uid/euid:0/0 gid/egid:0/0


The official PAX team sugests temporiary use:

CONFIG_PAX_MPROTECT_COMPAT=y
The current implementation of PAX_MPROTECT denies RWX allocations/mprotects by sending the proper error code to the application. For some broken userland, this can cause problems with Python or other applications. The current implementation however allows for applications like clamav to detect if JIT compilation/execution is allowed and to fall back gracefully to an interpreter-based mode if it does not. While we encourage everyone to use the current implementation as-is and push upstream to fix broken userland (note that the RWX logging option can assist with this), in some environments this may not be possible. Having to disable MPROTECT completely on certain binaries reduces the security benefit of PaX, so this option is provided for those environments to revert to the old behavior.

So using 'paxctl -m path_to_file' is bad way!

Or alternatively (this is bad way) you can use temporary work around this problem, switch of memory protect for such binary file:

paxctl - m /usr/bin/mongod

Same if you wont connect to mongo and see trace like this:

$ mongo noc
MongoDB shell version: 2.0.2
#
# Fatal error in v8::Context::New()
# V8 is no longer usable
#
Sun Feb 12 12:46:50 mongo got signal 6 (Aborted), stack trace:
Sun Feb 12 12:46:50 0x1fea0ffc 0x1fe96560 0x42df6400 0x42df6422 0x4337359d 0x43374ac3 0x43273076 
mongo(_ZN5mongo15printStackTraceERSo+0x3c) [0x1fea0ffc]
mongo(_Z12quitAbruptlyi+0x35a) [0x1fe96560]
[0x42df6400]
[0x42df6422]
/lib/libc.so.6(gsignal+0x49) [0x4337359d]
/lib/libc.so.6(abort+0x17b) [0x43374ac3]
/usr/lib/libv8-3.6.6.11.so(+0x2a5076) [0x43273076]

and in kernel log:

Feb 12 12:46:50 noc kernel: [595850.936074] grsec: denied RWX mmap of <anonymous mapping> by /usr/bin/mongo[mongo:3063] uid/euid:999/999 gid/egid:999/999, parent /bin/bash[bash:7609] uid/euid:999/999 gid/egid:999/999
Feb 12 12:46:50 noc kernel: [595850.937392] grsec: denied RWX mmap of <anonymous mapping> by /usr/bin/mongo[mongo:3063] uid/euid:999/999 gid/egid:999/999, parent /bin/bash[bash:7609] uid/euid:999/999 gid/egid:999/999

You may try recompile mongodb with USE=-v8

Or alternatively you can run:

paxctl -m /usr/bin/mongo


Same problem is in some NOC dependencies... So while all of them are not fixed you should disable memory protection for full python2:

paxctl - m /usr/bin/python2


4. Trusted Path Execution technique in OS block all files executions when they are not root user owned and not located in directory owned by root!

If Linux kernel option:

CONFIG_GRKERNSEC_TPE=y

and in kernel log you see:

Feb 5 13:09:16 noc kernel: [ 94.154309] grsec: more alerts, logging disabled for 10 seconds
Feb 5 13:09:28 noc kernel: [ 106.326285] grsec: denied untrusted exec of /opt/noc/scripts/stdin-wrapper by /usr/bin/python2.7[python2.7:6635] uid/euid:999/999 gid/egid:0/0, parent /usr/bin/python2.7[python2.7:6626] uid/euid:999/999 gid/egid:0/0
Feb 5 13:09:32 noc kernel: [ 110.384911] grsec: denied untrusted exec of /opt/noc/scripts/stdin-wrapper by /usr/bin/python2.7[python2.7:6638] uid/euid:999/999 gid/egid:0/0, parent /usr/bin/python2.7[python2.7:6626] uid/euid:999/999 gid/egid:0/0

This is because of unsecured premitions. To fix it run as root user:

chown -R root:root /opt/noc/
cd /opt/noc/
./scripts/post-install
chown -R noc:noc /opt/noc/etc/*
chown -R noc:noc /opt/noc/local


PS:

If you wont help security improvements you can read: http://www.gentoo.org/proj/en/hardened/
Valgrind also will help to find memory licks.