If native libraries are not available the following message is displayed with every hadoop command: hadoop checknative
Some patches for fixing build a hadoop native library on os x 10.7/10.8. Jul 24, 2019 The native hadoop library does not work with Cygwin or the Mac OS X platform. It is supported on.nix platforms only. It is mainly used on the GNU/Linus platform. The WARN message that you are getting should be disregarded on Mac OS X as the. Dec 31, 2017 Issue Resolved! Unable to load native hadoop library for your platform. 官方文档说明native hadoop library不支持Mac OS X平台，不过可以通过homebrew安装环境后支持，如下： The native hadoop library is supported on.nix platforms only. The library does not to work with Cygwin or the Mac OS X platform. Apache Hadoop - add native libraries If native libraries are not available the following message is displayed with every hadoop command: hadoop checknative WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform. Using builtin-java classes where applicable. In my case, after I build hadoop on my 64 bit Linux mint OS, I replaced the native library in hadoop/lib. Still the problem persist. Then I figured out the hadoop pointing to hadoop/lib not to the hadoop/lib/native. So I just moved all content from native library to its parent.
Clone hadoop source code
Checkout the version
2.7.1source
会出现警告『WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform. Using builtin-java classes where applicable』，其实就是缺少本地库，不会影响 Hadoop 的使用，只是影响效率。解决方案可参考： Mac OSX 下 Hadoop 使用本地库提高效率 （未试验）.
Install required dependencies - OSX: use
brewor any other package manager
Build project and native dependencies with
maven
Copy newly created libraries to the hadoop installation
Add shell variables either to the
~/.bash_profile, or$HADOOP_INSTALL/etc/hadoop/hadoop-env.sh
Check if native libraries are available:
above rules do not cause bzip2 native libs to work in OSX. |
Apache Hadoop Download Mac
| brew install snappy |
| CPPFLAGS=-I/usr/local/include LDFLAGS=-L/usr/local/lib ant compile-native |
| chmod 644 build/native/Mac_OS_X-x86_64-64/lib/* |
| ls -l /usr/local/lib/libsnappy.dylib |
| ls -l build/native/Mac_OS_X-x86_64-64/lib/libhadoop.dylib |
| commit a5681a94b83e7314a46863f811226e4e51967595 |
| Author: David Phillips <david@acz.org> |
| Date: Wed Oct 10 18:34:24 2012 -0700 |
| Compile native libraries on Mac OS X |
| diff --git a/src/native/Makefile.am b/src/native/Makefile.am |
| index c70de7d..6268877 100644 |
| --- a/src/native/Makefile.am |
| +++ b/src/native/Makefile.am |
| @@ -33,9 +33,7 @@ export PLATFORM = $(shell echo $$OS_NAME | tr [A-Z] [a-z]) |
| AM_CPPFLAGS = @JNI_CPPFLAGS@ -I$(HADOOP_NATIVE_SRCDIR)/src |
| -Isrc/org/apache/hadoop/io/compress/zlib |
| - -Isrc/org/apache/hadoop/io/compress/snappy |
| - -Isrc/org/apache/hadoop/io/nativeio |
| - -Isrc/org/apache/hadoop/security |
| + -Isrc/org/apache/hadoop/io/compress/snappy |
| AM_LDFLAGS = @JNI_LDFLAGS@ -m$(JVM_DATA_MODEL) |
| AM_CFLAGS = -g -Wall -fPIC -O2 -m$(JVM_DATA_MODEL) |
| @@ -43,13 +41,7 @@ lib_LTLIBRARIES = libhadoop.la |
| libhadoop_la_SOURCES = src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c |
| src/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.c |
| src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c |
| - src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c |
| - src/org/apache/hadoop/security/getGroup.c |
| - src/org/apache/hadoop/security/JniBasedUnixGroupsMapping.c |
| - src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c |
| - src/org/apache/hadoop/io/nativeio/file_descriptor.c |
| - src/org/apache/hadoop/io/nativeio/errno_enum.c |
| - src/org/apache/hadoop/io/nativeio/NativeIO.c |
| + src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c |
| libhadoop_la_LDFLAGS = -version-info 1:0:0 $(AM_LDFLAGS) |
| libhadoop_la_LIBADD = -ldl -ljvm |
| diff --git a/src/native/acinclude.m4 b/src/native/acinclude.m4 |
| index 266ac87..48481b3 100644 |
| --- a/src/native/acinclude.m4 |
| +++ b/src/native/acinclude.m4 |
| @@ -13,8 +13,10 @@ AC_CACHE_CHECK([Checking for the 'actual' dynamic-library for '-l$1'], ac_cv_lib |
| ac_cv_libname_$1='`objdump -p conftest | grep NEEDED | grep $1 | sed 's/W*NEEDEDW*(.*)W*$/'1'/'`' |
| elif test ! -z '`which ldd | grep -v 'no ldd'`'; then |
| ac_cv_libname_$1='`ldd conftest | grep $1 | sed 's/^[[[^A-Za-z0-9]]]*([[[A-Za-z0-9.]]]*)[[[^A-Za-z0-9]]]*=>.*$/'1'/'`' |
| + elif test ! -z '`which otool | grep -v 'no otool'`'; then |
| + ac_cv_libname_$1='`otool -L conftest | grep $1 | sed -e 's/^[ ]*//' -e 's/ .*//' -e 's/.*/(.*)$/1/'`'; |
| else |
| - AC_MSG_ERROR(Can't find either 'objdump' or 'ldd' to compute the dynamic library for '-l$1') |
| + AC_MSG_ERROR(Can't find either 'objdump', 'ldd' or 'otool' to compute the dynamic library for '-l$1') |
| fi |
| else |
| ac_cv_libname_$1=libnotfound.so |
| diff --git a/src/native/configure.ac b/src/native/configure.ac |
| index 71ff66d..87b34a1 100644 |
| --- a/src/native/configure.ac |
| +++ b/src/native/configure.ac |
| @@ -56,7 +56,7 @@ JNI_LDFLAGS=' |
| if test $JAVA_HOME != ' |
| then |
| JNI_LDFLAGS='-L$JAVA_HOME/jre/lib/$OS_ARCH/server' |
| - JVMSOPATH=`find $JAVA_HOME/jre/ -name libjvm.so | head -n 1` |
| + JVMSOPATH=`find $JAVA_HOME/jre/ -name 'libjvm.*' | head -n 1` |
| JNI_LDFLAGS='$JNI_LDFLAGS -L`dirname $JVMSOPATH`' |
| fi |
| ldflags_bak=$LDFLAGS |