Build Apache Arrow/Parquet and its dependencies on rocky9
Add xsimd, Thrift, Snappy, LZ4 and Arrow/Parquet before DuckDB, in the same order as debian13, using lib64 as the library directory. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
0bd9d85880
commit
8f24cf0e17
2 files changed
+94
-1
No files matched your search
@@ -53,6 +53,6 @@
|
|||||||
|
|
||||||
The image includes position-independent static builds of these libraries. Protobuf and libxml2 enable zlib support, and LibDwarf enables compressed debug section support using zlib and Zstd. ZeroMQ enables CURVE security using static libsodium. CZMQ enables UUID support using static util-linux libuuid while disabling undeclared optional integrations. librdkafka includes SSL, zlib, Zstd, and bundled LZ4 support while disabling undeclared optional integrations. spdlog uses external fmt, glog uses gflags, and all Meta libraries share the same release tag. Apache Arrow is built with Parquet (including OpenSSL-based Parquet encryption) and Snappy, LZ4, zlib, and Zstd compression using only the pinned system dependencies above (`ARROW_DEPENDENCY_SOURCE=SYSTEM`), with the bundled jemalloc and mimalloc allocators disabled. Apache Thrift only builds its C++ library, and its headers sit next to fbthrift's under `include/thrift` without overlapping.
|
The image includes position-independent static builds of these libraries. Protobuf and libxml2 enable zlib support, and LibDwarf enables compressed debug section support using zlib and Zstd. ZeroMQ enables CURVE security using static libsodium. CZMQ enables UUID support using static util-linux libuuid while disabling undeclared optional integrations. librdkafka includes SSL, zlib, Zstd, and bundled LZ4 support while disabling undeclared optional integrations. spdlog uses external fmt, glog uses gflags, and all Meta libraries share the same release tag. Apache Arrow is built with Parquet (including OpenSSL-based Parquet encryption) and Snappy, LZ4, zlib, and Zstd compression using only the pinned system dependencies above (`ARROW_DEPENDENCY_SOURCE=SYSTEM`), with the bundled jemalloc and mimalloc allocators disabled. Apache Thrift only builds its C++ library, and its headers sit next to fbthrift's under `include/thrift` without overlapping.
|
||||||
|
|
||||||
**Pending on rocky9 and alpine3.24:** xsimd, Apache Thrift, Snappy, LZ4, and Apache Arrow / Parquet are currently built only in the `debian13` image.
|
**Pending on alpine3.24:** xsimd, Apache Thrift, Snappy, LZ4, and Apache Arrow / Parquet are currently built only in the `debian13` and `rocky9` images.
|
||||||
|
|
||||||
**Alpine-only exception:** the `alpine3.24` image does not build Folly, Fizz, Wangle, mvfst, fbthrift, or CacheLib. These aren't on Meta's supported-OS list, and Folly's portability layer reaches for glibc-only macros/symbols with no musl branch (e.g. `off64_t`/`lseek64`/`pread64` in `folly/portability/Unistd.h`, `__ELF_NATIVE_CLASS` in `folly/debugging/symbolizer/Elf.cpp`), so patching each one as it surfaces risks introducing subtle bugs in code this project doesn't own. `debian13` and `rocky9` are glibc-based and build the full list above, Meta libraries included.
|
**Alpine-only exception:** the `alpine3.24` image does not build Folly, Fizz, Wangle, mvfst, fbthrift, or CacheLib. These aren't on Meta's supported-OS list, and Folly's portability layer reaches for glibc-only macros/symbols with no musl branch (e.g. `off64_t`/`lseek64`/`pread64` in `folly/portability/Unistd.h`, `__ELF_NATIVE_CLASS` in `folly/debugging/symbolizer/Elf.cpp`), so patching each one as it surfaces risks introducing subtle bugs in code this project doesn't own. `debian13` and `rocky9` are glibc-based and build the full list above, Meta libraries included.
|
||||||
@@ -683,6 +683,99 @@ RUN --mount=type=bind,from=sources,source=libproxyprotocol-v1.3.3.tar.gz,target=
|
|||||||
&& cp src/src/proxy_protocol.h /usr/local/include/ \
|
&& cp src/src/proxy_protocol.h /usr/local/include/ \
|
||||||
&& rm -rf src
|
&& rm -rf src
|
||||||
|
|
||||||
|
RUN --mount=type=bind,from=sources,source=xsimd-14.2.0.tar.gz,target=/tmp/xsimd.tar.gz \
|
||||||
|
mkdir src \
|
||||||
|
&& tar --extract --gzip --file=/tmp/xsimd.tar.gz --directory=src --strip-components=1 \
|
||||||
|
&& cmake -G Ninja -S src -B build -DCMAKE_BUILD_TYPE=Release \
|
||||||
|
-DBUILD_TESTS=OFF \
|
||||||
|
-DCMAKE_INSTALL_LIBDIR=lib64 \
|
||||||
|
&& cmake --build build --parallel "$(nproc)" --target install \
|
||||||
|
&& rm -rf build src
|
||||||
|
|
||||||
|
RUN --mount=type=bind,from=sources,source=thrift-v0.22.0.tar.gz,target=/tmp/thrift.tar.gz \
|
||||||
|
mkdir src \
|
||||||
|
&& tar --extract --gzip --file=/tmp/thrift.tar.gz --directory=src --strip-components=1 \
|
||||||
|
&& cmake -G Ninja -S src -B build -DCMAKE_BUILD_TYPE=Release \
|
||||||
|
-DBUILD_COMPILER=OFF \
|
||||||
|
-DBUILD_SHARED_LIBS=OFF \
|
||||||
|
-DBUILD_TESTING=OFF \
|
||||||
|
-DBUILD_TUTORIALS=OFF \
|
||||||
|
-DCMAKE_INSTALL_DIR=/usr/local/lib64/cmake \
|
||||||
|
-DCMAKE_POSITION_INDEPENDENT_CODE=ON \
|
||||||
|
-DLIB_INSTALL_DIR=/usr/local/lib64 \
|
||||||
|
-DPKGCONFIG_INSTALL_DIR=/usr/local/lib64/pkgconfig \
|
||||||
|
-DWITH_AS3=OFF \
|
||||||
|
-DWITH_C_GLIB=OFF \
|
||||||
|
-DWITH_CPP=ON \
|
||||||
|
-DWITH_JAVA=OFF \
|
||||||
|
-DWITH_JAVASCRIPT=OFF \
|
||||||
|
-DWITH_LIBEVENT=OFF \
|
||||||
|
-DWITH_NODEJS=OFF \
|
||||||
|
-DWITH_OPENSSL=OFF \
|
||||||
|
-DWITH_PYTHON=OFF \
|
||||||
|
-DWITH_QT5=OFF \
|
||||||
|
-DWITH_ZLIB=OFF \
|
||||||
|
&& cmake --build build --parallel "$(nproc)" --target install \
|
||||||
|
&& sed -i 's|^libdir=.*|libdir=${prefix}/lib64|' \
|
||||||
|
/usr/local/lib64/pkgconfig/thrift.pc \
|
||||||
|
&& rm -rf build src
|
||||||
|
|
||||||
|
RUN --mount=type=bind,from=sources,source=snappy-1.2.2.tar.gz,target=/tmp/snappy.tar.gz \
|
||||||
|
mkdir src \
|
||||||
|
&& tar --extract --gzip --file=/tmp/snappy.tar.gz --directory=src --strip-components=1 \
|
||||||
|
&& cmake -G Ninja -S src -B build -DCMAKE_BUILD_TYPE=Release \
|
||||||
|
-DBUILD_SHARED_LIBS=OFF \
|
||||||
|
-DCMAKE_INSTALL_LIBDIR=lib64 \
|
||||||
|
-DCMAKE_POSITION_INDEPENDENT_CODE=ON \
|
||||||
|
-DSNAPPY_BUILD_BENCHMARKS=OFF \
|
||||||
|
-DSNAPPY_BUILD_TESTS=OFF \
|
||||||
|
&& cmake --build build --parallel "$(nproc)" --target install \
|
||||||
|
&& rm -rf build src
|
||||||
|
|
||||||
|
RUN --mount=type=bind,from=sources,source=lz4-v1.10.0.tar.gz,target=/tmp/lz4.tar.gz \
|
||||||
|
mkdir src \
|
||||||
|
&& tar --extract --gzip --file=/tmp/lz4.tar.gz --directory=src --strip-components=1 \
|
||||||
|
&& cmake -G Ninja -S src/build/cmake -B build -DCMAKE_BUILD_TYPE=Release \
|
||||||
|
-DBUILD_SHARED_LIBS=OFF \
|
||||||
|
-DBUILD_STATIC_LIBS=ON \
|
||||||
|
-DCMAKE_INSTALL_LIBDIR=lib64 \
|
||||||
|
-DCMAKE_POSITION_INDEPENDENT_CODE=ON \
|
||||||
|
-DLZ4_BUILD_CLI=OFF \
|
||||||
|
-DLZ4_POSITION_INDEPENDENT_LIB=ON \
|
||||||
|
&& cmake --build build --parallel "$(nproc)" --target install \
|
||||||
|
&& rm -rf build src
|
||||||
|
|
||||||
|
RUN --mount=type=bind,from=sources,source=arrow-apache-arrow-25.0.1.tar.gz,target=/tmp/arrow.tar.gz \
|
||||||
|
mkdir src \
|
||||||
|
&& tar --extract --gzip --file=/tmp/arrow.tar.gz --directory=src --strip-components=1 \
|
||||||
|
&& cmake -G Ninja -S src/cpp -B build -DCMAKE_BUILD_TYPE=Release \
|
||||||
|
-DARROW_BUILD_BENCHMARKS=OFF \
|
||||||
|
-DARROW_BUILD_EXAMPLES=OFF \
|
||||||
|
-DARROW_BUILD_SHARED=OFF \
|
||||||
|
-DARROW_BUILD_STATIC=ON \
|
||||||
|
-DARROW_BUILD_TESTS=OFF \
|
||||||
|
-DARROW_BUILD_UTILITIES=OFF \
|
||||||
|
-DARROW_DEPENDENCY_SOURCE=SYSTEM \
|
||||||
|
-DARROW_DEPENDENCY_USE_SHARED=OFF \
|
||||||
|
-DARROW_IPC=ON \
|
||||||
|
-DARROW_JEMALLOC=OFF \
|
||||||
|
-DARROW_MIMALLOC=OFF \
|
||||||
|
-DARROW_PARQUET=ON \
|
||||||
|
-DARROW_POSITION_INDEPENDENT_CODE=ON \
|
||||||
|
-DARROW_WITH_BROTLI=OFF \
|
||||||
|
-DARROW_WITH_BZ2=OFF \
|
||||||
|
-DARROW_WITH_LZ4=ON \
|
||||||
|
-DARROW_WITH_SNAPPY=ON \
|
||||||
|
-DARROW_WITH_ZLIB=ON \
|
||||||
|
-DARROW_WITH_ZSTD=ON \
|
||||||
|
-DCMAKE_INSTALL_LIBDIR=lib64 \
|
||||||
|
-DCMAKE_POSITION_INDEPENDENT_CODE=ON \
|
||||||
|
-DPARQUET_BUILD_EXAMPLES=OFF \
|
||||||
|
-DPARQUET_BUILD_EXECUTABLES=OFF \
|
||||||
|
-DPARQUET_REQUIRE_ENCRYPTION=ON \
|
||||||
|
&& cmake --build build --parallel "$(nproc)" --target install \
|
||||||
|
&& rm -rf build src
|
||||||
|
|
||||||
RUN --mount=type=bind,from=sources,source=duckdb-v1.5.6.tar.gz,target=/tmp/duckdb.tar.gz \
|
RUN --mount=type=bind,from=sources,source=duckdb-v1.5.6.tar.gz,target=/tmp/duckdb.tar.gz \
|
||||||
--mount=type=bind,from=sources,source=duckdb-ui-0628932f74b1e2e9eaf68a05b22d737455541dca.tar.gz,target=/tmp/duckdb-ui.tar.gz \
|
--mount=type=bind,from=sources,source=duckdb-ui-0628932f74b1e2e9eaf68a05b22d737455541dca.tar.gz,target=/tmp/duckdb-ui.tar.gz \
|
||||||
mkdir src \
|
mkdir src \
|
||||||
|
|||||||
Reference in new issue
Block a user