Skip to content

fix(build): Include build.rs in published crates #1199

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

colinmarc
Copy link

This fixes a very niche build problem: depending on datafusion-python as an rlib, and then attempting to build on macOS.

Because of the presence of an 'include' configuration in Cargo.toml, everything else is excluded - as well as build.rs, which means the necessary pyo3 linker flags aren't set when building.

I verified this fixes my build issue using a git dependency on datafusion-python:

datafusion-python = { git = "https://github.com/apache/datafusion-python", tag = "48.0.0" }

Whereas just using the crates.io version doesn't build:

    error: linking with `cc` failed: exit status: 1
        |
        = note:  "cc" "-Wl,-exported_symbols_list" "-Wl,/var/folders/b5/5n9p78x14yn8z4ky6fxtdp280000gn/T/rustc7RNcRS/list"
      "/var/folders/b5/5n9p78x14yn8z4ky6fxtdp280000gn/T/rustc7RNcRS/symbols.o" "<17 object files omitted>"
      "/Users/colinmarc/dev/bpln/all-events/target/release/deps/{libpyo3_log-bd50c2e7d1a9be99.rlib,libarc_swap-ad5cf4ebc23fa86d.rlib,libpyo3_async_runtimes-eba16526ca814ba0.rlib,libdatafusion_ffi-93bcb82d1826af3c.rlib,libsemver-69712b2f6934f52b.rlib,libasync_ffi-109cfda0a79f6f46.rlib,libdatafusion_proto-98671f6b4c156261.rlib,libdatafusion_proto_common-c828cce4df9cba68.rlib,libpbjson-e8c3fb6b43fe6a5d.rlib,libbase64-aabdebdc40eac9d2.rlib,libprost-492503f16444bcb4.rlib,libabi_stable-7f2a8da44cc2ad72.rlib,libconst_panic-04abeecc901c5835.rlib,librepr_offset-aec98a1518dda497.rlib,libtstr-e05c375dcbbdf609.rlib,libgenerational_arena-b27652eb370a1900.rlib,liblibloading-c9cfdbf7935ce071.rlib,libcrossbeam_channel-b3c1336cf04f651d.rlib,libabi_stable_shared-4a32241e1181b964.rlib,libcore_extensions-6be478146b530258.rlib,libdatafusion-68a5090fb1c58149.rlib,libdatafusion_functions_table-a785450daa220237.rlib,libdatafusion_functions_window-5c8ad3b057064cc3.rlib,libdatafusion_functions_nested-67c018d9521f14b2.rlib,libdatafusion_optimizer-52cbf43a83dc754a.rlib,libdatafusion_catalog_listing-7240cba6b8bb5026.rlib,libdatafusion_datasource_parquet-1d0a14c6642ff3d7.rlib,libdatafusion_physical_optimizer-07e0639703d8adc2.rlib,libdatafusion_datasource_avro-4b527ad125549f8f.rlib,libdatafusion_datasource_json-819f00f9c2f06d50.rlib,libdatafusion_datasource_csv-65d7c49418d40a12.rlib,libdatafusion_sql-c3b0e476dda0b9bd.rlib,libdatafusion_catalog-2958831fff8d6b8e.rlib,libdatafusion_datasource-a3b673b920baa280.rlib,libglob-5ed07e1ef250c612.rlib,libdatafusion_session-2b1ac2b9b5f38399.rlib,libasync_compression-8b5063c331ce847f.rlib,libbzip2-4f06fb5ed9de7d54.rlib,libdatafusion_physical_plan-2c5fe747d192870e.rlib,libdatafusion_common_runtime-5f702cc8d21973d0.rlib,libdatafusion_functions_aggregate-6b7b6f62fc4cf9c4.rlib,libdatafusion_physical_expr-1224453ebb9347b0.rlib,libpetgraph-d52f52b88facb45f.rlib,libfixedbitset-178cc05d890be25d.rlib,libdatafusion_functions-41d93652653ef031.rlib,libhex-b6c5adf57041980f.rlib,libsha2-c9b63e0c039043f3.rlib,libblake3-b47f6144315c4d77.rlib,libconstant_time_eq-42fd34e10c578200.rlib,libarrayvec-f9543f3f5ae864ab.rlib,libarrayref-d31fc091e2d6ee0e.rlib,libblake2-f4bf0046e980e172.rlib,libdatafusion_execution-c55886d381b3d34d.rlib,libtempfile-a1b2a5a4c563dc3b.rlib,libfastrand-36f87d3ad77c6fd5.rlib,librustix-aa2aaf9c3cec78aa.rlib,liberrno-79ccaecae0769ee1.rlib,libdashmap-6146eda158b025d0.rlib,libcrossbeam_utils-93015497c3e214cc.rlib,libdatafusion_expr-387d959328307058.rlib,libdatafusion_functions_aggregate_common-d90b999257ee962c.rlib,libdatafusion_functions_window_common-57329e93b7b40877.rlib,libdatafusion_doc-4d6559444419a12e.rlib,libdatafusion_physical_expr_common-d485d08f09369c57.rlib,libdatafusion_expr_common-5ef60fe07d8ee1b6.rlib,libdatafusion_common-934a95f38743bfaf.rlib,libhashbrown-80c15c43a5999630.rlib,libparquet-6c7cbfd00f9488c6.rlib,libbrotli-7ab64b7a0bc1c6e8.rlib,libbrotli_decompressor-ee61c12d1794f7c3.rlib,liballoc_stdlib-0497de1978d9f033.rlib,liballoc_no_stdlib-19d7b7eef48a3ff3.rlib,libflate2-9cafbdaee45ba358.rlib,libminiz_oxide-5a7d86077f7b9bd8.rlib,libadler2-5fc4e824e4457afb.rlib,liblibz_rs_sys-ed17e8a28580c3a2.rlib,libzlib_rs-d96186838fec8b4a.rlib,libobject_store-6edbb4329d27bd6c.rlib,libhumantime-5e0e4d17e02e6915.rlib,libthiserror-effe5e3ac7db3b8f.rlib,libwalkdir-9576fe72c80a1302.rlib,libsame_file-1fb48a6cc3f84c12.rlib,libparking_lot-1c1bc5ffa45d3ab6.rlib,libparking_lot_core-29fe49a404d7f533.rlib,liblock_api-a8dec283b0b7515c.rlib,libscopeguard-c1c5590a3fd72fb5.rlib,librustls_pemfile-432c5a98bc793fc7.rlib,librand-9516c305f7708ff0.rlib,librand_chacha-de4fd04099850197.rlib,librand_core-c5592f420480d246.rlib,libquick_xml-489731081c9f4e25.rlib,libmd5-ecdd0a59f23205fe.rlib,libitertools-13b9d0655b2429f3.rlib,libeither-20f8714b75d621d9.rlib,libreqwest-5e3084d5cec20c6e.rlib,libserde_urlencoded-5042827945b6a31b.rlib,libhyper_rustls-224c761b353653e2.rlib,libwebpki_roots-7026cf1d449d14b4.rlib,librustls_native_certs-d906b1ed148a3439.rlib,libsecurity_framework-2087cf7c1037269c.rlib,libcore_foundation-6b0653818f6f8a46.rlib,libsecurity_framework_sys-3d967eaa6d5db32a.rlib,libtokio_rustls-283618c8ca1d0dba.rlib,librustls-f7941ad045182529.rlib,libwebpki-94a5de63cdded17c.rlib,libring-e7703330386703aa.rlib,libuntrusted-c90860e50999b2a9.rlib,librustls_pki_types-0041ed224af372d3.rlib,libzeroize-9022ecac0a5e4acb.rlib,libtower_http-30f42cd6330d5ccd.rlib,libiri_string-0347e74622b08586.rlib,libtower-9503d8b3450d06d1.rlib,libtower_layer-f2e5f5dfcab61a2e.rlib,libhyper_util-1528424faa1a1748.rlib,libipnet-59b76d180a7ed2c3.rlib,libtower_service-2b92735bb08efeaa.rlib,libhyper-6e0744af25ac9bb9.rlib,libwant-79f9ecf12d0e2f65.rlib,libtry_lock-d5b41966a8a8def6.rlib,libhttparse-ef4be4395f90b1ab.rlib,libh2-dfc8aeaf325014af.rlib,libtracing-edd79e603cfb2c6e.rlib,libtracing_core-adf72d8e51cfce6a.rlib,libatomic_waker-834bc70eb5a5ee4c.rlib,libtokio_util-072198e6abe236af.rlib,libhttp_body_util-b3bc1a1aaae7b453.rlib,libhttp_body-75975b120e974f59.rlib,liburl-e285a33fec72a732.rlib,libidna-14cae32c7750eb8b.rlib,libutf8_iter-86aeeea8b517fd05.rlib,libidna_adapter-01f5c27feba1aa00.rlib,libicu_properties-de77c15a0df5c0e0.rlib,libicu_properties_data-5ca75e326589e0ed.rlib,libicu_normalizer-26c421f9ad902b44.rlib,libsmallvec-249d3f4ca8754cd8.rlib,libicu_normalizer_data-832c02018fb20fdc.rlib,libicu_collections-d9edb28e1775c80c.rlib,libpotential_utf-caba7d3c0f82f7bd.rlib,libicu_provider-677e08df1f065d0d.rlib,libicu_locale_core-c93769ac58b78da0.rlib,libtinystr-b4c0d3e234080f71.rlib,liblitemap-7434bcab13789da1.rlib,libwriteable-3595c6a4727da580.rlib,libzerovec-4d93715f0ed3cf04.rlib,libzerotrie-20ffe62c782ed7db.rlib,libyoke-2397f33fa66e22e1.rlib,libstable_deref_trait-1ad5d0b738b52bb9.rlib,libzerofrom-68088c4d8a6d3565.rlib,libform_urlencoded-8511452e00f9e778.rlib,libpercent_encoding-30f7b203e5794502.rlib,libhttp-45aadb77363e617b.rlib,libfnv-e07de106c7b4c98c.rlib,libsync_wrapper-068989de3d3e9fb3.rlib,libtokio-fa259681e99ba4f4.rlib,libsocket2-aae2154fbabf005f.rlib,libmio-261f0c46b28be8f2.rlib,libfutures-c5d96c7fcbbc14e1.rlib,libfutures_executor-67bf974f0b56d822.rlib,libfutures_util-1c3fe680cafecb70.rlib,libfutures_io-f4e11ec4606227af.rlib,libslab-4f719ac61e3db7c6.rlib,libfutures_channel-0cd37d2768807004.rlib,libpin_project_lite-d6b4c3abaa6ba590.rlib,libfutures_sink-0c37eddd73a396c5.rlib,libfutures_task-227a24be170cf210.rlib,libpin_utils-7565c1c1b0d23345.rlib,libfutures_core-73cf6b1d709bf5f9.rlib,libthrift-3009cb079a023204.rlib,libordered_float-40d7d306ca8c64ca.rlib,libthreadpool-07469e391275dec2.rlib,libnum_cpus-eb9b30df06252e68.rlib,libinteger_encoding-aaa7155d90d30c4f.rlib,libbyteorder-99912f0ff4e26454.rlib,libapache_avro-3204a2ccbb43cc6c.rlib,librand-c85d250afe322256.rlib,librand_chacha-f0f4cae7e338eae2.rlib,libppv_lite86-97de4abe00feffa7.rlib,librand_core-81fd3c91bb54801c.rlib,libgetrandom-d93181f1ee66e022.rlib,libzstd-fafe94de6cb95623.rlib,libzstd_safe-82265c9d926be9f7.rlib,libzstd_sys-6576136844b57447.rlib,libsnap-8b39be63489ddb57.rlib,libstrum-4fa9c7db083a41ed.rlib,libserde_bytes-a1c48fa0a9358c6f.rlib,libtyped_builder-8773f0b700ae2c31.rlib,libthiserror-b114ffb56e089b69.rlib,libregex_lite-1cd5459a9075a5b9.rlib,libdigest-7d57b25a08b94040.rlib,libsubtle-fba2f0087773609b.rlib,libconst_oid-6fc9dbda5666a297.rlib,libblock_buffer-21f68a0774c20711.rlib,libcrypto_common-9d5b7d24ca0d1cc9.rlib,libgeneric_array-c52cfd15b4c81270.rlib,libtypenum-dbc94d9ac6ecc909.rlib,libuuid-f7ee5ffe9fc05ec2.rlib,libxz2-fc787732f1ebc2b1.rlib,liblzma_sys-bb85373e4479f82a.rlib,libbzip2-d914a72e469559f1.rlib,libbzip2_sys-d2d139929083af2d.rlib,liblibflate-8b21986f86eb0661.rlib,libdary_heap-f4a7bbd38e9aefc5.rlib,libadler32-7c8e6a0369083f50.rlib,liblibflate_lz77-13c7bc3f6a43dba7.rlib,librle_decode_fast-ccfd9a82bdfba950.rlib,libcore2-d3ea4253b3ae26d0.rlib,libbigdecimal-30e928dee9fc5700.rlib,liblibm-82228d2d19a02ab8.rlib,libcrc32fast-f8aca6d2b09ed99b.rlib,libarrow-8a1f7ce8c9dd14f7.rlib,libarrow_row-84eb08d4cd961ffd.rlib,libarrow_pyarrow-5622f52f998232d6.rlib,libarrow_json-4b0a30102174a6de.rlib,libsimdutf8-46b6d81e66194e68.rlib,libindexmap-21d88474d1f1486d.rlib,libarrow_ipc-85d999ef949f9f5a.rlib,liblz4_flex-b12f51b3c2452d9a.rlib,libtwox_hash-1d2f8dbc720fa8f7.rlib,libflatbuffers-16321d00244e270f.rlib,libarrow_csv-05870c3b279a0398.rlib,libcsv-86a9275ba06878ec.rlib,libcsv_core-7faf0ea49beed3df.rlib,libarrow_string-c8f4d5c67fe11223.rlib,libregex-a26b2741ad471a02.rlib,libregex_automata-63be80d6e35fd497.rlib,libaho_corasick-ebe662c7d81718e6.rlib,libregex_syntax-2fd5791d4664d0bb.rlib,libarrow_cast-3ef77d16a9989f53.rlib,libatoi-898e3c475248cd16.rlib,libbase64-93cc24d04722c6b2.rlib,libcomfy_table-b31f844dae7a2715.rlib,libunicode_segmentation-08caeb8119b6a428.rlib,libunicode_width-819d17f485789e82.rlib,liblexical_core-6b29bf7482d4ac1a.rlib,liblexical_write_float-95a6204f3a06f52c.rlib,liblexical_write_integer-d38178122b23ea98.rlib,liblexical_parse_float-b2b676c7cae99aec.rlib,liblexical_parse_integer-bf0edbbc6c845c6c.rlib,liblexical_util-dff9ce4f975beea9.rlib,libstatic_assertions-b5177fa3f151f88a.rlib,libarrow_arith-8f71231d05b60d21.rlib,libarrow_ord-b8bbf587e071d183.rlib,libarrow_select-149bab3643fafe47.rlib,libarrow_array-cac0bf1b95860c27.rlib,libchrono_tz-7976adf5e08a5faf.rlib,libphf-e4bcc5492e1638c4.rlib,libphf_shared-29089568c7a98d73.rlib,libsiphasher-5e073432ba80b0ec.rlib,libahash-76010dfde000bb47.rlib,libgetrandom-a8a2fba13b7c3926.rlib,libzerocopy-93ff1ffbd9c920f9.rlib,libhashbrown-c09e8c50d4e4de7e.rlib,libfoldhash-8add571487c208a9.rlib,libequivalent-eaa0685fbf36c459.rlib,liballocator_api2-bf6b28bb29149b15.rlib,libchrono-16a492c8d6535c05.rlib,libiana_time_zone-9838eca935254022.rlib,libcore_foundation_sys-c7bdeb0be8f778de.rlib,libarrow_data-ad1393f0813db00b.rlib,libarrow_schema-b2763afed1d75869.rlib,libserde_json-d8d7def69aaeab54.rlib,libmemchr-010e50dec96889d4.rlib,libitoa-9897d6f33baa8efe.rlib,libryu-89145247fa865d8b.rlib,libbitflags-0b752ee354c1714f.rlib,libarrow_buffer-df745097521deb5d.rlib,libbytes-e9a5beea09e177f1.rlib,libhalf-1d5a96a0d129616c.rlib,libnum-1191270fef722447.rlib,libnum_iter-864a6772fe1c3728.rlib,libnum_rational-42bf3d6e0977e09b.rlib,libnum_complex-04cb518d1a34e126.rlib,libnum_bigint-f614862af97b5053.rlib,libserde-379e177ca80553ea.rlib,libnum_integer-9ea6283701072edd.rlib,libnum_traits-73dfdd8565e95ecb.rlib,libsqlparser-dae7132ab45c41eb.rlib,librecursive-ebdee60bb2e2bbbb.rlib,libstacker-b6c1c942456734b9.rlib,libpsm-d45da3fd3c987ab9.rlib,liblog-f848965e9d671f27.rlib,libpyo3-90b98f762f1a9d3b.rlib,libcfg_if-34bf03306b1dc143.rlib,libmemoffset-b977bfb2f9030c4a.rlib,libonce_cell-068f7909549c15e0.rlib,libpyo3_ffi-75cd3589555f4be7.rlib,libunindent-47c92a8720c117c0.rlib,libmimalloc-72a66481084f7fdc.rlib,liblibmimalloc_sys-ad4f99036bc921e8.rlib,liblibc-0caa3c72f476e2c6.rlib}.rlib"
      "<sysroot>/lib/rustlib/aarch64-apple-darwin/lib/{libstd-*,libpanic_unwind-*,libobject-*,libmemchr-*,libaddr2line-*,libgimli-*,librustc_demangle-*,libstd_detect-*,libhashbrown-*,librustc_std_workspace_alloc-*,libminiz_oxide-*,libadler2-*,libunwind-*,libcfg_if-*,liblibc-*,liballoc-*,librustc_std_workspace_core-*,libcore-*,libcompiler_builtins-*}.rlib"
      "-framework" "Security" "-framework" "CoreFoundation" "-liconv" "-lSystem" "-lc" "-lm" "-arch" "arm64" "-mmacosx-version-min=11.0.0" "-L"
      "/Users/colinmarc/dev/bpln/all-events/target/release/build/bzip2-sys-ce2535ad003224bd/out/lib" "-L" "/Users/colinmarc/dev/bpln/all-events/target/release/build/lzma-sys-e3b54090af14827e/out"
      "-L" "/Users/colinmarc/dev/bpln/all-events/target/release/build/zstd-sys-89467c99fe98418d/out" "-L" "/Users/colinmarc/dev/bpln/all-events/target/release/build/ring-6c070c10dae4d85c/out"
      "-L" "/Users/colinmarc/dev/bpln/all-events/target/release/build/psm-478f69e03f9c498e/out" "-L" "/Users/colinmarc/dev/bpln/all-events/target/release/build/blake3-b9ff590313ad109f/out" "-L"
      "/Users/colinmarc/dev/bpln/all-events/target/release/build/libmimalloc-sys-b64c9c1ed3774578/out" "-o" "/Users/colinmarc/dev/bpln/all-events/target/release/deps/libdatafusion_python-62a9be6a9f738ad3.dylib"
      "-Wl,-dead_strip" "-dynamiclib" "-nodefaultlibs"
        = note: some arguments are omitted. use `--verbose` to show all linker arguments
        = note: ld: warning: object file (/Users/colinmarc/dev/bpln/all-events/target/release/deps/libblake3-b47f6144315c4d77.rlib[5](a1edd97dd51cd48d-blake3_neon.o)) was built for newer 'macOS' version (15.5)
      than being linked (11.0)
                Undefined symbols for architecture arm64:
                  "_PyBaseObject_Type", referenced from:
                      <snip>
                      ...
                ld: symbol(s) not found for architecture arm64
                clang: error: linker command failed with exit code 1 (use -v to see invocation)


      error: could not compile `datafusion-python` (lib) due to 1 previous error
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `env -u CARGO MACOSX_DEPLOYMENT_TARGET="11.0" PYO3_ENVIRONMENT_SIGNATURE="cpython-3.12-64bit"
      PYO3_PYTHON="/Users/colinmarc/.cache/uv/builds-v0/.tmppBVRox/bin/python" PYTHON_SYS_EXECUTABLE="/Users/colinmarc/.cache/uv/builds-v0/.tmppBVRox/bin/python" "cargo" "rustc" "--features"
      "pyo3/extension-module" "--features" "python" "--message-format" "json-render-diagnostics" "--manifest-path" "/Users/colinmarc/dev/bpln/all-events/crates/bpln-df-extensions/Cargo.toml" "--release" "--lib"
      "--crate-type" "cdylib" "--" "-C" "link-arg=-undefined" "-C" "link-arg=dynamic_lookup" "-C" "link-args=-Wl,-install_name,@rpath/bauplan_datafusion_extensions.abi3.so"`
      Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/Users/colinmarc/.cache/uv/builds-v0/.tmppBVRox/bin/python', '--compatibility', 'off'] returned non-zero exit status 1

Note that you can verify build.rs is missing this way:

curl -fsSL https://crates.io/api/v1/crates/datafusion-python/48.0.0/download -o - | tar tf - | grep build.rs

This fixes a very niche build problem: depending on datafusion-python as
an rlib, and then attempting to build on macOS.

Because of the presence of an 'include' configuration in Cargo.toml,
everything else is excluded - including build.rs, which means the
necessary pyo3 linker flags aren't set.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant