Getting to compile Python for WASM/WASI, funny enough the first step is installing the Python-based tools that make "wasienv":
$ pip install wasienv Processing /home/user/.cache/pip/wheels/15/72/ed/35789dff12f6ca3d9cd98454519aff343d102da520b98326dd/wasienv-0.5.4-py3-none-any.whl Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from wasienv) (2.22.0) Installing collected packages: wasienv Successfully installed wasienv-0.5.4
Now following instructions from https://github.com/wapm-packages/python/blob/master/README.rst, and tinkering a bit with python cmake options...
$ git clone --filter=blob:none https://github.com/wapm-packages/python.git $ mkdir -p python/wasi/install && cd python/wasi
$ wasimake cmake -DCMAKE_INSTALL_PREFIX:PATH="$(realpath install)" -DUSE_SYSTEM_LIBRARIES=OFF .. -- The ASM compiler identification is unknown -- Found assembler: /home/agustin/bin/wasicc -- Warning: Did not find file Compiler/-ASM ... -- Configuring done -- Generating done -- Build files have been written to: /home/agustin/wasm/python/python-wasi
Ready to make it are we?
$ make -j7 [ 0%] Built target extension_testcapi [ 5%] Built target pgen [ 6%] Building C object CMakeBuild/libpython/CMakeFiles/_freeze_importlib.dir/__/__/Python-3.6.7/Objects/listobject.c.obj [ 6%] Building C object CMakeBuild/libpython/CMakeFiles/_freeze_importlib.dir/__/__/Python-3.6.7/Objects/longobject.c.obj [ 7%] Building C object CMakeBuild/libpython/CMakeFiles/_freeze_importlib.dir/__/__/Python-3.6.7/Objects/memoryobject.c.obj [ 7%] Building C object CMakeBuild/libpython/CMakeFiles/_freeze_importlib.dir/__/__/Python-3.6.7/Objects/moduleobject.c.obj [ 7%] Building C object CMakeBuild/libpython/CMakeFiles/_freeze_importlib.dir/__/__/Python-3.6.7/Objects/methodobject.c.obj [ 7%] Building C object CMakeBuild/libpython/CMakeFiles/_freeze_importlib.dir/__/__/Python-3.6.7/Objects/object.c.obj [ 7%] Building C object CMakeBuild/libpython/CMakeFiles/_freeze_importlib.dir/__/__/Python-3.6.7/Objects/obmalloc.c.obj In file included from:1: /home/agustin/.local/lib/python3.8/site-packages/wasienv/stubs/preamble.h:30:9: warning: 'ESHUTDOWN' macro redefined [-Wmacro-redefined] #define ESHUTDOWN 0 ... 6 warnings generated. [ 42%] Linking C executable _freeze_importlib [ 52%] Built target _freeze_importlib [ 52%] Generating ../../../Python-3.6.7/Python/importlib_external.h, ../../../Python-3.6.7/Python/importlib.h cannot open '/home/user/wasm/python/Python-3.6.7/Lib/importlib/_bootstrap_external.py' for reading make[2]: *** [CMakeBuild/libpython/CMakeFiles/libpython-static.dir/build.make:64: ../Python-3.6.7/Python/importlib_external.h] Error 1 make[1]: *** [CMakeFiles/Makefile2:1179: CMakeBuild/libpython/CMakeFiles/libpython-static.dir/all] Error 2 make: *** [Makefile:141: all] Error 2
But the file is there, what magic is forbidding freeze_importlib from opening it? Let's check with strace...
$ strace ./_freeze_importlib /home/user/wasm/python/Python-3.6.7/Lib/importlib/_bootstrap_external.py /home/user/wasm/python/Python-3.6.7/Python/importlib_external.h
execve("./_freeze_importlib", ["./_freeze_importlib", "/home/user/wasm/python/Python"..., "/home/user/wasm/python/Python"...], 0x7ffd0490d690 /* 58 vars */) = 0
...
access("/home/user/bin/wasirun", R_OK) = 0
rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fed289eaa10) = 69585
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {sa_handler=0x55d072a77480, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fed28a300c0}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fed28a300c0}, 8) = 0
wait4(-1, cannot open '/home/user/wasm/python/Python-3.6.7/Lib/importlib/_bootstrap_external.py' for reading (errno=44)
fopen: No such file or directory
Looks like the binary is wrapped by "wasirun", I guess to run it inside a virtualroot sandbox, and some required paths are not visible there.
Time to debug...
"DEBUG" can be enabled by tweaking wasirun/tools.py:
Now the same call to make dumps what wasienv is doing:
[ 52%] Generating ../../../Python-3.6.7/Python/importlib_external.h, ../../../Python-3.6.7/Python/importlib.h wasienv run process: wasmer run --dir=. --enable-all /home/user/wasm/python/python-wasi/CMakeBuild/libpython/_freeze_importlib.wasm -- /home/user/wasm/python/Python-3.6.7/Lib/importlib/_bootstrap_external.py /home/user/wasm/python/Python-3.6.7/Python/importlib_external.h cannot open '/home/agustin/wasm/python/Python-3.6.7/Lib/importlib/_bootstrap_external.py' for reading (errno=44) fopen: No such file or directory
The problem is that only current dir "." is exposed in the WASM context. Let's call wasmer run directly with dir=/
$ wasmer run --dir / --enable-all /home/user/wasm/python/python-wasi/CMakeBuild/libpython/_freeze_importlib.wasm -- /home/user/wasm/python/Python-3.6.7/Lib/importlib/_bootstrap_external.py /home/user/wasm/python/Python-3.6.7/Python/importlib_external.h
error: failed to run `/home/user/wasm/python/python-wasi/CMakeBuild/libpython/_freeze_importlib.wasm`
│ 1: RuntimeError: indirect call type mismatch
at _PyCFunction_FastCallDict (_freeze_importlib.wasm[1839]:0x15c040)
at _PyObject_FastCallDict (_freeze_importlib.wasm[658]:0x86973)
at callmethod (_freeze_importlib.wasm[737]:0x905ee)
at _PyObject_CallMethodId (_freeze_importlib.wasm[736]:0x90451)
at flush_std_files (_freeze_importlib.wasm[2975]:0x279fee)
at Py_FinalizeEx (_freeze_importlib.wasm[2980]:0x27afa9)
at Py_Finalize (_freeze_importlib.wasm[2979]:0x27aeed)
at main (_freeze_importlib.wasm[42]:0x4a08)
at __original_main (_freeze_importlib.wasm[7190]:0x69da7e)
at _start (_freeze_importlib.wasm[30]:0x3ca8)
╰─▶ 2: bad_sig
Now the files are reachable, and feeze_importlib is crashing.
We need better debug information...
Meet "The Pain of Debugging WebAssembly".
