Windows 11 bug: Instead of .exe file StaticCompiler.jl creates two files: hello.ll and wrapper.c

[JohnClaw]

I typed intsructions from readme:

using StaticCompiler, StaticTools
hello() = println(c"Hello, world!")
compile_executable(hello, (), "c:/binary")

[PhilippeMaincon]

I find the same behaviour on Windows 10.

StaticCompiler.jl on Windows would be incredibly useful for me. I am not competent to help with compiling, alas.

[bluebug]

The code causing the problem is here, \c should be /c:

cmd \c clang # Not clear if the cmd \c is necessary

[PhilippeMaincon]

I made the change (on my machine) as @bluebug suggests, and this causes an error (which actually means progress):
'clang' is not recognized as an internal or external command, operable program or batch file.

I assume clang is a C compiler.

Do I already have clang installed with Julia and/or StaticCompiler.jl and I need to do something to make CMD know about it, or is clang not a part of the installation, and I have to install clang myself?

[bluebug]

I made the change (on my machine) as @bluebug suggests, and this causes an error (which actually means progress): 'clang' is not recognized as an internal or external command, operable program or batch file.

I assume clang is a C compiler.

Do I already have clang installed with Julia and/or StaticCompiler.jl and I need to do something to make CMD know about it, or is clang not a part of the installation, and I have to install clang myself?

I'm using clang.exe from preinstalled llvm and add "C:\Program Files\LLVM\bin" into windows paths.

clang --version 
clang version 17.0.6 
Target: x86_64-pc-windows-msvc 
Thread model: posix 
InstalledDir: C:\Program Files\LLVM\bin 

maybe you can find clang.exe from the path "C:\Users\yourname\.julia\artifacts"

[Thomas008]

The problem is in the function generate_executable() of StaticCopmiler.jl in the line
cmd \c clang # Not clear ...
It must be
cmd /c clang

It is kind of a typo that was unfortunately integrated.

[Thomas008]

The clang.exe in the artifacts does not work for Windows, as pointed out in JuliaPackaging/Yggdrasil#8015
There are versions of clang that work for Windows. They have to be integrated into the Clang_jll - package, i.e. into the artifacts.

[PhilippeMaincon]

Hi @Thomas008.

First, thank you for developping StaticCompiler, that's a very important work.

From JuliaPackaging/Yggdrasil#8015, I get the impression you have a workaround, using a local clang installation.

If that is the case, a "how to" (starting from installing clang) would be very valuable. Do you get an executable from releases.llvm.org ? Do we need to install LLVM or just Clang...?

[Thomas008]

Hi @PhilippeMaincon
thank you! Just a correction: I did not develop StaticCompiler, but adapt it to Windows. Yes, I did a workaround.
First you would add the package StaticCompiler. However, the file StaticCompiler.jl has still a small bug:
The line must be cmd /c clang (instead of cmd \c clang) as mentioned above.
Second, you need a clang that works for Windows. You probably don't need the whole LLVM project, only clang. The clang versions I used for Windows are e.g. on
https://github.com/llvm/llvm-project/releases/tag/llvmorg-17.0.5
and
https://github.com/mstorsjo/llvm-mingw/releases

However, we want to have repaired the clang in the artifact of the package Clang_jll, such that in StaticCompiler.jl one can use the function clang() provided by the package Clang_jll instead of applying cmd /c clang.

[anthonyanikos]

Hello, can i ask, this is not solved yet? I have absolutely same problems on Windows and dont understand what do ;(

[Thomas008]

One problem is that on Windows the clang in the artifact of Clang_jll does not work yet. You have to install a clang that works on Windows on your own. Do you have clang?

[anthonyanikos]

We made it! (c)

1. Run vs_BuildTools.exe and check:
   a) MSVC 143 - VS 2022.
   b). CMake c++ for Windows
   c). SDK For windows 10 (latest)

2. Get here https://github.com/llvm/llvm-project/releases/tag/llvmorg-17.0.5
   and https://github.com/mstorsjo/llvm-mingw/releases
   LLVM-17.0.5-win64.exe and install

3. My julia was 1.10.5 win 64

after it:

Pkg.add("StaticCompiler") and may be also StaticTools

and:

using StaticCompiler, StaticTools
hello() = println(c"Hello, world!")
compile_executable(hello, (), "c:/binary")

will generate u exe near 100kb.

Thank u a lot to everyone who made it possible!

Can anybody say, how can i compile .jl file?

Anthony

[Thomas008]

Congratulations!

Can anybody say, how can i compile .jl file?

I don't know, what you mean. In the example you described, you compile the function hello() (that is implemented within some jl-file).

[bluebug]

Belated test(√ passed):

1. remove LLVM from path
2. update StaticCompiler to new version
3. switch julia from 1.11.0 rc3 to 1.10.5
4. test hello code

windows 11
julia 1.10.5
StaticCompiler v0.7.2
=============
clang version 18.1.4
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: D:\llvm18\bin

using StaticCompiler, StaticTools
hello() = println(c"Hello, world!")
paths = ENV["PATH"] * ";D:\\llvm18\\bin"
withenv("PATH"=>paths) do
           compile_executable(hello, (), "./")
end

"E:\\codes\\test\\hello.exe"

Additional test(× not passed):

1. add Clang
2. with path of clang.exe in Clang
3. test hello code

using StaticCompiler, StaticTools
using Clang
hello() = println(c"Hello, world!")
paths = ENV["PATH"] * ";" * Clang.LLVM_DIR * "tools"
withenv("PATH"=>paths) do
           compile_executable(hello, (), "./")
end

ERROR: failed process: Process(`clang -Wno-override-module ./wrapper.c ./hello.ll -o ./hello.exe`, ProcessExited(3221225781)) [3221225781]

some guesses:

1. Clang(v15) is outdated
2. when execute clang.exe in Clang.LLVM_DIR * "tools" has not output in window terminal also

[anthonyanikos]

Hello! Thank u a lot.
I am stucked with stupid may be problem:
How can i use println(c"Hello") with some string variable instead "Hello" ?

my_str="Hello"
println(my_str)

I ve tested tens of variants with C-like string, memory aloc functions with no luck - Julia is arguing me with symbol outrange

error LNK2019: reference to an unresolved external symbol ijl_apply_generic in function julia_println_1162.

thank u a lot!
Anyway it's cool to have highlevel language with C compilation.. Still impressed! WOW! Its like may be Mojo but works on Windows!!!

[bluebug]

Hello! Thank u a lot. I am stucked with stupid may be problem: How can i use println(c"Hello") with some string variable instead "Hello" ?

my_str="Hello" println(my_str)

I ve tested tens of variants with C-like string, memory aloc functions with no luck - Julia is arguing me with symbol outrange

error LNK2019: reference to an unresolved external symbol ijl_apply_generic in function julia_println_1162.

thank u a lot! Anyway it's cool to have highlevel language with C compilation.. Still impressed! WOW! Its like may be Mojo but works on Windows!!!

you should take a look at StaticTools, c"Hello" => StaticTools.@c_str("Hello") => StaticTools.StaticString("Hello"), so you can use StaticTools.@c_str(my_str) or StaticTools.StaticString(my_str).

help?> StaticTools.@c_str
  @c_str -> StaticString

  Construct a StaticString, such as c"Foo".

  A StaticString should generally behave like a base Julia String, but is explicitly null-terminated,
  mutable, and standalone-StaticCompiler safe (does not require libjulia).

  Examples
  ========

  julia> c"Hello there!"
  c"Hello there!"

  julia> c"foo" == "foo"
  true

help?> StaticTools.StaticString
  StaticString{N}

  A stringy type which should generally behave like a base Julia String, but is explicitly null-terminated,
  mutable, and standalone-StaticCompiler safe (does not require libjulia).

  Can be constructed with the c"..." string macro.

  Unlike base Julia Strings, slicing does not create a copy, but rather a view. You are responsible for
  ensuring that any such views are null-terminated if you wish to pass them to any functions (including
  most system IO) that expect null-termination.

  Indexing a StaticString out of bounds does not throw a BoundsError; much as if @inbounds were enabled,
  indexing a StaticString incurs a strict promise by the programmer that the specified index is inbounds.
  Breaking this promise will result in segfaults or memory corruption.

  Examples
  ========

  julia> s = c"Hello world!"
  c"Hello world!"

  julia> s[8:12] = c"there"; s
  c"Hello there!"

  julia> s[1:5]
  StringView: "Hello"

  julia> s[1:5] == "Hello"
  true

  julia> StaticString(s[1:5])
  c"Hello"

  ─────────────────────────────────────────────────────────────────────────────────────────────────────────

  StaticString{N}(undef)

  Construct an uninitialized N-byte StaticString

  StaticString(data::NTuple{N,UInt8})

  Construct a StaticString containing the N bytes specified by data. To yield a valid string, data must be
  null-terminated, i.e., end in 0x00.

  StaticString(s::AbstractStaticString)

  Construct a StaticString containing the same data as the input string s.

  Examples
  ========

  julia> data = (0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x77, 0x6f, 0x72, 0x6c, 0x64, 0x21, 0x00);

  julia> s = StaticString(data)
  c"Hello world!"

  julia> StaticString(s[1:5])
  c"Hello"

But this is irrelevant to this topic, so this topic should be closed.

[anthonyanikos]

Sorry for offtop, but still no luck ;(

[bluebug]

according to my limited knowledge:

1. should not use String type causing julia GC, as StaticCompiler did not link libjulia:

Limitations
GC-tracked allocations and global variables do not work with compile_executable or compile_shlib. This has some interesting consequences, including that all functions within the function you want to compile must either be inlined or return only native types (otherwise Julia would have to allocate a place to put the results, which will fail).
Since error handling relies on libjulia, you can only throw errors from standalone-compiled (compile_executable / compile_shlib) code if an explicit overload has been defined for that particular error with @device_override (see [quirks.jl](https://github.com/tshort/StaticCompiler.jl/blob/master/src/quirks.jl)).
Type instability. Type unstable code cannot currently be statically compiled via this package.
Doesn't work on Windows (but works in WSL on Windows 10+). PRs welcome.

2. have a look at StaticTools.jl/test/testmallocstring.jl at main · brenhinkeller/StaticTools.jl (github.com)

3. one test work at my julia env:

function hello()
           c_hello = c"hello "
           c_name = c"world"
           c_greet = c_hello * c_name
           println(c_greet)
       end
hello (generic function with 1 method)

julia>

julia> compile_executable(hello, (), "./")
"E:\\codes\\test\\hello.exe"

[Thomas008]

The steps mentioned by @bluebug

remove LLVM from path
update StaticCompiler to new version
switch julia from 1.11.0 rc3 to 1.10.5
test hello code

add Clang
with path of clang.exe in Clang
test hello code

suggest to me that there is a strong need for two important things:

1. get the new version of StaticCompiler run also for the julia 1.11
2. get a working clang for Windows into the artifact of Clang_jll (used by StaticCompiler)

[anthonyanikos]

Hello, thank u very mouch for answering!
But stll can't do simplest output to screen. What i need, just print 1+2 for example. How can i do that? It's not string variable, its result of computation...

with respect, Anthony

[bluebug]

Hello, thank u very mouch for answering! But stll can't do simplest output to screen. What i need, just print 1+2 for example. How can i do that? It's not string variable, its result of computation...

with respect, Anthony

one test passed, and more info you can find in doc: https://brenhinkeller.github.io/StaticTools.jl/dev/

using StaticCompiler, StaticTools

function add(a, b)
    a + b
end

# 1. use StaticTools.printf to replace println, doc: https://brenhinkeller.github.io/StaticTools.jl/dev/#StaticTools.printf-Tuple{MallocString}
# 2. return 0 => pretend to be a C program
function hello()
    c_hello = c"hello"
    c_name = c"world"
    printf(c"%s", c_hello)
    printf(c" %s!\n", c_name)

    a = 5
    b = 6
    c = add(a, b)
    printf(c"%d\n", c)

    return 0
end

compile_executable(hello, (), "./")

[bluebug]

The steps mentioned by @bluebug

remove LLVM from path
update StaticCompiler to new version
switch julia from 1.11.0 rc3 to 1.10.5
test hello code

add Clang
with path of clang.exe in Clang
test hello code

suggest to me that there is a strong need for two important things:

1. get the new version of StaticCompiler run also for the julia 1.11
2. get a working clang for Windows into the artifact of Clang_jll (used by StaticCompiler)

some hack about Clang test and passed, so maybe StaticCompiler can work with Clang/Clang_jll using correct env 😄

1. using julia 1.10.5 env, add Clang/Clang_jll

  [40e3b903] Clang v0.18.3
  [81625895] StaticCompiler v0.7.2
  [86c06d3c] StaticTools v0.8.10
⌅ [0ee61d77] Clang_jll v15.0.7+10

2. wrap compile_executable with hack env using Clang_jll.PATH and Clang_jll.clang()

using StaticCompiler, StaticTools

function add(a, b)
    a + b
end

# 1. use StaticTools.printf to replace println, doc: https://brenhinkeller.github.io/StaticTools.jl/dev/#StaticTools.printf-Tuple{MallocString}
# 2. return 0 => pretend to be a C program
function hello()
    c_hello = c"hello"
    c_name = c"world"
    printf(c"%s", c_hello)
    printf(c" %s!\n", c_name)

    a = 5
    b = 6
    c = add(a, b)
    printf(c"%d\n", c)

    return 0
end


using Clang_jll

clang_lib = normpath(joinpath(Clang_jll.PATH[], "..\\bin"))
paths = ENV["PATH"] * ";" * clang_lib * ";" * Clang_jll.PATH[]

withenv("PATH" => paths) do
    clang() do clang_exe
        compile_executable(hello, (), "./")
    end
end

[anthonyanikos]

Hello, thank u very mouch for answering! But stll can't do simplest output to screen. What i need, just print 1+2 for example. How can i do that? It's not string variable, its result of computation...
with respect, Anthony

one test passed, and more info you can find in doc: https://brenhinkeller.github.io/StaticTools.jl/dev/

using StaticCompiler, StaticTools

function add(a, b)
    a + b
end

# 1. use StaticTools.printf to replace println, doc: https://brenhinkeller.github.io/StaticTools.jl/dev/#StaticTools.printf-Tuple{MallocString}
# 2. return 0 => pretend to be a C program
function hello()
    c_hello = c"hello"
    c_name = c"world"
    printf(c"%s", c_hello)
    printf(c" %s!\n", c_name)

    a = 5
    b = 6
    c = add(a, b)
    printf(c"%d\n", c)

    return 0
end

compile_executable(hello, (), "./")

Thank u very mouch, now it's fully works!!

With respect, Anthony

[bluebug]

Final conclusion, Clang_jll.clang() set paths into env but lost path of clang dlls causing clang.exe crash, so I patch local StaticCompiler.jl and the hello test passed without hack.

# file StaticCompiler.jl
...
import Clang_jll # to get some paths from Clang_jll
...
function generate_executable(funcs::Union{Array,Tuple}, path=tempname(), name=fix_name(first(first(funcs))), filename=name;
                             demangle = true,
                             cflags = ``,
                             target::StaticTarget=StaticTarget(),
                             llvm_to_clang::Bool = Sys.iswindows(),
                             kwargs...
                             )
...
        if llvm_to_clang # (required on Windows)
            # Use clang (llc) to generate an executable from the LLVM IR
            cclang = if Sys.iswindows()
                exec_path *= ".exe"
                # add clang paths into PATH env, so that clang.exe can load dlls from the paths
                clang_dlls = normpath(joinpath(Clang_jll.PATH[], "..\\bin"))
                clang_libs = Clang_jll.LIBPATH[]
                clang_exe = Clang_jll.clang_path
                PATH = clang_dlls * ";" * clang_libs * ";" * ENV["PATH"]
                setenv(`$clang_exe`,"PATH"=>PATH)
            elseif Sys.isapple()
                `clang`
            else
                clang()
            end
            run(`$cclang -Wno-override-module $wrapper_path $obj_or_ir_path -o $exec_path`)
...

[Thomas008]

I tried those workarounds, but get these error:
clang: error: unable to execute command: program not executable.
When calling clang.exe in the artifact, error messages occur that some dll's are not found. They are neither in the tools-, nor in the bin-directory. More specific the following dll's are not found:
libclang-cpp.dll, libLLVM-15jl.dll, libstdc++-6.dll.
At least libclang-cpp.dll is in the bin-directory. But libLLVM-15jl.dllandlibstdc++-6.dll` are not there.
Do you have any suggestions, how to repair this? Does Clang_jll not provide these dll's?

[bluebug]

I tried those workarounds, but get these error: clang: error: unable to execute command: program not executable. When calling clang.exe in the artifact, error messages occur that some dll's are not found. They are neither in the tools-, nor in the bin-directory. More specific the following dll's are not found: libclang-cpp.dll, libLLVM-15jl.dll, libstdc++-6.dll. At least libclang-cpp.dll is in the bin-directory. But libLLVM-15jl.dllandlibstdc++-6.dll` are not there. Do you have any suggestions, how to repair this? Does Clang_jll not provide these dll's?

@Thomas008 we can find the llvm libs in julia bin path, and libclang libs in Clang.jl bin path.

The main reason is that Clang_jll.clang() only add 4 julia paths into env PATH, but without libclang libs in the bin path of Clang.jl, causing error when executing clang.exe in windows.

julia> using Clang_jll

julia> clang_lib = normpath(joinpath(Clang_jll.PATH[], "..\\bin"))
"C:\\Users\\***\\.julia\\artifacts\\7d2877da43b3fef993c8488c267bb543f27414f8\\bin"

julia> readdir(clang_lib)
2-element Vector{String}:
 "libclang-cpp.dll"
 "libclang.dll"

julia> readdir(Sys.BINDIR)
59-element Vector{String}:
......
 "libLLVM-15jl.dll"
......
 "libstdc++-6.dll"
......

julia> c = Clang_jll.clang();

julia> split(c.env[1],";")
59-element Vector{SubString{String}}:
 "PATH=C:\\Users\\***\\AppData\\Local\\julias\\julia-1.10\\bin"
 "C:\\Users\\***\\AppData\\Local\\julias\\julia-1.10\\bin\\..\\lib\\julia"
 "C:\\Users\\***\\AppData\\Local\\julias\\julia-1.10\\bin\\..\\lib"
 "C:\\Users\\***\\AppData\\Local\\julias\\julia-1.10\\bin"
......

julia> split(ENV["PATH"],";")
55-element Vector{SubString{String}}:
......

the fatest patch without touch Clang_jll, just add Clang.jl bin path to cmd env PATH in file StaticCompiler.jl:

# file StaticCompiler.jl
...
+ import Clang_jll # to get some paths from Clang_jll
...
function generate_executable(funcs::Union{Array,Tuple}, path=tempname(), name=fix_name(first(first(funcs))), filename=name;
                             demangle = true,
                             cflags = ``,
                             target::StaticTarget=StaticTarget(),
                             llvm_to_clang::Bool = Sys.iswindows(),
                             kwargs...
                             )
...
        if llvm_to_clang # (required on Windows)
            # Use clang (llc) to generate an executable from the LLVM IR
            cclang = if Sys.iswindows()
                exec_path *= ".exe"
-               `clang`
+              # add clang paths into PATH env, so that clang.exe can load dlls from the paths
+              clang_lib = normpath(joinpath(Clang_jll.PATH[], "..\\bin"))
+              paths = clang_lib * ";" * Clang_jll.PATH[] * ";" * ENV["PATH"]
+              withenv(clang, "PATH" => paths)
            elseif Sys.isapple()
                `clang`
            else
                clang()
            end
            run(`$cclang -Wno-override-module $wrapper_path $obj_or_ir_path -o $exec_path`)
...