Dogfooding instructions for .NET 8 NativeAOT on macOS x64 via Rosetta

[christianscheuer]

We're extremely happy that NativeAOT on Mac x64/arm64 should now work in .NET 8 daily builds.
As one of the early proponents of CoreRT, we're looking forward to help beta test this in a large real world application.

However, I'm running into issues with using the .NET 8 daily builds of the SDK with ILC.
We need to build macOS universal binaries (eventually) so all of our builds of x64 are done via Rosetta 2 on an Apple Silicon device.

Here's the repro:
https://github.com/christianscheuer/nativeaot-net8

Steps:

• I installed macOS x64 SDK version 8.0.100-alpha.1.22605.4 from github.com/dotnet/installer
• All commands are run from a Rosetta Terminal.
• The SDK host is x64, the SDK is x64.
• Run ./build.sh which essentially just ensures we're in Rosetta, then runs dotnet publish -c Release -r osx-x64

Expected result:

• It builds

Actual result:

MSBuild version 17.5.0-preview-22564-01+d14b74d6f for .NET
  Determining projects to restore...
  Restored /Users/chr/d/d2/nativeaot-net8/nativeaot-net8.csproj (in 327 ms).
/usr/local/share/dotnet/x64/sdk/8.0.100-alpha.1.22605.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(275,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/Users/chr/d/d2/nativeaot-net8/nativeaot-net8.csproj]
  nativeaot-net8 -> /Users/chr/d/d2/nativeaot-net8/bin/Release/net8.0/osx-x64/nativeaot-net8.dll
/Users/chr/.nuget/packages/microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/build/Microsoft.DotNet.ILCompiler.SingleEntry.targets(51,5): warning : Delete explicit 'Microsoft.DotNet.ILCompiler' package reference in your project file. Explicit 'Microsoft.DotNet.ILCompiler' package reference can run into version errors. [/Users/chr/d/d2/nativeaot-net8/nativeaot-net8.csproj]
/Users/chr/.nuget/packages/microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/build/Microsoft.NETCore.Native.Publish.targets(70,5): error : Add a PackageReference for 'runtime.osx-arm64.Microsoft.DotNet.ILCompiler' to allow cross-compilation for x64 [/Users/chr/d/d2/nativeaot-net8/nativeaot-net8.csproj]

What's weird is that it claims it can't do cross-compilation, which I'm not actually doing. This is the x64 SDK running in an x64 terminal trying to build an x64 release. Furthermore, it requests the arm64 PackageReference. Adding that reference changes nothing, it reports the same error. It also complains about an explicit package reference that I'm not seeing.

I tried following the instructions on dotnet/installer to add the Nuget.config, but that ends up in yet other errors.

I shared the output of dotnet --info below.

Is there anything I should do differently to consume and test .NET 8 NativeAOT for macOS x64?
Are there dogfooding instructions for this scenario somewhere that I haven't located?

`dotnet --info` output:

.NET SDK:
 Version:   8.0.100-alpha.1.22605.4
 Commit:    af14110139

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  12.5
 OS Platform: Darwin
 RID:         osx.12-x64
 Base Path:   /usr/local/share/dotnet/x64/sdk/8.0.100-alpha.1.22605.4/

Host:
  Version:      8.0.0-alpha.1.22559.2
  Architecture: x64
  Commit:       dd7fdb723d

.NET SDKs installed:
  5.0.408 [/usr/local/share/dotnet/x64/sdk]
  6.0.403 [/usr/local/share/dotnet/x64/sdk]
  8.0.100-alpha.1.22605.4 [/usr/local/share/dotnet/x64/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 5.0.17 [/usr/local/share/dotnet/x64/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.11 [/usr/local/share/dotnet/x64/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.0-alpha.1.22558.5 [/usr/local/share/dotnet/x64/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 5.0.17 [/usr/local/share/dotnet/x64/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.11 [/usr/local/share/dotnet/x64/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.0-alpha.1.22557.12 [/usr/local/share/dotnet/x64/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.0-alpha.1.22559.2 [/usr/local/share/dotnet/x64/shared/Microsoft.NETCore.App]

Other architectures found:
  arm64 [/usr/local/share/dotnet]
    registered at [/etc/dotnet/install_location_arm64]

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

cc @jkotas

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

Issue Details

---

We're extremely happy that NativeAOT on Mac x64/arm64 should now work in .NET 8 daily builds.
As one of the early proponents of CoreRT, we're looking forward to help beta test this in a large real world application.

However, I'm running into issues with using the .NET 8 daily builds of the SDK with ILC.
We need to build macOS universal binaries (eventually) so all of our builds of x64 are done via Rosetta 2 on an Apple Silicon device.

Here's the repro:
https://github.com/christianscheuer/nativeaot-net8

Steps:

• I installed macOS x64 SDK version 8.0.100-alpha.1.22605.4 from github.com/dotnet/installer
• All commands are run from a Rosetta Terminal.
• The SDK host is x64, the SDK is x64.
• Run ./build.sh which essentially just ensures we're in Rosetta, then runs dotnet publish -c Release -r osx-x64

Expected result:

• It builds

Actual result:

MSBuild version 17.5.0-preview-22564-01+d14b74d6f for .NET
  Determining projects to restore...
  Restored /Users/chr/d/d2/nativeaot-net8/nativeaot-net8.csproj (in 327 ms).
/usr/local/share/dotnet/x64/sdk/8.0.100-alpha.1.22605.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(275,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/Users/chr/d/d2/nativeaot-net8/nativeaot-net8.csproj]
  nativeaot-net8 -> /Users/chr/d/d2/nativeaot-net8/bin/Release/net8.0/osx-x64/nativeaot-net8.dll
/Users/chr/.nuget/packages/microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/build/Microsoft.DotNet.ILCompiler.SingleEntry.targets(51,5): warning : Delete explicit 'Microsoft.DotNet.ILCompiler' package reference in your project file. Explicit 'Microsoft.DotNet.ILCompiler' package reference can run into version errors. [/Users/chr/d/d2/nativeaot-net8/nativeaot-net8.csproj]
/Users/chr/.nuget/packages/microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/build/Microsoft.NETCore.Native.Publish.targets(70,5): error : Add a PackageReference for 'runtime.osx-arm64.Microsoft.DotNet.ILCompiler' to allow cross-compilation for x64 [/Users/chr/d/d2/nativeaot-net8/nativeaot-net8.csproj]

What's weird is that it claims it can't do cross-compilation, which I'm not actually doing. This is the x64 SDK running in an x64 terminal trying to build an x64 release. Furthermore, it requests the arm64 PackageReference. Adding that reference changes nothing, it reports the same error. It also complains about an explicit package reference that I'm not seeing.

I tried following the instructions on dotnet/installer to add the Nuget.config, but that ends up in yet other errors.

I shared the output of dotnet --info below.

Is there anything I should do differently to consume and test .NET 8 NativeAOT for macOS x64?
Are there dogfooding instructions for this scenario somewhere that I haven't located?

`dotnet --info` output:

.NET SDK:
 Version:   8.0.100-alpha.1.22605.4
 Commit:    af14110139

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  12.5
 OS Platform: Darwin
 RID:         osx.12-x64
 Base Path:   /usr/local/share/dotnet/x64/sdk/8.0.100-alpha.1.22605.4/

Host:
  Version:      8.0.0-alpha.1.22559.2
  Architecture: x64
  Commit:       dd7fdb723d

.NET SDKs installed:
  5.0.408 [/usr/local/share/dotnet/x64/sdk]
  6.0.403 [/usr/local/share/dotnet/x64/sdk]
  8.0.100-alpha.1.22605.4 [/usr/local/share/dotnet/x64/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 5.0.17 [/usr/local/share/dotnet/x64/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.11 [/usr/local/share/dotnet/x64/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.0-alpha.1.22558.5 [/usr/local/share/dotnet/x64/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 5.0.17 [/usr/local/share/dotnet/x64/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.11 [/usr/local/share/dotnet/x64/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.0-alpha.1.22557.12 [/usr/local/share/dotnet/x64/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.0-alpha.1.22559.2 [/usr/local/share/dotnet/x64/shared/Microsoft.NETCore.App]

Other architectures found:
  arm64 [/usr/local/share/dotnet]
    registered at [/etc/dotnet/install_location_arm64]

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

cc @jkotas

Author:	christianscheuer
Assignees:	-
Labels:	area-NativeAOT-coreclr
Milestone:	-

[jkotas]

This is the x64 SDK running in an x64 terminal trying to build an x64 release.

The native AOT compiler defaults to running natively:

runtime/src/coreclr/nativeaot/BuildIntegration/Microsoft.DotNet.ILCompiler.SingleEntry.targets

Line 20 in c8f9f29

<OSHostArch>$([System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture.ToString().ToLowerInvariant)</OSHostArch>

You should be able to override this by setting IlcHostArch to x64. It will force the compiler to run as x64 emulated by Rosetta.

[agocke]

I believe the warning you're seeing is this bug: #78647. It's an artifact of how we're downloading the package now. The warning shouldn't be appearing.

[christianscheuer]

Thanks, both. @jkotas I can confirm setting IlcHostArch to x64 worked. Is that the right default thing to do though (IlcHostArch using native arch instead of parent process arch)?
I imagine this would be confusing to anybody else who builds x64 on Apple Silicon (which would be anybody who needs to build universal binaries) - but maybe this would be fixed instead by making cross compilation "just work"?

The assumption we have is that if the shell is x64, every process under it should run in x64 via Rosetta. That's how all of our other build tools do it (node, yarn, electron, Makefiles, CMake, clang, etc.) and it makes it very easy to target multiple archs on an Apple Silicon machine and never worry about cross compilation because every process just inherits the arch from its parent.

[jkotas]

I agree with you that it would make more sense to use the SDK architecture as the host architecture for the IL compiler. PublishReadyToRun compilation works like that. There is no reason for PublishAot to be different.

[agocke]

Hmm, personally it sounds like the process architecture is the right default, no? I believe the x64 terminal config in MacOS works by changing the process arch, but not the machine arch. If the publish takes the current process arch then you could use the -arm64 SDK to easily compile for x64 by just running it in the x64 process.

[am11]

I tried following the instructions on dotnet/installer to add the Nuget.config,

Note that this file's name is case sensitive on Unix. Either all lower: nuget.config or exactly this casing: NuGet.Config.

I simply added an entry in the existing global config located on macOS at: ~/.nuget/NuGet/NuGet.Config

    <add key="dotnet8" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet8/nuget/v3/index.json" />

(after the existing line <add key="nuget.org"..)

With that in place, the cross-compilation works as well:

% dotnet8 new console -n helloworldapp          

# workaround for https://github.com/dotnet/installer/pull/14991
# (templates are currently generating net7.0 projects by default until that PR is merged)
% sed -i '' 's/net7/net8/' helloworldapp/helloworldapp.csproj 

% dotnet8 publish helloworldapp -a x64 -o helloworldapp/dist -p:PublishAot=true

% file helloworldapp/dist/helloworldapp                                        
helloworldapp/dist/helloworldapp: Mach-O 64-bit executable x86_64

dotnet8 is an alias defined in ~/.zprofile:

alias dotnet8=~/.dotnet8/dotnet

This is a refresh script ~/update-daily-build.sh (which I run once a day):

#!/usr/bin/env bash

SECONDS=0

DEST="$HOME/.dotnet8"

# recreate destination directory
rm -rf "$DEST"
mkdir "$DEST"

# download and install
curl -sSL https://aka.ms/dotnet/8.0.1xx/daily/dotnet-sdk-osx-arm64.tar.gz | tar xzf - -C "$DEST"

# refresh caches
"$DEST"/dotnet nuget locals all --clear

"$DEST"/dotnet --info

duration=$SECONDS
echo "$(($duration / 60)) minutes and $(($duration % 60)) seconds elapsed."

[vitek-karas]

If the publish takes the current process arch then you could use the -arm64 SDK to easily compile for x64 by just running it in the x64 process.

I think we should run everything with the same architecture as the SDK which started it, unless it's an explicit choice to do otherwise. Having a random mix is just confusing, and honestly unexpected. Not counting that there's no guarantee that there will be an arm64 runtime available on the machine, so this could even fail to run.

[christianscheuer]

Thanks so much, everybody.

@am11 these instructions on consuming daily builds are extremely valuable, thank you for sharing. So much better not to pollute the regular dotnet installation with these daily builds but have more control. The nuget package clearing is neat too.

I got the x64 build working as cross compilation now with @am11's instructions now using the arm64 SDK, and it works just fine using either x64 or arm64 terminal.

I can also target arm64 using the arm64 SDK using those instructions, when in an arm64 terminal. Curiously, if I am in a Rosetta terminal and invoke the arm64 SDK, instead of getting a nice error saying I shouldn't do that, it fails with linker errors.

To repro that, follow @am11's instructions to install dotnet8.

• Open Rosetta Terminal (or type arch -x86_64 zsh to open a sub shell in x64 mode).
• Follow @am11's repro instructions, then run this at the publish step: dotnet8 publish helloworldapp -a arm64 -o helloworldapp/distarm -p:PublishAot=true

dotnet8 publish output

MSBuild version 17.5.0-preview-22564-01+d14b74d6f for .NET
  Determining projects to restore...
  Restored /Users/chr/d/d2/helloworldapp/helloworldapp.csproj (in 89 ms).
/Users/chr/.dotnet8/sdk/8.0.100-alpha.1.22605.6/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(275,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/Users/chr/d/d2/helloworldapp/helloworldapp.csproj]
  helloworldapp -> /Users/chr/d/d2/helloworldapp/bin/Debug/net8.0/osx-arm64/helloworldapp.dll
/Users/chr/.nuget/packages/microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/build/Microsoft.DotNet.ILCompiler.SingleEntry.targets(51,5): warning : Delete explicit 'Microsoft.DotNet.ILCompiler' package reference in your project file. Explicit 'Microsoft.DotNet.ILCompiler' package reference can run into version errors. [/Users/chr/d/d2/helloworldapp/helloworldapp.csproj]
  Generating native code
  ld: warning: ignoring file /Users/chr/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/framework/libSystem.Net.Security.Native.a, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
  ld: warning: ignoring file /Users/chr/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/sdk/libbootstrapper.a, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
  ld: warning: ignoring file /Users/chr/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/sdk/libstdc++compat.a, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
  ld: warning: ignoring file obj/Debug/net8.0/osx-arm64/native/helloworldapp.o, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
  ld: warning: ignoring file /Users/chr/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/framework/libSystem.Security.Cryptography.Native.Apple.a, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
  ld: warning: ignoring file /Users/chr/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/sdk/libRuntime.WorkstationGC.a, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
  ld: warning: ignoring file /Users/chr/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/framework/libSystem.Security.Cryptography.Native.OpenSsl.a, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
  ld: warning: ignoring file /Users/chr/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/framework/libSystem.Globalization.Native.a, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
  ld: warning: ignoring file /Users/chr/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/framework/libSystem.Native.a, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
  ld: warning: ignoring file /Users/chr/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/framework/libSystem.IO.Compression.Native.a, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
  Undefined symbols for architecture x86_64:
    "_main", referenced from:
       implicit entry/start for main executable
  ld: symbol(s) not found for architecture x86_64
  clang: error: linker command failed with exit code 1 (use -v to see invocation)
/Users/chr/.nuget/packages/microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/build/Microsoft.NETCore.Native.targets(346,5): error MSB3073: The command ""clang" "obj/Debug/net8.0/osx-arm64/native/helloworldapp.o" -o "bin/Debug/net8.0/osx-arm64/native/helloworldapp" /Users/chr/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/sdk/libbootstrapper.a /Users/chr/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/sdk/libRuntime.WorkstationGC.a /Users/chr/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/sdk/libstdc++compat.a /Users/chr/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/framework/libSystem.Native.a /Users/chr/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/framework/libSystem.Globalization.Native.a /Users/chr/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/framework/libSystem.IO.Compression.Native.a /Users/chr/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/framework/libSystem.Net.Security.Native.a /Users/chr/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/framework/libSystem.Security.Cryptography.Native.Apple.a /Users/chr/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/8.0.0-alpha.1.22559.2/framework/libSystem.Security.Cryptography.Native.OpenSsl.a -g -Wl,-rpath,'@executable_path' -ldl -lm -lobjc -lswiftCore -lswiftFoundation -lz -licucore -L/usr/lib/swift -framework CoreFoundation -framework CryptoKit -framework Foundation -framework Security -framework GSS" exited with code 1. [/Users/chr/d/d2/helloworldapp/helloworldapp.csproj]

[agocke]

If the publish takes the current process arch then you could use the -arm64 SDK to easily compile for x64 by just running it in the x64 process.

I think we should run everything with the same architecture as the SDK which started it, unless it's an explicit choice to do otherwise. Having a random mix is just confusing, and honestly unexpected. Not counting that there's no guarantee that there will be an arm64 runtime available on the machine, so this could even fail to run.

I think @christianscheuer summed up my understanding of the Apple approach to these things, and I agree this is the way it seems to work:

The assumption we have is that if the shell is x64, every process under it should run in x64 via Rosetta. That's how all of our other build tools do it (node, yarn, electron, Makefiles, CMake, clang, etc.) and it makes it very easy to target multiple archs on an Apple Silicon machine and never worry about cross compilation because every process just inherits the arch from its parent.

In this case we would target the arch of the process. For Native AOT at least, the concept of not having an x64 runtime doesn't matter -- the binary will have the correct architecture by definition.

[agocke]

Actually, when thinking about it I'm not sure everything works the way just described. In particular, does running in an x64 environment always run the x64 version of those tools, or do those tools just assume the architecture of the machine by the process they're running in?

This is something to investigate. Is clang running as an x64 process or is clang targeting x64? If it's running as an x64 process that does simplify things -- this would basically be a native targeting scenario, not a cross-targeting scenario.

That said, there is some inefficiency here, as in theory there could be performance improvements to actually doing cross-compilation using the arm64 SDK.