[MERGE #1717 @dilijev] MicroBuild v2: Fix binary and package version metadata; refactor and improve scripts.

Merge pull request #1717 from dilijev:microbuildv2

MicroBuild v2: Fix binary and package version metadata; refactor and improve scripts.

Binary and Package Versions
===========================

Binary File Version Metadata
----------------------------

Enables correct embedding of version string in file info. All binaries will now
embed File Version metadata as follows:

    <MAJOR>.<MINOR>.<PATCH>.0 (<branch>:<hash>.<build_high>.<build_low>.<push_date>)

Example:

    1.2.1.0 (release/1.2:9786fcdbcaa6963a5ccc6648b7aa9bd9dd51d9ec.00008.12122.161007-1756)

`MAJOR`, `MINOR`, and `PATCH` correspond to SemVer fields and will match the
version of ChakraCore the package corresponds to. The fourth field would have
been used to indicate security fixes (a.k.a. `QFE`) but because a fourth field
is not allowed in NuGet package version numbers, but is standard for versions of
binaries produced from VS projects, we are keeping the fourth field in the
binary data and locking it to 0, and omitting it from NuGet package version
numbers and release numbers from now on.

Release NuGet Package Versions
------------------------------

The format of version strings for released NuGet packages will continue to be:

    Microsoft.ChakraCore.<MAJOR>.<MINOR>.<PATCH>.nupkg

Example:

    Microsoft.ChakraCore.1.2.1.nupkg

Preview NuGet Package Versions
------------------------------

The format of version strings for preview NuGet packages will now be:

    Microsoft.ChakraCore.<MAJOR>.<MINOR>.<PATCH>-<SPECIAL>.nupkg

Where `<SPECIAL>` is:

    preview-<build_high>-<build_low>

This solves the problem of getting enough information into the package preview
version string (`<SPECIAL>`) to order builds correctly while still guaranteeing
that we fit the requirements for the `<SPECIAL>` field:

1. It must start with a letter
2. It can only contain the characters [a-zA-Z0-9-]
3. It must be 20 characters or less

Example:

    Microsoft.ChakraCore.1.2.1-preview-00008-12122.nupkg

Miscellaneous Version-Related Changes
-------------------------------------

Add `PATCH` version to `CommonDefines.h` so that it doesn't need to be manually
coded into builds. (By reducing user intervention for package versions we make
it more feasible to automatically release preview packages from multiple
branches.)

Extract `MAJOR`, `MINOR`, `PATCH`, and `QFE` version numbers from the source
code by grepping for values of the `#defines` that correspond to those values.

Refactorings and Script Parameters
==================================

Merge `pogo_build.ps1` into `run_build.ps1`. Passing the required pogo-related
flags to `run_build.ps1` produces a POGO build. No need to have separate
scripts.

Compute `ObjectDirectory` and use that for `Chakra.Generated.BuildInfo.props`
unless `TF_BUILD_BUILDDIRECTORY` is specified in the PATH.

Infer `$binpath`.

For many parameters that were previously specified as environment variables from
the environment of the build infrastructure, it is now possible to specify them
as script parameters directly, where script parameters will take precedence over
the values of the corresponding environment variables. This can be helpful when
running scripts manually where you want to quickly experiment with the value of
a variable without having to write it to the environment first. Since it is
sometime valuable to get that variable from the environment, if the script
parameter is unspecified, the value of the environment variable will be used
instead. Additionally, if any parameter is unspecified by either script
parameter or environment variable, a reasonable default is encoded in the script
logic.

In order of precedence, all values will be determined by checking for values in
the following order, if applicable:

1. Script parameter
2. Environment variable
3. Script logic to retrieve value
4. Default value

Add parameters with corresponding environment variables to specify skipping
certain paths through the build scripts, which can save time when testing
certain changes by ignoring expensive but unimportant parts of the builds.
Specifying an environment value at the time the build is queued is easier than
modifying the build definition, so adding the environment variable option adds
convenience when testing with the production build definitions.

* `run_build.ps1 -skipPogo` or set `SKIP_POGO` to a non-empty string will skip
  the POGO phase of the build, if it would have been a POGO build. Useful if the
  build binaries produced at the correct locations is enough and you're most
  interested in testing the non-POGO-related aspects of the build scripts.
* `post_build.ps1 -skipTests` or set `SKIP_TESTS` to a non-empty string will
  skip running tests, which can be lengthy. This is useful if the quality of the
  binaries produced is not important and you're interested in testing other
  aspects of the build scripts.

Miscellaneous
-------------

Reduce dependencies on data from build server environments so that these scripts
can be used to perform local builds.

Update `UseValueOrDefault` to take any number of arguments.

Reduce dependency on `locate_msbuild.ps1`.

Add `$BuildSubtype` to 'not enough info' error.

Check for the existence of `nuget.exe` before trying to invoke it. This helps
when running in environments where NuGet is not installed or is not available.
Can avoid errors related to NuGet package restore when running on a build server
with a custom NuGet location and a specific NuGet Task separate from our
scripts. Keeping the NuGet invocation in our scripts allows us to still do local
builds without MicroBuild infrastructure configured on the dev machine (we don't
take a hard dependency on the aforementioned NuGet Task.

Author: dilijev

.gitignore

@@ -32,3 +32,5 @@ Build/Chakra.Core.VC.opendb
 test/benchmarks/*.txt
 test/benchmarks/*.dpl
 *.nupkg
+_DROP
+packages.config

Build/Common.Build.Default.props

@@ -58,7 +58,6 @@
     <PlatformPathNameAlt Condition="'$(Platform)'=='x64'">amd64</PlatformPathNameAlt>
   </PropertyGroup>
 
-
   <!-- Default output directories -->
   <PropertyGroup>
     <OutBaseDir Condition="'$(OutBaseDir)'!=''">$(OutBaseDir)\$(SolutionName)</OutBaseDir>
@@ -68,7 +67,11 @@
   </PropertyGroup>
 
   <!-- Import generated build info -->
-  <Import Project="$(TF_BUILD_BUILDDIRECTORY)\Chakra.Generated.BuildInfo.props" Condition="'$(TF_BUILD_BUILDDIRECTORY)' != '' AND exists('$(TF_BUILD_BUILDDIRECTORY)\Chakra.Generated.BuildInfo.props')" />
+  <PropertyGroup>
+    <ObjectDirectory Condition="'$(TF_BUILD_BUILDDIRECTORY)'!=''">$(TF_BUILD_BUILDDIRECTORY)</ObjectDirectory>
+    <ObjectDirectory Condition="'$(TF_BUILD_BUILDDIRECTORY)'==''">$(IntBaseDir)\obj\$(PlatformPathName.ToLower())_$(Configuration.ToLower())</ObjectDirectory>
+  </PropertyGroup>
+  <Import Project="$(ObjectDirectory)\Chakra.Generated.BuildInfo.props" Condition="'$(ObjectDirectory)'!='' AND exists('$(ObjectDirectory)\Chakra.Generated.BuildInfo.props')" />
 
   <!-- Output directories -->
   <PropertyGroup>

Build/scripts/finalize_build.ps1

@@ -101,7 +101,7 @@ $buildFlavorJson | Add-Member -type NoteProperty -name flavor -value $Env:BuildC
 $buildFlavorJson | Add-Member -type NoteProperty -name subtype -value $Env:BuildSubtype
 
 $buildFlavorJson | ConvertTo-Json | Write-Output
-$buildFlavorJson | ConvertTo-Json | Out-File $buildFlavorJsonFile -Encoding ascii
+$buildFlavorJson | ConvertTo-Json | Out-File $buildFlavorJsonFile -Encoding utf8
 
 #
 # Copy outputs to metadata directory

Build/scripts/init_build.ps1

@@ -12,23 +12,47 @@
 # before running the Pre-Build script.
 
 param (
+    [ValidateSet("x86", "x64", "arm", "")]
+    [string]$arch = "",
+    [ValidateSet("debug", "release", "test", "codecoverage", "")]
+    [string]$flavor = "",
+    [ValidateSet("default", "codecoverage", "pogo")]
+    [string]$subtype = "default",
+    [string]$buildtype,
+
     [string]$envConfigScript = "ComputedEnvironment.cmd",
 
     [string[]]$supportedPogoBuildTypes = @("x64_release", "x86_release"),
 
-    [Parameter(Mandatory=$True)]
+    [string]$verMajor = "",
+    [string]$verMinor = "",
+    [string]$verPatch = "",
+    [string]$verSecurity = "",
+
+    [string]$dropRoot,
+
+    [switch]$cleanBinDir,
+
     [string]$oauth
 )
 
-# If $Env:BuildType is specified, extract BuildPlatform and BuildConfiguration
-# Otherwise, if $Env:BuildPlatform and $Env:BuildConfiguration are specified, construct $BuildType
-$BuildPlatform = $Env:BuildPlatform
-$BuildConfiguration = $Env:BuildConfiguration
-$BuildType = $Env:BuildType
-$BuildSubtype = "default" # will remain as "default" or become e.g. "pogo", "codecoverage"
+#
+# Define values for variables based on parameters and environment variables
+# with default values in case the environment variables are not defined.
+#
+
+. $PSScriptRoot\util.ps1
+$gitExe = GetGitPath
+
+$BuildType = UseValueOrDefault $buildtype $Env:BuildType
+$BuildPlatform = UseValueOrDefault $arch $Env:BuildPlatform
+$BuildConfiguration = UseValueOrDefault $flavor $Env:BuildConfiguration
+$BuildSubtype = UseValueOrDefault $subtype $Env:BuildSubtype
 
-if (Test-Path Env:\BuildType) {
-    $BuildType = $Env:BuildType
+# If $BuildType is specified, extract BuildPlatform and BuildConfiguration
+# Otherwise, if $BuildPlatform and $BuildConfiguration are specified, construct $BuildType
+# $BuildSubtype will remain as "default" if not already specified, or become e.g. "pogo", "codecoverage"
+if ($BuildType) {
     $buildTypeSegments = $BuildType.split("_")
     $BuildPlatform = $buildTypeSegments[0]
     $BuildConfiguration = $buildTypeSegments[1]
@@ -42,12 +66,10 @@ if (Test-Path Env:\BuildType) {
         $BuildSubtype = "codecoverage" # keep information about codecoverage in the subtype
     }
 
-    if (-not ($BuildSubtype -in @("default","pogo","codecoverage"))) {
+    if (-not ($BuildSubtype -in @("default", "pogo", "codecoverage"))) {
         Write-Error "Unsupported BuildSubtype: $BuildSubtype"
     }
-} elseif ((Test-Path Env:\BuildPlatform) -and (Test-Path Env:\BuildConfiguration)) {
-    $BuildPlatform = $Env:BuildPlatform
-    $BuildConfiguration = $Env:BuildConfiguration
+} elseif ($BuildPlatform -and $BuildConfiguration) {
     $BuildType = "${BuildPlatform}_${BuildConfiguration}"
 } else {
     Write-Error (@"
@@ -56,67 +78,92 @@ if (Test-Path Env:\BuildType) {
         BuildType={0}
         BuildPlatform={1}
         BuildConfiguration={2}
+        BuildSubtype={3}
 
-"@ -f $Env:BuildType, $Env:BuildPlatform, $Env:BuildConfiguration)
+"@ -f $BuildType, $BuildPlatform, $BuildConfiguration, $BuildSubtype)
 
     exit 1
 }
 
+$CommitHash = UseValueOrDefault $Env:BUILD_SOURCEVERSION $(iex "${gitExe} rev-parse HEAD")
+
+$branchFullName  = UseValueOrDefault $Env:BUILD_SOURCEBRANCH $(iex "${gitExe} rev-parse --symbolic-full-name HEAD")
+
+$SourcesDirectory = UseValueOrDefault $Env:BUILD_SOURCESDIRECTORY $(GetRepoRoot)
+$BinariesDirectory = UseValueOrDefault (Join-Path $SourcesDirectory "Build\VcBuild")
+$ObjectDirectory = Join-Path $BinariesDirectory "obj\${BuildPlatform}_${BuildConfiguration}"
+
+$DropRoot = UseValueOrDefault $dropRoot $Env:DROP_ROOT (Join-Path $(GetRepoRoot) "_DROP")
+
 # set up required variables and import pre_post_util.ps1
 $arch = $BuildPlatform
 $flavor = $BuildConfiguration
 $OuterScriptRoot = $PSScriptRoot # Used in pre_post_util.ps1
 . "$PSScriptRoot\pre_post_util.ps1"
 
-$gitExe = GetGitPath
-
 $BuildName = ConstructBuildName -arch $BuildPlatform -flavor $BuildConfiguration -subtype $BuildSubtype
 
-$branch = $Env:BUILD_SOURCEBRANCH
-if (-not $branch) {
-    $branch = iex "$gitExe rev-parse --symbolic-full-name HEAD"
-}
-
-$BranchName = $branch.split('/',3)[2]
+$BranchName = $branchFullName.split('/',3)[2]
 $BranchPath = $BranchName.replace('/','\')
-$CommitHash = $Env:BUILD_SOURCEVERSION
+
 if (-not $CommitHash) {
-    $CommitHash = iex "$gitExe rev-parse HEAD"
+    $CommitHash = iex "${gitExe} rev-parse HEAD"
 }
+$CommitShortHash = $(iex "${gitExe} rev-parse --short $CommitHash")
 
-$Username = (iex "$gitExe log $CommitHash -1 --pretty=%ae").split('@')[0]
-$CommitDateTime = [DateTime]$(iex "$gitExe log $CommitHash -1 --pretty=%aD")
+$Username = (iex "${gitExe} log $CommitHash -1 --pretty=%ae").split('@')[0]
+$CommitDateTime = [DateTime]$(iex "${gitExe} log $CommitHash -1 --pretty=%aD")
 $CommitTime = Get-Date $CommitDateTime -Format yyMMdd.HHmm
 
 #
 # Get Build Info
 #
 
-$info = GetBuildInfo $oauth $CommitHash
+$buildPushDate = $null
+$buildPushIdString = $null
 
-$BuildPushDate = [datetime]$info.push.date
-$PushDate = Get-Date $BuildPushDate -Format yyMMdd.HHmm
+if (-not $oauth)
+{
+    $buildPushIdPart1 = 65535
+    $buildPushIdPart2 = 65535
+    $buildPushIdString = "65535.65535"
+    $buildPushDate = [DateTime]$CommitDateTime
+}
+else
+{
+    $info = GetBuildInfo $oauth $CommitHash
+    $_, $buildPushIdPart1, $buildPushIdPart2, $buildPushIdString = GetBuildPushId $info
+    $buildPushDate = [DateTime]$info.push.date
+}
+
+$PushDate = Get-Date $buildPushDate -Format yyMMdd.HHmm
+
+$VersionMajor       = UseValueOrDefault $verMajor       $Env:VERSION_MAJOR  (GetVersionField "CHAKRA_CORE_MAJOR_VERSION")       "0"
+$VersionMinor       = UseValueOrDefault $verMinor       $Env:VERSION_MINOR  (GetVersionField "CHAKRA_CORE_MINOR_VERSION")       "0"
+$VersionPatch       = UseValueOrDefault $verPatch       $Env:VERSION_PATCH  (GetVersionField "CHAKRA_CORE_PATCH_VERSION")       "0"
+$VersionSecurity    = UseValueOrDefault $verSecurity    $Env:VERSION_QFE    (GetVersionField "CHAKRA_CORE_VERSION_RELEASE_QFE") "0"
 
-$buildPushId, $buildPushIdPart1, $buildPushIdPart2, $buildPushIdString = GetBuildPushId $info
+$VersionString = "${VersionMajor}.${VersionMinor}.${VersionPatch}" # Only use MAJOR.MINOR.PATCH to align with SemVer
 
-$VersionMajor = UseValueOrDefault "$Env:VERSION_MAJOR" "1"
-$VersionMinor = UseValueOrDefault "$Env:VERSION_MINOR" "2"
-$VersionPatch = UseValueOrDefault "$Env:VERSION_PATCH" "0"
-$VersionQFE   = UseValueOrDefault "$Env:VERSION_QFE"   "0"
+$buildVersionString = "{0}-{1}" -f $buildPushIdPart1.ToString("00000"), $buildPushIdPart2.ToString("00000")
+$PreviewVersionString = "${VersionString}-preview-${buildVersionString}"
 
-$VersionString = "${VersionMajor}.${VersionMinor}.${VersionPatch}.${VersionQFE}"
-$PreviewVersionString = "${VersionString}-preview"
+$ShortBranch = "commit"
+if ($BranchName -eq "master") {
+    $ShortBranch = "master"
+} elseif ($BranchName.StartsWith("release")) {
+    $ShortBranch = $BranchName.replace("release/","")
+}
 
 # unless it is a build branch, subdivide the output directory by month
 if ($BranchPath.StartsWith("build")) {
     $YearAndMonth = ""
 } else {
-    $YearAndMonth = (Get-Date $BuildPushDate -Format yyMM) + "\"
+    $YearAndMonth = (Get-Date $buildPushDate -Format yyMM) + "\"
 }
 
 $BuildIdentifier = "${buildPushIdString}_${PushDate}_${Username}_${CommitHash}"
 $ComputedDropPathSegment = "${BranchPath}\${YearAndMonth}${BuildIdentifier}"
-$BinariesDirectory = "${Env:BUILD_SOURCESDIRECTORY}\Build\VcBuild"
 $ObjectDirectory = "${BinariesDirectory}\obj\${BuildPlatform}_${BuildConfiguration}"
 
 # Create a sentinel file for each build flavor to track whether the build is complete.
@@ -128,36 +175,43 @@ This could mean that the build is in progress, or that it was unable to run to c
 The contents of this directory should not be relied on until the build completes.
 "@
 
-$DropPath = Join-Path $Env:DROP_ROOT $ComputedDropPathSegment
+$DropPath = Join-Path $DropRoot $ComputedDropPathSegment
 New-Item -ItemType Directory -Force -Path $DropPath
-New-Item -ItemType Directory -Force -Path (Join-Path $Env:BUILD_SOURCESDIRECTORY "test\logs")
+New-Item -ItemType Directory -Force -Path (Join-Path $SourcesDirectory "test\logs")
 New-Item -ItemType Directory -Force -Path (Join-Path $BinariesDirectory "buildlogs")
 New-Item -ItemType Directory -Force -Path (Join-Path $BinariesDirectory "logs")
 
 $FlavorBuildIncompleteFile = Join-Path $DropPath "${BuildType}.incomplete"
 
 if (-not (Test-Path $FlavorBuildIncompleteFile)) {
     ($buildIncompleteFileContentsString -f "Build of ${BuildType}") `
-        | Out-File $FlavorBuildIncompleteFile -Encoding Ascii
+        | Out-File $FlavorBuildIncompleteFile -Encoding utf8
 }
 
-$PogoConfig = $supportedPogoBuildTypes -contains "${Env:BuildPlatform}_${Env:BuildConfiguration}"
+$PogoConfig = $supportedPogoBuildTypes -contains "${BuildPlatform}_${BuildConfiguration}"
 
 # Write the $envConfigScript
 
 @"
 set BranchName=${BranchName}
+set ShortBranch=${ShortBranch}
 set BranchPath=${BranchPath}
 set YearAndMonth=${YearAndMonth}
 set BuildIdentifier=${BuildIdentifier}
 
-set buildPushIdString=${buildPushIdString}
+set VersionMajor=${VersionMajor}
+set VersionMinor=${VersionMinor}
+set VersionPatch=${VersionPatch}
+set VersionSecurity=${VersionSecurity}
+
+set BuildPushIdString=${buildPushIdString}
 set VersionString=${VersionString}
 set PreviewVersionString=${PreviewVersionString}
 set PushDate=${PushDate}
 set CommitTime=${CommitTime}
 set Username=${Username}
 set CommitHash=${CommitHash}
+set CommitShortHash=${CommitShortHash}
 
 set ComputedDropPathSegment=${ComputedDropPathSegment}
 set BinariesDirectory=${BinariesDirectory}
@@ -174,24 +228,30 @@ set FlavorBuildIncompleteFile=${FlavorBuildIncompleteFile}
 set PogoConfig=${PogoConfig}
 
 "@ `
-    | Out-File $envConfigScript -Encoding Ascii
+    | Out-File $envConfigScript -Encoding ASCII
 
 # Use the VSTS environment vars to construct a backwards-compatible VSO build environment
 # for the sake of reusing the pre-build and post-build scripts as they are.
 
 @"
-set TF_BUILD_SOURCEGETVERSION=LG:${branch}:${CommitHash}
+set TF_BUILD_SOURCEGETVERSION=LG:${branchFullName}:${CommitHash}
 set TF_BUILD_DROPLOCATION=${BinariesDirectory}
 
-set TF_BUILD_SOURCESDIRECTORY=${Env:BUILD_SOURCESDIRECTORY}
+set TF_BUILD_SOURCESDIRECTORY=${SourcesDirectory}
 set TF_BUILD_BUILDDIRECTORY=${ObjectDirectory}
 set TF_BUILD_BINARIESDIRECTORY=${BinariesDirectory}
 
+REM The following variables are only used for logging build metadata.
 set TF_BUILD_BUILDDEFINITIONNAME=${Env:BUILD_DEFINITIONNAME}
 set TF_BUILD_BUILDNUMBER=${Env:BUILD_BUILDNUMBER}
 set TF_BUILD_BUILDURI=${Env:BUILD_BUILDURI}
 "@ `
-    | Out-File $envConfigScript -Encoding Ascii -Append
+    | Out-File $envConfigScript -Encoding ASCII -Append
+
+# Print contents of $envConfigScript as a sanity check
+Write-Output ""
+Get-Content $envConfigScript | Write-Output
+Write-Output ""
 
 # Export VSO variables that can be consumed by other VSO tasks where the task
 # definition in VSO itself needs to know the value of the variable.
@@ -216,10 +276,10 @@ Write-Output "Setting VSO variable VSO_VersionString = ${VersionString}"
 Write-Output "##vso[task.setvariable variable=VSO_VersionString;]${VersionString}"
 
 #
-# Clean up files that might have been left behind from a previous build.
+# Optionally ($cleanBinDir): clean up files that might have been left behind from a previous build.
 #
 
-if ((Test-Path Env:\BUILD_BINARIESDIRECTORY) -and (Test-Path "$Env:BUILD_BINARIESDIRECTORY"))
+if ($BinariesDirectory -and (Test-Path "$BinariesDirectory") -and $cleanBinDir)
 {
-    Remove-Item -Verbose "${Env:BUILD_BINARIESDIRECTORY}\*" -Recurse
+    Remove-Item -Verbose "${BinariesDirectory}\*" -Recurse
 }