[class-parse] support Module AttributeInfo (#1097)

Fixes: #1096

Context: https://stackoverflow.com/questions/57358750/module-info-class-file-is-different-in-the-module-jar-file-and-compiled-module-i
Context: 678c4bd
Context: b274a67

JDK 9 adds support for [modules][0], which are (kinda sorta) like
.NET Assemblies: modules can depend upon other modules, export
packages, etc.

In particular:

> **exports and exports…to.** An exports module directive specifies
> one of the module’s packages whose `public` types (and their nested
> `public` and `protected` types) should be accessible to code in all
> other modules.

This allows an equivalent to the [C# `internal` access modifier][1]:
`public` types in a *non-`export`ed package* should be treated as
"internal", while `public` types in an `export`ed package are
"fully public".

Update `Xamarin.Android.Tools.Bytecode.dll` to extract the module-
related information, then update `ClassPath` so that it updates all
`public` types *outside* of the "exported" packages to have an
`//*/@annotated-visibility` attribute value of `module-info`.
(See also commit b274a67, which added `//*/@@annotated-visibility`.)
If there is *already* an `//*/@annotated-visibility` value, then we
*append* ` module-info` to the attribute value.

We use `//*/@annotated-visibility` because we are concerned about
introducing an ABI break into AndroidX-related bindings because of
type visibility changes.  If this isn't a concern, it should be
possible to use Metadata to remove those types:

	<attr path="//class[@annotated-visibility]"
	    name="visibility">kotlin-internal</attr>
	<attr path="//interface[@annotated-visibility]"
	    name="visibility">kotlin-internal</attr>

`class-parse` command-line parsing has been altered.  There is now a
"global `ClassPath`", which will be used to hold `.class` files
provided on the command-line.  `.jar` and `.jmod` files provided on
the command-line will be given their own `ClassPath` instances, and
`module-info.class`-based annotated-visibility fixups are specific to
each `ClassPath` instance.  Global files are processed together.
There is thus no way for `module-info.class` visibility changes from
`a.jar` to impact `b.jar`.  After visibilities are fixed up, we then
merge everything into the "global" `ClassPath` instance before
transforming to XML.

Additionally, `class-parse --dump` can now accept `.jar` files,
and will dump out *all* `.class` filers within the `.jar` file.
To make this output easier, each "entry" starts with a "header" of
`-- Begin {ClassFile.FullJniName}`, and a blank link will be printed
between each entry.

`tests/Xamarin.Android.Tools.Bytecode-Tests` has been updated to:

 1. Contain a `module-info.java`, which declares a `com.xamarin`
    module.

 2. Add a new `com.xamarin.internal.PublicClassNotInModuleExports`
    type which is *not* in the `com.xamarin` package, but instead
    a *nested* package.  The type is `public`.

 3. Build a `xatb.jar` artifact

This makes for a simple one-off test:

	% dotnet build tests/Xamarin.Android.Tools.Bytecode-Tests/*.csproj
	% dotnet build tools/class-parse/*.csproj
	% dotnet bin/Debug-net7.0/class-parse.dll \
	  tests/Xamarin.Android.Tools.Bytecode-Tests/obj/Debug-net7.0/xatb.jar
	…
	    <class
	      name="PublicClassNotInModuleExports"
	      …
	      annotated-visibility="module-info" />

Note that `com.xamarin.internal.PublicClassNotInModuleExports` now
has an XML attribute `annotated-visibility="module-info"`.

Aside: the commit message of 678c4bd sadly overlooked this
[clarification][2] for why `kotlin-internal` was introduced:

> Note: we introduce and use a new `//*/@visibility` value of
> `kotlin-internal` because `internal` is an *existing* value that may
> be used in `Metadata.xml` files, e.g. making `public` API `internal`
> so that it can still be used in the binding, but isn't *public*.

Aside: a discovered oddity: `jar cf …` *modifies* `module-info.class`,
adding a `ModulePackages` attribute!  (Specifically, if you compare
the "on-disk" `module-info.class` to the one within
`tests/Xamarin.Android.Tools.Bytecode-Tests/obj/$(Configuration)/xatb.jar`,
they differ in size!)

[0]: https://www.oracle.com/corporate/features/understanding-java-9-modules.html
[1]: https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/internal
[2]: #793 (comment)

Author: jonpryor

src/Xamarin.Android.Tools.Bytecode/AttributeInfo.cs

@@ -44,6 +44,8 @@ public class AttributeInfo {
 		public  const   string  InnerClasses            = "InnerClasses";
 		public  const   string  LocalVariableTable      = "LocalVariableTable";
 		public  const   string  MethodParameters        = "MethodParameters";
+		public  const   string  Module                  = "Module";
+		public  const   string  ModulePackages          = "ModulePackages";
 		public  const   string  Signature               = "Signature";
 		public  const   string  SourceFile              = "SourceFile";
 		public  const   string  StackMapTable           = "StackMapTable";
@@ -79,6 +81,8 @@ public string Name {
 			{ typeof (InnerClassesAttribute),           InnerClasses },
 			{ typeof (LocalVariableTableAttribute),     LocalVariableTable },
 			{ typeof (MethodParametersAttribute),       MethodParameters },
+			{ typeof (ModuleAttribute),                 Module },
+			{ typeof (ModulePackagesAttribute),         ModulePackages },
 			{ typeof (RuntimeVisibleAnnotationsAttribute),		RuntimeVisibleAnnotations },
 			{ typeof (RuntimeInvisibleAnnotationsAttribute),	RuntimeInvisibleAnnotations },
 			{ typeof (SignatureAttribute),              Signature },
@@ -98,6 +102,7 @@ internal static string GetAttributeName<T>()
 		public static AttributeInfo CreateFromStream (ConstantPool constantPool, Stream stream)
 		{
 			var nameIndex   = stream.ReadNetworkUInt16 ();
+			var constant = constantPool [nameIndex];
 			var name        = ((ConstantPoolUtf8Item) constantPool [nameIndex]).Value;
 			var attr        = CreateAttribute (name, constantPool, nameIndex, stream);
 			return attr;
@@ -114,6 +119,8 @@ static AttributeInfo CreateAttribute (string name, ConstantPool constantPool, us
 			case InnerClasses:          return new InnerClassesAttribute (constantPool, nameIndex, stream);
 			case LocalVariableTable:    return new LocalVariableTableAttribute (constantPool, nameIndex, stream);
 			case MethodParameters:      return new MethodParametersAttribute (constantPool, nameIndex, stream);
+			case Module:                return new ModuleAttribute (constantPool, nameIndex, stream);
+			case ModulePackages:        return new ModulePackagesAttribute (constantPool, nameIndex, stream);
 			case RuntimeVisibleAnnotations:		return new RuntimeVisibleAnnotationsAttribute (constantPool, nameIndex, stream);
 			case RuntimeInvisibleAnnotations:	return new RuntimeInvisibleAnnotationsAttribute (constantPool, nameIndex, stream);
 			case RuntimeInvisibleParameterAnnotations:	return new RuntimeInvisibleParameterAnnotationsAttribute (constantPool, nameIndex, stream);
@@ -503,6 +510,118 @@ public override string ToString ()
 		}
 	}
 
+	// https://docs.oracle.com/javase/specs/jvms/se11/html/jvms-4.html#jvms-4.7.25
+	public sealed class ModuleAttribute : AttributeInfo
+	{
+		ushort	moduleNameIndex;
+		public  string          ModuleName {
+			get {return ((ConstantPoolModuleItem) ConstantPool [moduleNameIndex]).Name.Value;}
+		}
+
+		public  ModuleFlags     ModuleFlags { get; private set; }
+
+		ushort  moduleVersionIndex;
+		public string? ModuleVersion {
+			get {return ((ConstantPoolUtf8Item) ConstantPool [moduleVersionIndex])?.Value;}
+		}
+
+		public  Collection<ModuleRequiresInfo>          Requires {get;} = new ();
+		public  Collection<ModuleExportsPackageInfo>    Exports {get;}  = new ();
+		public  Collection<ModuleOpensPackageInfo>      Opens {get;}    = new ();
+		public  Collection<ConstantPoolClassItem>       Uses {get;}     = new ();
+		public  Collection<ModuleProvidesInfo>          Provides {get;} = new ();
+
+		public ModuleAttribute (ConstantPool constantPool, ushort nameIndex, Stream stream)
+			: base (constantPool, nameIndex, stream)
+		{
+			var attribute_length        = stream.ReadNetworkUInt32 ();
+
+			moduleNameIndex             = stream.ReadNetworkUInt16 ();
+			ModuleFlags                 = (ModuleFlags) stream.ReadNetworkUInt16 ();
+			moduleVersionIndex          = stream.ReadNetworkUInt16 ();
+
+			var requires_count          = stream.ReadNetworkUInt16 ();
+			for (int i = 0; i < requires_count; ++i) {
+				Requires.Add (new ModuleRequiresInfo (constantPool, stream));
+			}
+
+			var exports_count           = stream.ReadNetworkUInt16 ();
+			for (int i = 0; i < exports_count; ++i) {
+				Exports.Add (new ModuleExportsPackageInfo (constantPool, stream));
+			}
+
+			var opens_count             = stream.ReadNetworkUInt16 ();
+			for (int i = 0; i < opens_count; ++i) {
+				Opens.Add (new ModuleOpensPackageInfo (constantPool, stream));
+			}
+
+			var uses_count              = stream.ReadNetworkUInt16 ();
+			for (int i = 0; i < uses_count; ++i) {
+				var uses_index = stream.ReadNetworkUInt16 ();
+				Uses.Add ((ConstantPoolClassItem) constantPool [uses_index]);
+			}
+
+			var provides_count          = stream.ReadNetworkUInt16 ();
+			for (int i = 0; i < provides_count; ++i) {
+				Provides.Add (new ModuleProvidesInfo (constantPool, stream));
+			}
+		}
+
+		public override string ToString ()
+		{
+			var s = new StringBuilder ()
+				.Append ("Module(").AppendLine ()
+				.Append ("  ").Append (nameof (ModuleName)).Append ("='").Append (ModuleName).AppendLine ("', ")
+				.Append ("  ").Append (nameof (ModuleVersion)).Append ("='").Append (ModuleVersion).Append ("'");
+			AppendString (s, nameof (Requires), Requires);
+			AppendString (s, nameof (Exports), Exports);
+			AppendString (s, nameof (Opens), Opens);
+			AppendString (s, nameof (Uses), Uses.Select (u => $"UsesService({u.Name})").ToList ());
+			AppendString (s, nameof (Provides), Provides);
+			s.Append (")");
+
+			return s.ToString ();
+		}
+
+		static StringBuilder AppendString<T> (StringBuilder s, string collectionName, IList<T> items)
+		{
+			if (items.Count == 0) {
+				return s;
+			}
+			s.AppendLine (",");
+			s.Append ("  ").Append (collectionName).AppendLine ("={");
+			s.Append ("    ").Append (items [0]);
+			for (int i = 1; i < items.Count; ++i) {
+				s.AppendLine (",");
+				s.Append ("    ");
+				s.Append (items [i]);
+			}
+			return s.Append ("}");
+		}
+	}
+
+	// https://docs.oracle.com/javase/specs/jvms/se11/html/jvms-4.html#jvms-4.7.26
+	public sealed class ModulePackagesAttribute : AttributeInfo {
+		public  Collection<ConstantPoolPackageItem>     Packages    {get;} = new ();
+
+		public ModulePackagesAttribute (ConstantPool constantPool, ushort nameIndex, Stream stream)
+			: base (constantPool, nameIndex, stream)
+		{
+			var attribute_length    = stream.ReadNetworkUInt32 ();
+
+			var package_count       = stream.ReadNetworkUInt16 ();
+			for (int i = 0; i < package_count; ++i) {
+				var package_index = stream.ReadNetworkUInt16 ();
+				Packages.Add ((ConstantPoolPackageItem) constantPool [package_index]);
+			}
+		}
+
+		public override string ToString ()
+		{
+			return $"ModulePackages({{{string.Join (", ", Packages.Select (p => p.Name.Value))}}})";
+		}
+	}
+
 	// https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.7.16
 	public sealed class RuntimeVisibleAnnotationsAttribute : AttributeInfo
 	{

src/Xamarin.Android.Tools.Bytecode/ClassFile.cs

@@ -48,8 +48,15 @@ public ClassFile (Stream stream)
 			Attributes      = new AttributeCollection (ConstantPool, stream);
 
 			int e = stream.ReadByte ();
-			if (e >= 0)
-				throw new BadImageFormatException ("Stream has trailing data?!");
+			if (e >= 0) {
+				var trailing    = new System.Text.StringBuilder ();
+				trailing.AppendFormat ("{0:x2}", (byte) e);
+				while ((e = stream.ReadByte ()) >= 0) {
+					trailing.Append (" ");
+					trailing.AppendFormat ("{0:x2}", (byte) e);
+				}
+				throw new BadImageFormatException ($"Stream has trailing data?! {{{trailing}}}");
+			}
 		}
 
 		public static bool IsClassFile (Stream stream)
@@ -213,6 +220,10 @@ public enum ClassAccessFlags {
 		Synthetic   = 0x1000,
 		Annotation  = 0x2000,
 		Enum        = 0x4000,
+		Module      = 0x8000,
+
+		// This is not a real Java ClassAccessFlags, it is used to denote non-exported public types
+		Internal    = 0x10000000,
 	}
 }
 

src/Xamarin.Android.Tools.Bytecode/ClassPath.cs

@@ -88,9 +88,6 @@ static bool ShouldLoadEntry (ZipArchiveEntry entry)
 			if (entry.Length == 0)
 				return false;
 
-			if (entry.Name == "module-info.class")
-				return false;
-
 			if (entry.Name.EndsWith (".jnilib", StringComparison.OrdinalIgnoreCase))
 				return false;
 
@@ -111,6 +108,14 @@ public void Add (ClassFile classFile)
 			classFiles.Add (classFile);
 		}
 
+		public void Add (ClassPath classPath, bool removeModules = true)
+		{
+			classPath.FixupModuleVisibility (removeModules);
+			foreach (var c in classPath.classFiles) {
+				Add (c);
+			}
+		}
+
 		public ReadOnlyDictionary<string, List<ClassFile>> GetPackages ()
 		{
 			return new ReadOnlyDictionary<string, List<ClassFile>> (classFiles
@@ -360,6 +365,7 @@ public XElement ToXElement ()
 				FixUpParametersFromClasses ();
 
 			KotlinFixups.Fixup (classFiles);
+			FixupModuleVisibility (removeModules: true);
 
 			var packagesDictionary = GetPackages ();
 			var api = new XElement ("api",
@@ -375,6 +381,42 @@ packagesDictionary [p].OrderBy (c => c.ThisClass.Name.Value, StringComparer.Ordi
 			return api;
 		}
 
+		public void FixupModuleVisibility (bool removeModules)
+		{
+			var publicPackages  = new HashSet<string> ();
+
+			var moduleFiles     = classFiles.Where (c => c.AccessFlags == ClassAccessFlags.Module)
+				.ToList ();
+			if (moduleFiles.Count == 0) {
+				return;
+			}
+			foreach (var moduleFile in moduleFiles) {
+				if (removeModules) {
+					classFiles.Remove (moduleFile);
+				}
+				foreach (var moduleAttr in moduleFile.Attributes.OfType<ModuleAttribute> ()) {
+					foreach (var export in moduleAttr.Exports) {
+						publicPackages.Add (export.Exports);
+					}
+				}
+			}
+
+			foreach (var c in classFiles) {
+				if (!c.AccessFlags.HasFlag (ClassAccessFlags.Public)) {
+					continue;
+				}
+				var jniName     = c.ThisClass.Name.Value;
+				var packageEnd  = jniName.LastIndexOf ('/');
+				if (packageEnd < 0) {
+					continue;
+				}
+				var package     = jniName.Substring (0, packageEnd);
+				if (!publicPackages.Contains (package)) {
+					c.AccessFlags = KotlinFixups.SetVisibility (c.AccessFlags, ClassAccessFlags.Internal);
+				}
+			}
+		}
+
 		public void SaveXmlDescription (string fileName)
 		{
 			var encoding    = new UTF8Encoding (encoderShouldEmitUTF8Identifier: false);
@@ -395,5 +437,7 @@ public void SaveXmlDescription (TextWriter textWriter)
 				contents.Save (writer);
 			textWriter.WriteLine ();
 		}
+
+		public IEnumerable<ClassFile> GetClassFiles () => classFiles;
 	}
 }

src/Xamarin.Android.Tools.Bytecode/Kotlin/KotlinFixups.cs

@@ -102,7 +102,7 @@ static void FixupClassVisibility (ClassFile klass, KotlinClass metadata)
 		}
 
 		// Passing null for 'newVisibility' parameter means 'package-private'
-		static ClassAccessFlags SetVisibility (ClassAccessFlags existing, ClassAccessFlags? newVisibility)
+		internal static ClassAccessFlags SetVisibility (ClassAccessFlags existing, ClassAccessFlags? newVisibility)
 		{
 			// First we need to remove any existing visibility flags,
 			// without modifying other flags like Abstract