Skip to content

Commit 7574f16

Browse files
jonpryorjonpryor
authored
[generator] Add generator --with-javadoc-xml=FILE support. (#687)
Fixes: #642 Context: dotnet/android#4789 Context: dotnet/android#5200 Commit 69e1b80 added `tools/java-source-utils`, which along with b588ef5, can parse Java source code and extract Javadoc comments from Android API-30 sources into an XML file: into an XML file: # API-30 `sources` package contains both `Object.java` and `Object.annotated.java`; # Skip the `.annotated.java` files $ find $HOME/android-toolchain/sdk/platforms/android-30/src/{android,java,javax,org} -iname \*.java \ | grep -v '\.annotated\.' > sources.txt $ java -jar java-source-utils.jar -v \ --source "$HOME/android-toolchain/sdk/platforms/android-30/src" \ --output-javadoc android-javadoc.xml \ @sources.txt What can we *do* with the generated `android-javadoc.xml`? `android-javadoc.xml` contains parameter names, and thus can be used with `class-parse --docspath`; see commit 806082f. What we *really* want to do is make the Javadoc information *useful* to consumers of the binding assembly. This means that we want a C# XML Documentation file for the binding assembly. The most straightforward way to get a C# XML Documentation File is to emit [C# XML Documentation Comments][0] into the binding source code! Add a new `generator --with-javadoc-xml=FILE` option. When specified, `FILE` will be treated as an XML file containing the output from `java-source-utils.jar --output-javadoc` (69e1b80), and all `<javadoc/>` elements within the XML file will be associated with C# types and members to emit, based on the `//@jni-signature` and `//@name` attributes, as appropriate. When the bindings are written to disk, the Javadoc comments will be translated to C# XML Documentation comments, in a "best effort" basis. (THIS WILL BE INCOMPLETE.) To perform the Javadoc-to-C# XML Documentation comments conversion, add a new Irony-based grammar to `Java.Interop.Tools.JavaSource.dll`, in the new `Java.Interop.Tools.JavaSource.SourceJavadocToXmldocParser` type, which parses the Javadoc content and translates to XML. In addition to transforming the Javadoc comments into C# XML documentation comments, we *also* want to provide "upstream information" in the form of: 1. A URL to the corresponding online Javadoc HTML documentation. 2. A copyright notice disclaimer. Allow provision of this information by updating `java-source-utils.jar` to support the new options: --doc-copyright FILE Copyright information for Javadoc. Should be in mdoc(5) XML, to be held within <remarks/>. Stored in //javadoc-metadata/copyright. --doc-url-prefix URL Base URL for links to documentation. Stored in //javadoc-metadata/link/@Prefix. --doc-url-style STYLE STYLE of URLs to generate for member links. Stored in //javadoc-metadata/link/@Style. Supported styles include: - developer.android.com/reference@2020-Nov The new `/api/javadoc-metadata/link@prefix` and `/api/javadoc-metadata/link@style` XML attributes, stored within `java-source-utils.jar --output-javadoc` XML output, allow construction of a URL to the Java member. For example, given: java -jar java-source-utils.jar \ --doc-url-prefix https://developer.android.com/reference \ --doc-url-style developer.android.com/reference@2020-Nov Then `generator` can emit the C# documentation comment for `java.lang.Object.equals(Object)`: /// <format type="text/html"> /// <a href="https://developer.android.com/reference/java/lang/Object#equals(java.lang.Object)">Java documentation for <tt>java.lang.Object.equals(java.lang.Object)</tt>.</a> /// </format> The copyright notice disclaimer is supported by `java-source-utils.jar --doc-copyright FILE`; the contents of `FILE` are inserted into the `/api/javadoc-metadata/copyright` element, and will be copied into the output of every C# XML documentation block. Example output is at: * <https://gist.github.com/jonpryor/004f01f4cd5ff32299ff590ba7a2fe0e> Unfortunately, converting Javadoc to C# XML Documentation Comments is not a "zero cost" operation. Add a new `generator --doc-comment-verbosity=STYLE` option to control how "complete" the generated documentation comments are: --doc-comment-verbosity=STYLE STYLE of C# documentation comments to emit. Defaults to `full`. STYLE may be: * `intellisense`: emit <summary>, <param>, <returns>, <exception>. * `full`: plus <remarks>, <altmember>, ... Using `--doc-comment-verbosity=full` will *most* impact build times. TODO: * `SourceJavadocToXmldocParser` doesn't support many constructs. [0]: https://docs.microsoft.com/en-us/dotnet/csharp/codedoc
1 parent 7d197f1 commit 7574f16

37 files changed

+2354
-22
lines changed

‎src/Java.Interop.Tools.JavaSource/Java.Interop.Tools.JavaSource/IronyExtensions.cs‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
using System;
2+
using System.Collections.Generic;
3+
using System.IO;
4+
using System.Linq;
5+
using System.Text;
6+
using Irony.Ast;
7+
using Irony.Parsing;
8+
9+
namespace Java.Interop.Tools.JavaSource {
10+
11+
static class IronyExtensions {
12+
13+
public static void MakePlusRule (this NonTerminal star, Grammar grammar, BnfTerm delimiter)
14+
{
15+
star.Rule = grammar.MakePlusRule (star, delimiter);
16+
}
17+
18+
public static void MakeStarRule (this NonTerminal star, Grammar grammar, BnfTerm delimiter, BnfTerm of)
19+
{
20+
star.Rule = grammar.MakeStarRule (star, delimiter, of);
21+
}
22+
23+
public static void MakeStarRule (this NonTerminal star, Grammar grammar, BnfTerm of)
24+
{
25+
star.Rule = grammar.MakeStarRule (star, of);
26+
}
27+
}
28+
}

‎src/Java.Interop.Tools.JavaSource/Java.Interop.Tools.JavaSource/JavadocInfo.cs‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
using System;
2+
using System.Collections.Generic;
3+
using System.Collections.ObjectModel;
4+
using System.IO;
5+
using System.Linq;
6+
using System.Xml.Linq;
7+
8+
using Irony.Ast;
9+
using Irony.Parsing;
10+
11+
namespace Java.Interop.Tools.JavaSource {
12+
13+
sealed class JavadocInfo {
14+
public readonly ICollection<XNode> Exceptions = new Collection<XNode> ();
15+
public readonly ICollection<XNode> Extra = new Collection<XNode> ();
16+
public readonly ICollection<XNode> Remarks = new Collection<XNode> ();
17+
public readonly ICollection<XNode> Parameters = new Collection<XNode> ();
18+
public readonly ICollection<XNode> Returns = new Collection<XNode> ();
19+
20+
public override string ToString ()
21+
{
22+
return new XElement ("Javadoc",
23+
new XElement (nameof (Parameters), Parameters),
24+
new XElement (nameof (Remarks), Remarks),
25+
new XElement (nameof (Returns), Returns),
26+
new XElement (nameof (Exceptions), Exceptions),
27+
new XElement (nameof (Extra), Extra))
28+
.ToString ();
29+
}
30+
}
31+
}

‎src/Java.Interop.Tools.JavaSource/Java.Interop.Tools.JavaSource/SourceJavadocToXmldocGrammar.BlockTagsBnfTerms.cs‎

Lines changed: 239 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,239 @@
1+
using System;
2+
using System.Collections.Generic;
3+
using System.IO;
4+
using System.Linq;
5+
using System.Xml.Linq;
6+
7+
using Irony.Ast;
8+
using Irony.Parsing;
9+
10+
namespace Java.Interop.Tools.JavaSource {
11+
12+
public partial class SourceJavadocToXmldocGrammar {
13+
14+
// https://docs.oracle.com/javase/7/docs/technotes/tools/windows/javadoc.html#javadoctags
15+
public class BlockTagsBnfTerms {
16+
17+
internal BlockTagsBnfTerms ()
18+
{
19+
}
20+
21+
internal void CreateRules (SourceJavadocToXmldocGrammar grammar)
22+
{
23+
AllBlockTerms.Rule = AuthorDeclaration
24+
| ApiSinceDeclaration
25+
| DeprecatedDeclaration
26+
| DeprecatedSinceDeclaration
27+
| ExceptionDeclaration
28+
| ParamDeclaration
29+
| ReturnDeclaration
30+
| SeeDeclaration
31+
| SerialDataDeclaration
32+
| SerialFieldDeclaration
33+
| SinceDeclaration
34+
| ThrowsDeclaration
35+
| UnknownTagDeclaration
36+
| VersionDeclaration
37+
;
38+
BlockValue.Rule = grammar.HtmlTerms.ParsedCharacterData
39+
| grammar.HtmlTerms.InlineDeclaration
40+
;
41+
BlockValues.MakePlusRule (grammar, BlockValue);
42+
43+
AuthorDeclaration.Rule = "@author" + BlockValues;
44+
AuthorDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
45+
if (!grammar.ShouldImport (ImportJavadoc.AuthorTag))
46+
return;
47+
// Ignore; not sure how best to convert to Xmldoc
48+
FinishParse (context, parseNode);
49+
};
50+
51+
ApiSinceDeclaration.Rule = "@apiSince" + BlockValues;
52+
ApiSinceDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
53+
if (!grammar.ShouldImport (ImportJavadoc.SinceTag)) {
54+
return;
55+
}
56+
var p = new XElement ("para", "Added in API level ", AstNodeToXmlContent (parseNode.ChildNodes [1]), ".");
57+
FinishParse (context, parseNode).Remarks.Add (p);
58+
parseNode.AstNode = p;
59+
};
60+
61+
DeprecatedDeclaration.Rule = "@deprecated" + BlockValues;
62+
DeprecatedDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
63+
if (!grammar.ShouldImport (ImportJavadoc.DeprecatedTag)) {
64+
return;
65+
}
66+
var p = new XElement ("para", "This member is deprecated. ", AstNodeToXmlContent (parseNode.ChildNodes [1]));
67+
FinishParse (context, parseNode).Remarks.Add (p);
68+
parseNode.AstNode = p;
69+
};
70+
71+
DeprecatedSinceDeclaration.Rule = "@deprecatedSince" + BlockValues;
72+
DeprecatedSinceDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
73+
if (!grammar.ShouldImport (ImportJavadoc.DeprecatedTag)) {
74+
return;
75+
}
76+
var p = new XElement ("para", "This member was deprecated in API level ", AstNodeToXmlContent (parseNode.ChildNodes [1]), ".");
77+
FinishParse (context, parseNode).Remarks.Add (p);
78+
parseNode.AstNode = p;
79+
};
80+
81+
var nonSpaceTerm = new RegexBasedTerminal ("[^ ]", "[^ ]+") {
82+
AstConfig = new AstNodeConfig {
83+
NodeCreator = (context, parseNode) => parseNode.AstNode = parseNode.Token.Value,
84+
},
85+
};
86+
87+
ExceptionDeclaration.Rule = "@exception" + nonSpaceTerm + BlockValues;
88+
ExceptionDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
89+
if (!grammar.ShouldImport (ImportJavadoc.ExceptionTag)) {
90+
return;
91+
}
92+
// TODO: convert `nonSpaceTerm` into a proper CREF
93+
var e = new XElement ("exception",
94+
new XAttribute ("cref", string.Join ("", AstNodeToXmlContent (parseNode.ChildNodes [1]))),
95+
AstNodeToXmlContent (parseNode.ChildNodes [2]));
96+
FinishParse (context, parseNode).Exceptions.Add (e);
97+
parseNode.AstNode = e;
98+
};
99+
100+
ParamDeclaration.Rule = "@param" + nonSpaceTerm + BlockValues;
101+
ParamDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
102+
if (!grammar.ShouldImport (ImportJavadoc.ParamTag)) {
103+
return;
104+
}
105+
var p = new XElement ("param",
106+
new XAttribute ("name", string.Join ("", AstNodeToXmlContent (parseNode.ChildNodes [1]))),
107+
AstNodeToXmlContent (parseNode.ChildNodes [2]));
108+
FinishParse (context, parseNode).Parameters.Add (p);
109+
parseNode.AstNode = p;
110+
};
111+
112+
ReturnDeclaration.Rule = "@return" + BlockValues;
113+
ReturnDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
114+
if (!grammar.ShouldImport (ImportJavadoc.ReturnTag)) {
115+
return;
116+
}
117+
var r = new XElement ("returns",
118+
AstNodeToXmlContent (parseNode.ChildNodes [1]));
119+
FinishParse (context, parseNode).Returns.Add (r);
120+
parseNode.AstNode = r;
121+
};
122+
123+
SeeDeclaration.Rule = "@see" + BlockValues;
124+
SeeDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
125+
if (!grammar.ShouldImport (ImportJavadoc.SeeTag)) {
126+
return;
127+
}
128+
// TODO: @see supports multiple forms; see: https://docs.oracle.com/javase/7/docs/technotes/tools/windows/javadoc.html#see
129+
var e = new XElement ("seealso",
130+
new XAttribute ("cref", string.Join ("", AstNodeToXmlContent (parseNode.ChildNodes [1]))));
131+
FinishParse (context, parseNode).Extra.Add (e);
132+
parseNode.AstNode = e;
133+
};
134+
135+
SinceDeclaration.Rule = "@since" + BlockValues;
136+
SinceDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
137+
if (!grammar.ShouldImport (ImportJavadoc.SinceTag)) {
138+
return;
139+
}
140+
var p = new XElement ("para", "Added in ", AstNodeToXmlContent (parseNode.ChildNodes [1]), ".");
141+
FinishParse (context, parseNode).Remarks.Add (p);
142+
parseNode.AstNode = p;
143+
};
144+
145+
ThrowsDeclaration.Rule = "@throws" + nonSpaceTerm + BlockValues;
146+
ThrowsDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
147+
if (!grammar.ShouldImport (ImportJavadoc.ExceptionTag)) {
148+
return;
149+
}
150+
// TODO: convert `nonSpaceTerm` into a proper CREF
151+
var e = new XElement ("exception",
152+
new XAttribute ("cref", string.Join ("", AstNodeToXmlContent (parseNode.ChildNodes [1]))),
153+
AstNodeToXmlContent (parseNode.ChildNodes [2]));
154+
FinishParse (context, parseNode).Exceptions.Add (e);
155+
parseNode.AstNode = e;
156+
};
157+
158+
// Ignore serialization informatino
159+
SerialDeclaration.Rule = "@serial" + BlockValues;
160+
SerialDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
161+
if (!grammar.ShouldImport (ImportJavadoc.SerialTag)) {
162+
return;
163+
}
164+
FinishParse (context, parseNode);
165+
};
166+
167+
SerialDataDeclaration.Rule = "@serialData" + BlockValues;
168+
SerialDataDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
169+
if (!grammar.ShouldImport (ImportJavadoc.SerialTag)) {
170+
return;
171+
}
172+
FinishParse (context, parseNode);
173+
};
174+
175+
SerialFieldDeclaration.Rule = "@serialField" + BlockValues;
176+
SerialFieldDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
177+
if (!grammar.ShouldImport (ImportJavadoc.SerialTag)) {
178+
return;
179+
}
180+
FinishParse (context, parseNode);
181+
};
182+
183+
var unknownTagTerminal = new RegexBasedTerminal ("@[unknown]", @"@\S+") {
184+
Priority = TerminalPriority.Low,
185+
};
186+
unknownTagTerminal.AstConfig.NodeCreator = (context, parseNode) =>
187+
parseNode.AstNode = parseNode.Token.Value.ToString ();
188+
189+
190+
UnknownTagDeclaration.Rule = unknownTagTerminal + BlockValues;
191+
UnknownTagDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
192+
if (!grammar.ShouldImport (ImportJavadoc.Remarks)) {
193+
return;
194+
}
195+
Console.WriteLine ($"# Unsupported @block-tag value: {parseNode.ChildNodes [0].AstNode}");
196+
FinishParse (context, parseNode);
197+
};
198+
199+
// Ignore Version
200+
VersionDeclaration.Rule = "@version" + BlockValues;
201+
VersionDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
202+
if (!grammar.ShouldImport (ImportJavadoc.VersionTag)) {
203+
return;
204+
}
205+
FinishParse (context, parseNode);
206+
};
207+
}
208+
209+
public readonly NonTerminal AllBlockTerms = new NonTerminal (nameof (AllBlockTerms), ConcatChildNodes);
210+
211+
public readonly Terminal Cdata = new CharacterDataTerminal ("#CDATA", preserveLeadingWhitespace: true);
212+
/*
213+
public readonly Terminal Cdata = new RegexBasedTerminal (nameof (BlockValue), "[^<]*") {
214+
AstConfig = new AstNodeConfig {
215+
NodeCreator = (context, parseNode) => parseNode.AstNode = parseNode.Token.Value.ToString (),
216+
},
217+
};
218+
*/
219+
220+
public readonly NonTerminal BlockValue = new NonTerminal (nameof (BlockValue), ConcatChildNodes);
221+
public readonly NonTerminal BlockValues = new NonTerminal (nameof (BlockValues), ConcatChildNodes);
222+
public readonly NonTerminal AuthorDeclaration = new NonTerminal (nameof (AuthorDeclaration));
223+
public readonly NonTerminal ApiSinceDeclaration = new NonTerminal (nameof (ApiSinceDeclaration));
224+
public readonly NonTerminal DeprecatedDeclaration = new NonTerminal (nameof (DeprecatedDeclaration));
225+
public readonly NonTerminal DeprecatedSinceDeclaration = new NonTerminal (nameof (DeprecatedSinceDeclaration));
226+
public readonly NonTerminal ExceptionDeclaration = new NonTerminal (nameof (ExceptionDeclaration));
227+
public readonly NonTerminal ParamDeclaration = new NonTerminal (nameof (ParamDeclaration));
228+
public readonly NonTerminal ReturnDeclaration = new NonTerminal (nameof (ReturnDeclaration));
229+
public readonly NonTerminal SeeDeclaration = new NonTerminal (nameof (SeeDeclaration));
230+
public readonly NonTerminal SerialDeclaration = new NonTerminal (nameof (SerialDeclaration));
231+
public readonly NonTerminal SerialDataDeclaration = new NonTerminal (nameof (SerialDataDeclaration));
232+
public readonly NonTerminal SerialFieldDeclaration = new NonTerminal (nameof (SerialFieldDeclaration));
233+
public readonly NonTerminal SinceDeclaration = new NonTerminal (nameof (SinceDeclaration));
234+
public readonly NonTerminal ThrowsDeclaration = new NonTerminal (nameof (ThrowsDeclaration));
235+
public readonly NonTerminal UnknownTagDeclaration = new NonTerminal (nameof (UnknownTagDeclaration));
236+
public readonly NonTerminal VersionDeclaration = new NonTerminal (nameof (VersionDeclaration));
237+
}
238+
}
239+
}

0 commit comments

Comments
 (0)