Skip to content
Merged
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
b9d4a04
chore: removes old proof of concept
chalmerlowe Sep 11, 2025
5b4d538
removes old __init__.py
chalmerlowe Sep 11, 2025
132c571
Adds two utility files to handle basic tasks
chalmerlowe Sep 11, 2025
90b224e
Adds a configuration file for the microgenerator
chalmerlowe Sep 11, 2025
e071eab
Removes unused comment
chalmerlowe Sep 11, 2025
dc72a98
chore: adds noxfile.py for the microgenerator
chalmerlowe Sep 11, 2025
7318f0b
feat: microgen - adds two init file templates
chalmerlowe Sep 12, 2025
07910c5
feat: adds _helpers.py.js template
chalmerlowe Sep 12, 2025
dc54c99
Updates with two usage examples
chalmerlowe Sep 12, 2025
28de5f8
feat: adds two partial templates for creating method signatures
chalmerlowe Sep 12, 2025
c457754
feat: Add microgenerator __init__.py
chalmerlowe Sep 15, 2025
595e59f
feat: Add AST analysis utilities
chalmerlowe Sep 15, 2025
44a0777
feat: Add source file analysis capabilities
chalmerlowe Sep 15, 2025
3e9ade6
feat: adds code generation logic
chalmerlowe Sep 15, 2025
485b9d4
removes extraneous content
chalmerlowe Sep 15, 2025
a4276fe
feat: microgen - adds code generation logic
chalmerlowe Sep 15, 2025
1d0d036
feat: microgen - adds main execution and post-processing logic
chalmerlowe Sep 15, 2025
eff7223
minor tweak to markers
chalmerlowe Sep 15, 2025
0734bf8
feat: Add testing directory\n\nAdds the scripts/microgenerator/testin…
chalmerlowe Sep 16, 2025
510a87b
feat: Enhance to_snake_case to handle acronyms\n\nImproves the to_sna…
chalmerlowe Sep 16, 2025
a3117d8
feat: Add client.py.j2 template\n\nAdds the main Jinja2 template for …
chalmerlowe Sep 16, 2025
ae7d3e1
feat: Add _client_helpers.j2 partial template\n\nAdds a Jinja2 partia…
chalmerlowe Sep 16, 2025
0122907
Merge branch 'autogen' into feat/adds-client-helpers-template
chalmerlowe Oct 6, 2025
913f521
Update scripts/microgenerator/testing/constraints-3.13.txt
chalmerlowe Oct 6, 2025
3bf9f16
Update scripts/microgenerator/generate.py
chalmerlowe Oct 6, 2025
898eab9
Update scripts/microgenerator/generate.py
chalmerlowe Oct 6, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions scripts/microgenerator/generate.py
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,7 @@ def visit_Import(self, node: ast.Import) -> None:

def visit_ImportFrom(self, node: ast.ImportFrom) -> None:
"""Catches 'from X import Y' statements."""

if self._depth == 0: # Only top-level imports
module = node.module or ""
if not module:
Expand Down Expand Up @@ -158,6 +159,7 @@ def visit_ClassDef(self, node: ast.ClassDef) -> None:

self.structure.append(class_info)
self._current_class_info = class_info

self._depth += 1
self.generic_visit(node)
self._depth -= 1
Expand Down
49 changes: 49 additions & 0 deletions scripts/microgenerator/templates/partials/_client_helpers.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
{#
This is a partial template file intended to be included in other templates.
It contains helper methods for the BigQueryClient class.
#}

# --- HELPER METHODS ---
def _parse_dataset_path(self, dataset_path: str) -> Tuple[Optional[str], str]:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these helper functions going to be included in every generated client or is the inclusion of each individual function done on a selective basis?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for checking.

There is only one generated client and these three helper functions will be included one time via jinja templating imports in the file that the client class is placed in.

For context: The purpose of this microgenerator is to create code that will support customer transition from using a strictly handwritten BigQuery (BQ) library to using a GAPIC generated BQ library. Specifically it will allow the customer to use a single BigQueryClient instead of using seven very disparate *ServiceClients (e.g. JobServiceClient, TableServiceClient, ModelServiceClient) AND to do so in a way that feels very similar to how they use BigQuery now.

To help facilitate this, a proxy of certain attributes associated with each *ServiceClient will be added to new a centralized BigQueryClient and provide a passthrough so that users can use the attribute they want (e.g. JobServiceClient.list_jobs(), DatasetServiceClient.get_dataset(), TableServiceClient.update_table()) in the ServiceClient they want.

Our helpers are present so that customer can provide the BigQueryClient method calls with strings that can be parsed and passed successfully to the underlying attributes.

As an example, without the centralized BigQueryClient and the helpers:

To use TableServiceClient.list_tables(), the customer would need to instantiate a ListTableRequest() object which will require the customer to pass in info such as the project_id and dataset_id.

req = ListTableRequest(project="proj_id", dataset="dataset_id")
tableClient = TableServiceClient.list_tables(request=req)

For the past ten years, customers have not needed to do that extra step.

With the new BigQueryClient, the passthrough methods, and the helpers, the customer will have a user experience much closer to what they are used to, for example:

bqClient = BigQueryClient.list_tables("proj_id.dataset_id")

Our helpers take strings similar to proj_id.dataset_id and create an appropriate *Request object that is passed invisibly on behalf of the customer.

It may seem like a little thing, but it will hopefully ease the transition to using the new GAPIC generated library.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gkevinzheng see above.

"""
Helper to parse project_id and/or dataset_id from a string identifier.

Args:
dataset_path: A string in the format 'project_id.dataset_id' or
'dataset_id'.

Returns:
A tuple of (project_id, dataset_id).
"""
if "." in dataset_path:
# Use rsplit to handle legacy paths like `google.com:my-project.my_dataset`.
project_id, dataset_id = dataset_path.rsplit(".", 1)
return project_id, dataset_id
return self.project, dataset_path

def _parse_dataset_id_to_dict(self, dataset_id: "DatasetIdentifier") -> dict:
"""
Helper to create a dictionary from a project_id and dataset_id to pass
internally between helper functions.

Args:
dataset_id: A string or DatasetReference.

Returns:
A dict of {"project_id": project_id, "dataset_id": dataset_id_str }.
"""
if isinstance(dataset_id, str):
project_id, dataset_id_str = self._parse_dataset_path(dataset_id)
return {"project_id": project_id, "dataset_id": dataset_id_str}
elif isinstance(dataset_id, dataset_reference.DatasetReference):
return {
"project_id": dataset_id.project_id,
"dataset_id": dataset_id.dataset_id,
}
else:
raise TypeError(f"Invalid type for dataset_id: {type(dataset_id)}")

def _parse_project_id_to_dict(self, project_id: Optional[str] = None) -> dict:
"""Helper to create a request dictionary from a project_id."""
final_project_id = project_id or self.project
return {"project_id": final_project_id}
1 change: 1 addition & 0 deletions scripts/microgenerator/testing/constraints-3.13.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
1
Loading