Better error reporting for non-building code (e.g. moved to Opaque directly instead of Hybrid)

[johanbrandhorst]

Hi, I was trying out the tool today and ran into what I think is a failure (or I could be using it wrong). Given code that looks like this (in /services/athlete/activity.go):

package athlete

import (
	"context"
	"fmt"

	"connectrpc.com/connect"
	athletesv1 "github.com/catzkorn/trail-tools/gen/athletes/v1"
	"github.com/catzkorn/trail-tools/oidc"
	"github.com/catzkorn/trail-tools/store"
	"google.golang.org/protobuf/types/known/timestamppb"
)

func (s *Service) CreateActivity(ctx context.Context, req *connect.Request[athletesv1.CreateActivityRequest]) (*connect.Response[athletesv1.CreateActivityResponse], error) {
	_, ok := oidc.GetUserInfo(ctx)
	if !ok {
		return nil, connect.NewError(connect.CodeUnauthenticated, fmt.Errorf("unauthenticated"))
	}
	athleteID, err := store.StringToUUID(req.Msg.AthleteId)
	if err != nil {
		return nil, connect.NewError(connect.CodeInvalidArgument, fmt.Errorf("invalid athlete ID: %w", err))
	}
	activity, err := s.athletes.AddActivity(ctx, req.Msg.Name, athleteID)
	if err != nil {
		return nil, connect.NewError(connect.CodeInternal, fmt.Errorf("failed to add activity: %w", err))
	}
	return connect.NewResponse(&athletesv1.CreateActivityResponse{
		Activity: &athletesv1.Activity{
			Id:         store.UUIDToString(activity.ID),
			AthleteId:  store.UUIDToString(activity.AthleteID),
			Name:       req.Msg.Name,
			CreateTime: timestamppb.New(activity.CreateTime.Time),
		},
	}), nil
}

And the command

$ go run google.golang.org/open2opaque@latest rewrite ./services/athlete

I saw this error:

rewriting 1 packages:
  github.com/catzkorn/trail-tools/services/athlete
Starting the Blaze loader
Loading packages (in batches of up to 20)...
PROCESSED 1 packages (total patterns: 1)
        Last package:         github.com/catzkorn/trail-tools/services/athlete
        Total time:           161.676935ms
        Package profile:      TOTAL: 1.47597ms | start 292ns main/scheduled 1.46972ms main/fixed 5.666µs main/gotresp 292ns done
        Failures:             1 (100.00%)
        Average time:         161.676935ms
        Estimated until done: 0s
        Estimated done at:    2025-01-01 17:50:31.795823537 -0800 PST m=+0.239883240
        Error:                panic: can't process package: don't know type for expression *dst.Ident      0  *dst.Ident {
     1  .  Name: "AthleteId"
     2  .  Path: ""
     3  .  Decs: dst.IdentDecorations {
     4  .  .  NodeDecs: dst.NodeDecs {
     5  .  .  .  Before: None
     6  .  .  .  After: None
     7  .  .  }
     8  .  }
     9  }



Processed 1 packages:
        successfully analyzed: 0
        failed to load/rewrite: 1
        .go files rewritten: 0
F0101 17:50:31.795940  832174 rewrite.go:347] rewrite() failed: 1 packages could not be rewritten
exit status 1

This was my first attempt at using the tool, so I could be holding it wrong.

[stapelberg]

Hey @johanbrandhorst, long time no see — hope you’re doing well, and happy new year!

Thanks for giving the tool a try and reporting the issue. However, I cannot reproduce what you are running into:

% git clone https://github.com/catzkorn/trail-tools/
% cd trail-tools
% go run google.golang.org/open2opaque@latest rewrite ./services/athlete
Resolving Go package names...
rewriting 1 packages:
  ./services/athlete
Starting the Blaze loader
Loading packages (in batches of up to 20)...
PROCESSED 1 packages (total patterns: 1)
	Last package:         ./services/athlete
	Total time:           306.420691ms
	Package profile:      TOTAL: 31.78µs | start 340ns main/scheduled 25.759µs fix/fixed 411ns fix/wrotefiles 140ns fix/donestats 321ns main/fixed 4.548µs main/gotresp 261ns done
	Failures:             0 (0.00%)
	Average time:         306.420691ms
	Estimated until done: 0s
	Estimated done at:    2025-01-06 11:26:29.552047392 +0100 CET m=+0.638456283
	Error:                <nil>


Processed 1 packages:
	successfully analyzed: 1
	failed to load/rewrite: 0
	.go files rewritten: 0
go run google.golang.org/open2opaque@latest rewrite ./services/athlete  9,80s user 2,14s system 176% cpu 6,770 total

Could you share more details about which Go version and OS you are using, and whether anything is missing in the steps to reproduce please? Thank you.

[johanbrandhorst]

Doing great thank you 😄, and happy new year to you too! I should've been a bit more specific, but I can still reproduce this today with:

$ git fetch
$ git checkout origin/jbrandhorst/reproduce-error
$ go run google.golang.org/open2opaque@latest rewrite ./services/athlete
Resolving Go package names...
rewriting 1 packages:
  github.com/catzkorn/trail-tools/services/athlete
Starting the Blaze loader
Loading packages (in batches of up to 20)...
PROCESSED 1 packages (total patterns: 1)
        Last package:         github.com/catzkorn/trail-tools/services/athlete
        Total time:           162.507347ms
        Package profile:      TOTAL: 1.457998ms | start 291ns main/scheduled 1.449999ms main/fixed 7.375µs main/gotresp 333ns done
        Failures:             1 (100.00%)
        Average time:         162.507347ms
        Estimated until done: 0s
        Estimated done at:    2025-01-06 07:37:19.978629449 -0800 PST m=+0.220220636
        Error:                panic: can't process package: don't know type for expression *dst.Ident      0  *dst.Ident {
     1  .  Name: "AthleteId"
     2  .  Path: ""
     3  .  Decs: dst.IdentDecorations {
     4  .  .  NodeDecs: dst.NodeDecs {
     5  .  .  .  Before: None
     6  .  .  .  After: None
     7  .  .  }
     8  .  }
     9  }



Processed 1 packages:
        successfully analyzed: 0
        failed to load/rewrite: 1
        .go files rewritten: 0
F0106 07:37:19.978742  958333 rewrite.go:347] rewrite() failed: 1 packages could not be rewritten
exit status 1

Please try that and let me know if you see the same result.

[stapelberg]

Thank you for the reproduction steps.

I can reproduce the open2opaque error now, but I will note that your code does not build to begin with:

% go build ./...
go: downloading golang.org/x/net v0.33.0
# github.com/catzkorn/trail-tools/services/athlete
services/athlete/activity.go:19:47: req.Msg.AthleteId undefined (type *athletesv1.CreateActivityRequest has no field or method AthleteId)
services/athlete/activity.go:23:55: req.Msg.Name undefined (type *athletesv1.CreateActivityRequest has no field or method Name)
services/athlete/activity.go:28:3: unknown field Activity in struct literal of type athletesv1.CreateActivityResponse
services/athlete/activity.go:29:4: unknown field Id in struct literal of type athletesv1.Activity
services/athlete/activity.go:30:4: unknown field AthleteId in struct literal of type athletesv1.Activity
services/athlete/activity.go:31:4: unknown field Name in struct literal of type athletesv1.Activity
services/athlete/activity.go:31:24: req.Msg.Name undefined (type *athletesv1.CreateActivityRequest has no field or method Name)
services/athlete/activity.go:32:4: unknown field CreateTime in struct literal of type athletesv1.Activity
services/athlete/athlete.go:19:13: req.Msg.Name undefined (type *athletesv1.CreateAthleteRequest has no field or method Name)
services/athlete/athlete.go:26:53: req.Msg.Name undefined (type *athletesv1.CreateAthleteRequest has no field or method Name)
services/athlete/athlete.go:26:53: too many errors
# github.com/catzkorn/trail-tools/services/user
services/user/user.go:22:3: unknown field User in struct literal of type usersv1.GetCurrentUserResponse
services/user/user.go:23:4: unknown field Id in struct literal of type usersv1.User
services/user/user.go:24:4: unknown field Email in struct literal of type usersv1.User
services/user/user.go:25:4: unknown field Name in struct literal of type usersv1.User
services/user/user.go:26:4: unknown field GivenName in struct literal of type usersv1.User
services/user/user.go:27:4: unknown field FamilyName in struct literal of type usersv1.User
services/user/user.go:28:4: unknown field AvatarUrl in struct literal of type usersv1.User

Looking at the commit history, I see that you already re-generated your .proto files on the Opaque API level.

You’ll need to select the Hybrid API first. See https://protobuf.dev/reference/go/opaque-migration/#setup for instructions.

Let me know if this makes sense or if you run into further issues

[johanbrandhorst]

Ah, I see! My bad. I wonder if the tool could error if it detects that the hybrid API is not used on the underlying Go protobuf files? I clearly jumped ahead without reading the instructions, but I probably won't be the only one 😅.

[stapelberg]

Yes, I think that is a good idea. I haven’t quite decided (or experimented with) whether printing a couple of “these are common problems” messages in case of any failure is the best we can do, or if it makes sense to explicitly detect the API level and print a more targeted message. I’ll probably try the latter and do the former as well, regardless.

[stapelberg]

I haven’t quite decided (or experimented with) whether printing a couple of “these are common problems” messages in case of any failure is the best we can do, or if it makes sense to explicitly detect the API level and print a more targeted message. I’ll probably try the latter and do the former as well, regardless.

I looked into it a bit: While we do have a use-case at Google where we rely on partial type information, that case goes through a different loader entirely. So, most likely, we can use pkg.Errors in loaderblaze.go to detect code which does not type-check. I’ll give it a shot.

[stapelberg]

With commit f4fb071, the error message is now much clearer:

% (cd /tmp/rewrite/trail-tools && open2opaque rewrite ./services/athlete)
Resolving Go package names...
rewriting 1 packages:
  github.com/catzkorn/trail-tools/services/athlete
Starting the Blaze loader
Loading packages (in batches of up to 20)...
PROCESSED 1 packages (total patterns: 1)
	Last package:         github.com/catzkorn/trail-tools/services/athlete
	Total time:           558.970761ms
	Package profile:      TOTAL: 2.985µs | start 2.414µs main/gotresp 571ns done
	Failures:             1 (100.00%)
	Average time:         558.970761ms
	Estimated until done: 0s
	Estimated done at:    2025-01-16 13:01:26.632310741 +0100 CET m=+0.903954451
	Error:                Loading package failed:
[-: # github.com/catzkorn/trail-tools/services/athlete
services/athlete/activity.go:19:47: req.Msg.AthleteId undefined (type *athletesv1.CreateActivityRequest has no field or method AthleteId)
services/athlete/activity.go:23:55: req.Msg.Name undefined (type *athletesv1.CreateActivityRequest has no field or method Name)
services/athlete/activity.go:28:3: unknown field Activity in struct literal of type athletesv1.CreateActivityResponse
services/athlete/activity.go:29:4: unknown field Id in struct literal of type athletesv1.Activity
services/athlete/activity.go:30:4: unknown field AthleteId in struct literal of type athletesv1.Activity
services/athlete/activity.go:31:4: unknown field Name in struct literal of type athletesv1.Activity
services/athlete/activity.go:31:24: req.Msg.Name undefined (type *athletesv1.CreateActivityRequest has no field or method Name)
services/athlete/activity.go:32:4: unknown field CreateTime in struct literal of type athletesv1.Activity
services/athlete/athlete.go:19:13: req.Msg.Name undefined (type *athletesv1.CreateAthleteRequest has no field or method Name)
services/athlete/athlete.go:26:53: req.Msg.Name undefined (type *athletesv1.CreateAthleteRequest has no field or method Name)
services/athlete/athlete.go:26:53: too many errors /tmp/rewrite/trail-tools/services/athlete/activity.go:19:47: req.Msg.AthleteId undefined (type *athletesv1.CreateActivityRequest has no field or method AthleteId) /tmp/rewrite/trail-tools/services/athlete/activity.go:23:55: req.Msg.Name undefined (type *athletesv1.CreateActivityRequest has no field or method Name) /tmp/rewrite/trail-tools/services/athlete/activity.go:29:4: unknown field Id in struct literal of type athletesv1.Activity /tmp/rewrite/trail-tools/services/athlete/activity.go:30:4: unknown field AthleteId in struct literal of type athletesv1.Activity /tmp/rewrite/trail-tools/services/athlete/activity.go:31:24: req.Msg.Name undefined (type *athletesv1.CreateActivityRequest has no field or method Name) /tmp/rewrite/trail-tools/services/athlete/activity.go:31:4: unknown field Name in struct literal of type athletesv1.Activity /tmp/rewrite/trail-tools/services/athlete/activity.go:32:4: unknown field CreateTime in struct literal of type athletesv1.Activity /tmp/rewrite/trail-tools/services/athlete/activity.go:28:3: unknown field Activity in struct literal of type athletesv1.CreateActivityResponse /tmp/rewrite/trail-tools/services/athlete/athlete.go:19:13: req.Msg.Name undefined (type *athletesv1.CreateAthleteRequest has no field or method Name) /tmp/rewrite/trail-tools/services/athlete/athlete.go:26:53: req.Msg.Name undefined (type *athletesv1.CreateAthleteRequest has no field or method Name) /tmp/rewrite/trail-tools/services/athlete/athlete.go:32:24: req.Msg.Name undefined (type *athletesv1.CreateAthleteRequest has no field or method Name) /tmp/rewrite/trail-tools/services/athlete/athlete.go:32:4: unknown field Name in struct literal of type athletesv1.Athlete /tmp/rewrite/trail-tools/services/athlete/athlete.go:33:4: unknown field CreateTime in struct literal of type athletesv1.Athlete /tmp/rewrite/trail-tools/services/athlete/athlete.go:34:4: unknown field Id in struct literal of type athletesv1.Athlete /tmp/rewrite/trail-tools/services/athlete/athlete.go:31:3: unknown field Athlete in struct literal of type athletesv1.CreateAthleteResponse /tmp/rewrite/trail-tools/services/athlete/measure.go:20:13: req.Msg.HeartRateBpm undefined (type *athletesv1.CreateBloodLactateMeasureRequest has no field or method HeartRateBpm) /tmp/rewrite/trail-tools/services/athlete/measure.go:21:106: req.Msg.HeartRateBpm undefined (type *athletesv1.CreateBloodLactateMeasureRequest has no field or method HeartRateBpm) /tmp/rewrite/trail-tools/services/athlete/measure.go:23:48: req.Msg.ActivityId undefined (type *athletesv1.CreateBloodLactateMeasureRequest has no field or method ActivityId) /tmp/rewrite/trail-tools/services/athlete/measure.go:27:45: req.Msg.MmolPerLiter undefined (type *athletesv1.CreateBloodLactateMeasureRequest has no field or method MmolPerLiter) /tmp/rewrite/trail-tools/services/athlete/measure.go:32:119: req.Msg.MmolPerLiter undefined (type *athletesv1.CreateBloodLactateMeasureRequest has no field or method MmolPerLiter) /tmp/rewrite/trail-tools/services/athlete/measure.go:34:71: req.Msg.HeartRateBpm undefined (type *athletesv1.CreateBloodLactateMeasureRequest has no field or method HeartRateBpm) /tmp/rewrite/trail-tools/services/athlete/measure.go:40:4: unknown field MmolPerLiter in struct literal of type athletesv1.BloodLactateMeasure /tmp/rewrite/trail-tools/services/athlete/measure.go:41:4: unknown field HeartRateBpm in struct literal of type athletesv1.BloodLactateMeasure /tmp/rewrite/trail-tools/services/athlete/measure.go:42:4: unknown field CreateTime in struct literal of type athletesv1.BloodLactateMeasure /tmp/rewrite/trail-tools/services/athlete/measure.go:43:4: unknown field Id in struct literal of type athletesv1.BloodLactateMeasure /tmp/rewrite/trail-tools/services/athlete/measure.go:44:4: unknown field ActivityId in struct literal of type athletesv1.BloodLactateMeasure /tmp/rewrite/trail-tools/services/athlete/measure.go:39:3: unknown field BloodLactateMeasure in struct literal of type athletesv1.CreateBloodLactateMeasureResponse]


Processed 1 packages:
	successfully analyzed: 0
	failed to load/rewrite: 1
	.go files rewritten: 0

rewrite() failed: 1 packages could not be rewritten

Frequent mistakes include:

- Enabling the Opaque API before rewriting your code.
  Instead, set your .proto files to the Hybrid API
  before running open2opaque rewrite. For details, see
  https://protobuf.dev/reference/go/opaque-migration/

- Your code must build for the open2opaque tool to work.
  Check that a build shows no errors: go build ./...

- Incorrectly migrating .proto files to edition 2023.
  It is not sufficient to only replace the syntax line.
  To preserve your current behavior (proto2 or proto3)
  when migrating to edition 2023, follow this document:
  https://protobuf.dev/editions/features/#preserving