CI occasionally times out

[connorkuehl]

Chances are that this could also time out if you're lucky and run it on your local workstation with go test -race but I've only seen this crop up on CI runs and not on my machine (yet).

In this case, it seems like a call to Libvirt.Disconnect is never returning. The goroutines labeled "semacquire" are kind of interesting but they don't have an indicator that suggests how long they've been in that state.

stack trace

2022-08-04T22:21:08.0397547Z panic: test timed out after 10m0s
2022-08-04T22:21:08.0398049Z 
2022-08-04T22:21:08.0398246Z goroutine 165 [running]:
2022-08-04T22:21:08.0398567Z testing.(*M).startAlarm.func1()
2022-08-04T22:21:08.0488201Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/testing/testing.go:1788 +0xbb
2022-08-04T22:21:08.0488717Z created by time.goFunc
2022-08-04T22:21:08.0489175Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/time/sleep.go:180 +0x4a
2022-08-04T22:21:08.0489433Z 
2022-08-04T22:21:08.0489623Z goroutine 1 [chan receive, 9 minutes]:
2022-08-04T22:21:08.0489991Z testing.(*T).Run(0xc000108ea0, {0x73eb8c, 0xa}, 0x753288)
2022-08-04T22:21:08.0490462Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/testing/testing.go:1307 +0x752
2022-08-04T22:21:08.0490820Z testing.runTests.func1(0x0)
2022-08-04T22:21:08.0491225Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/testing/testing.go:1598 +0x9a
2022-08-04T22:21:08.0491609Z testing.tRunner(0xc000108ea0, 0xc000161bf8)
2022-08-04T22:21:08.0492037Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/testing/testing.go:1259 +0x230
2022-08-04T22:21:08.0492459Z testing.runTests(0xc000158080, {0x92c9e0, 0x3f, 0x3f}, {0x0, 0xc0000129c0, 0x931880})
2022-08-04T22:21:08.0492911Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/testing/testing.go:1596 +0x7cb
2022-08-04T22:21:08.0493487Z testing.(*M).Run(0xc000158080)
2022-08-04T22:21:08.0493871Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/testing/testing.go:1504 +0x9d2
2022-08-04T22:21:08.0494186Z main.main()
2022-08-04T22:21:08.0494479Z 	_testmain.go:167 +0x22c
2022-08-04T22:21:08.0494681Z 
2022-08-04T22:21:08.0494852Z goroutine 90 [semacquire]:
2022-08-04T22:21:08.0495549Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).handle(0x0, {0x7a0b70, 0xc0000ac000})
2022-08-04T22:21:08.0496126Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:622 +0x45
2022-08-04T22:21:08.0496662Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial.func1()
2022-08-04T22:21:08.0497217Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:611 +0x51
2022-08-04T22:21:08.0497749Z created by github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial
2022-08-04T22:21:08.0498297Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:610 +0x1b2
2022-08-04T22:21:08.0498568Z 
2022-08-04T22:21:08.0498732Z goroutine 123 [runnable]:
2022-08-04T22:21:08.0499837Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).handle(0x49fa65, {0x7a0b70, 0xc0000ac400})
2022-08-04T22:21:08.0500939Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:622 +0x45
2022-08-04T22:21:08.0501455Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial.func1()
2022-08-04T22:21:08.0501952Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:611 +0x51
2022-08-04T22:21:08.0502406Z created by github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial
2022-08-04T22:21:08.0502877Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:610 +0x1b2
2022-08-04T22:21:08.0503079Z 
2022-08-04T22:21:08.0503182Z goroutine 119 [semacquire]:
2022-08-04T22:21:08.0503650Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).handle(0x61cc6f, {0x7a0b70, 0xc0000ac300})
2022-08-04T22:21:08.0504182Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:622 +0x45
2022-08-04T22:21:08.0504631Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial.func1()
2022-08-04T22:21:08.0505099Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:611 +0x51
2022-08-04T22:21:08.0505544Z created by github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial
2022-08-04T22:21:08.0506186Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:610 +0x1b2
2022-08-04T22:21:08.0506390Z 
2022-08-04T22:21:08.0506489Z goroutine 95 [runnable]:
2022-08-04T22:21:08.0506943Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).handle(0x49fa65, {0x7a0b70, 0xc0000ac080})
2022-08-04T22:21:08.0507440Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:622 +0x45
2022-08-04T22:21:08.0507880Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial.func1()
2022-08-04T22:21:08.0508363Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:611 +0x51
2022-08-04T22:21:08.0508809Z created by github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial
2022-08-04T22:21:08.0509276Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:610 +0x1b2
2022-08-04T22:21:08.0509465Z 
2022-08-04T22:21:08.0509562Z goroutine 107 [runnable]:
2022-08-04T22:21:08.0509802Z net.isClosedChan(0xc0001a6de0)
2022-08-04T22:21:08.0510134Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/net/pipe.go:73 +0x65
2022-08-04T22:21:08.0510435Z net.(*pipe).read(0xc000158280, {0xc000434900, 0x1c, 0x0})
2022-08-04T22:21:08.0510770Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/net/pipe.go:153 +0xb6
2022-08-04T22:21:08.0511150Z net.(*pipe).Read(0x6f72e0, {0xc000434900, 0x1c, 0x1c})
2022-08-04T22:21:08.0511480Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/net/pipe.go:142 +0x55
2022-08-04T22:21:08.0511981Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).handle(0xc0001a2000, {0x7a0b70, 0xc000158280})
2022-08-04T22:21:08.0512474Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:623 +0x65
2022-08-04T22:21:08.0513018Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial.func1()
2022-08-04T22:21:08.0513490Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:611 +0x51
2022-08-04T22:21:08.0513945Z created by github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial
2022-08-04T22:21:08.0514411Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:610 +0x1b2
2022-08-04T22:21:08.0514609Z 
2022-08-04T22:21:08.0514713Z goroutine 115 [semacquire]:
2022-08-04T22:21:08.0515167Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).handle(0x49fa65, {0x7a0b70, 0xc0000ac200})
2022-08-04T22:21:08.0515649Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:622 +0x45
2022-08-04T22:21:08.0516092Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial.func1()
2022-08-04T22:21:08.0516555Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:611 +0x51
2022-08-04T22:21:08.0517004Z created by github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial
2022-08-04T22:21:08.0517474Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:610 +0x1b2
2022-08-04T22:21:08.0517661Z 
2022-08-04T22:21:08.0517762Z goroutine 127 [semacquire]:
2022-08-04T22:21:08.0518210Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).handle(0x49fa65, {0x7a0b70, 0xc0000ac500})
2022-08-04T22:21:08.0518696Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:622 +0x45
2022-08-04T22:21:08.0519135Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial.func1()
2022-08-04T22:21:08.0519600Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:611 +0x51
2022-08-04T22:21:08.0520041Z created by github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial
2022-08-04T22:21:08.0520503Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:610 +0x1b2
2022-08-04T22:21:08.0520694Z 
2022-08-04T22:21:08.0520792Z goroutine 131 [runnable]:
2022-08-04T22:21:08.0521056Z net.(*pipe).Read(0x6f72e0, {0xc0003b2380, 0x1c, 0x1c})
2022-08-04T22:21:08.0521382Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/net/pipe.go:143 +0xb2
2022-08-04T22:21:08.0521870Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).handle(0x49fa65, {0x7a0b70, 0xc0000ac600})
2022-08-04T22:21:08.0522354Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:623 +0x65
2022-08-04T22:21:08.0522858Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial.func1()
2022-08-04T22:21:08.0523320Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:611 +0x51
2022-08-04T22:21:08.0523764Z created by github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial
2022-08-04T22:21:08.0524231Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:610 +0x1b2
2022-08-04T22:21:08.0524422Z 
2022-08-04T22:21:08.0524510Z goroutine 135 [runnable]:
2022-08-04T22:21:08.0524777Z net.(*pipe).Read(0xc0000ac700, {0xc0003c4200, 0x1c, 0x1c})
2022-08-04T22:21:08.0525110Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/net/pipe.go:141 +0x290
2022-08-04T22:21:08.0525597Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).handle(0x49fa65, {0x7a0b70, 0xc0000ac700})
2022-08-04T22:21:08.0526073Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:623 +0x65
2022-08-04T22:21:08.0526508Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial.func1()
2022-08-04T22:21:08.0526981Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:611 +0x51
2022-08-04T22:21:08.0527417Z created by github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial
2022-08-04T22:21:08.0527881Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:610 +0x1b2
2022-08-04T22:21:08.0528076Z 
2022-08-04T22:21:08.0528171Z goroutine 139 [runnable]:
2022-08-04T22:21:08.0528614Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).handle(0x49fa65, {0x7a0b70, 0xc0000ac800})
2022-08-04T22:21:08.0529093Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:622 +0x45
2022-08-04T22:21:08.0529535Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial.func1()
2022-08-04T22:21:08.0530048Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:611 +0x51
2022-08-04T22:21:08.0530488Z created by github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial
2022-08-04T22:21:08.0530952Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:610 +0x1b2
2022-08-04T22:21:08.0531143Z 
2022-08-04T22:21:08.0531253Z goroutine 143 [semacquire]:
2022-08-04T22:21:08.0531707Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).handle(0x49fa65, {0x7a0b70, 0xc0000ac900})
2022-08-04T22:21:08.0532187Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:622 +0x45
2022-08-04T22:21:08.0532625Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial.func1()
2022-08-04T22:21:08.0533081Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:611 +0x51
2022-08-04T22:21:08.0533530Z created by github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial
2022-08-04T22:21:08.0533986Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:610 +0x1b2
2022-08-04T22:21:08.0534178Z 
2022-08-04T22:21:08.0534277Z goroutine 147 [semacquire]:
2022-08-04T22:21:08.0534722Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).handle(0x49fa65, {0x7a0b70, 0xc0000aca00})
2022-08-04T22:21:08.0535198Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:622 +0x45
2022-08-04T22:21:08.0535642Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial.func1()
2022-08-04T22:21:08.0536102Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:611 +0x51
2022-08-04T22:21:08.0536541Z created by github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial
2022-08-04T22:21:08.0536997Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:610 +0x1b2
2022-08-04T22:21:08.0537187Z 
2022-08-04T22:21:08.0537282Z goroutine 111 [runnable]:
2022-08-04T22:21:08.0537720Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).handle(0x49fa65, {0x7a0b70, 0xc0000acb00})
2022-08-04T22:21:08.0538203Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:622 +0x45
2022-08-04T22:21:08.0538639Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial.func1()
2022-08-04T22:21:08.0542432Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:611 +0x51
2022-08-04T22:21:08.0542956Z created by github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial
2022-08-04T22:21:08.0543421Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:610 +0x1b2
2022-08-04T22:21:08.0543614Z 
2022-08-04T22:21:08.0543734Z goroutine 150 [chan receive, 9 minutes]:
2022-08-04T22:21:08.0544137Z github.com/digitalocean/go-libvirt.(*Libvirt).getResponse(...)
2022-08-04T22:21:08.0544542Z 	/home/runner/work/go-libvirt/go-libvirt/rpc.go:292
2022-08-04T22:21:08.0545069Z github.com/digitalocean/go-libvirt.(*Libvirt).requestStream(0xc00020c180, 0x2, 0x20008086, {0x0, 0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0})
2022-08-04T22:21:08.0545540Z 	/home/runner/work/go-libvirt/go-libvirt/rpc.go:226 +0x1ec
2022-08-04T22:21:08.0545940Z github.com/digitalocean/go-libvirt.(*Libvirt).request(...)
2022-08-04T22:21:08.0546313Z 	/home/runner/work/go-libvirt/go-libvirt/rpc.go:202
2022-08-04T22:21:08.0546730Z github.com/digitalocean/go-libvirt.(*Libvirt).Disconnect(0xc00020c180)
2022-08-04T22:21:08.0547165Z 	/home/runner/work/go-libvirt/go-libvirt/libvirt.go:214 +0x65
2022-08-04T22:21:08.0547670Z github.com/digitalocean/go-libvirt.TestEvents(0xc0001a2000)
2022-08-04T22:21:08.0554375Z 	/home/runner/work/go-libvirt/go-libvirt/libvirt_test.go:430 +0xaa8
2022-08-04T22:21:08.0554688Z testing.tRunner(0xc0001a2000, 0x753288)
2022-08-04T22:21:08.0555028Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/testing/testing.go:1259 +0x230
2022-08-04T22:21:08.0555308Z created by testing.(*T).Run
2022-08-04T22:21:08.0555621Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/testing/testing.go:1306 +0x727
2022-08-04T22:21:08.0555804Z 
2022-08-04T22:21:08.0555917Z goroutine 151 [select, 9 minutes]:
2022-08-04T22:21:08.0563276Z net.(*pipe).read(0xc0000acc00, {0xc000532000, 0x1c, 0xc0001a3040})
2022-08-04T22:21:08.0563644Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/net/pipe.go:159 +0x249
2022-08-04T22:21:08.0563945Z net.(*pipe).Read(0xc0000a8180, {0xc000532000, 0x1c, 0x1c})
2022-08-04T22:21:08.0564273Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/net/pipe.go:142 +0x55
2022-08-04T22:21:08.0564842Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).handle(0x61cc6f, {0x7a0b70, 0xc0000acc00})
2022-08-04T22:21:08.0565344Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:623 +0x65
2022-08-04T22:21:08.0565807Z github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial.func1()
2022-08-04T22:21:08.0566287Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:611 +0x51
2022-08-04T22:21:08.0566741Z created by github.com/digitalocean/go-libvirt/libvirttest.(*MockLibvirt).Dial
2022-08-04T22:21:08.0567205Z 	/home/runner/work/go-libvirt/go-libvirt/libvirttest/libvirt.go:610 +0x1b2
2022-08-04T22:21:08.0567407Z 
2022-08-04T22:21:08.0567519Z goroutine 152 [select, 9 minutes]:
2022-08-04T22:21:08.0567803Z net.(*pipe).read(0xc0000acc80, {0xc000167000, 0x1000, 0x449e30})
2022-08-04T22:21:08.0568135Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/net/pipe.go:159 +0x249
2022-08-04T22:21:08.0568445Z net.(*pipe).Read(0x43f6fe, {0xc000167000, 0x1000, 0x1000})
2022-08-04T22:21:08.0568773Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/net/pipe.go:142 +0x55
2022-08-04T22:21:08.0569088Z bufio.(*Reader).Read(0xc00020c420, {0xc0001ac00c, 0x4, 0x4})
2022-08-04T22:21:08.0569418Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/bufio/bufio.go:227 +0x4db
2022-08-04T22:21:08.0569746Z io.ReadAtLeast({0x799240, 0xc00020c420}, {0xc0001ac00c, 0x4, 0x4}, 0x4)
2022-08-04T22:21:08.0570077Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/io/io.go:328 +0xde
2022-08-04T22:21:08.0570316Z io.ReadFull(...)
2022-08-04T22:21:08.0570646Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/io/io.go:347
2022-08-04T22:21:08.0571070Z github.com/digitalocean/go-libvirt/socket.pktlen({0x799240, 0xc00020c420})
2022-08-04T22:21:08.0571526Z 	/home/runner/work/go-libvirt/go-libvirt/socket/socket.go:267 +0x68
2022-08-04T22:21:08.0572022Z github.com/digitalocean/go-libvirt/socket.listen({0x799240, 0xc00020c420}, {0x799520, 0xc00020c180})
2022-08-04T22:21:08.0572492Z 	/home/runner/work/go-libvirt/go-libvirt/socket/socket.go:216 +0x45
2022-08-04T22:21:08.0573029Z github.com/digitalocean/go-libvirt/socket.(*Socket).listenAndRoute(0xc00009a2d0)
2022-08-04T22:21:08.0573490Z 	/home/runner/work/go-libvirt/go-libvirt/socket/socket.go:205 +0x6f
2022-08-04T22:21:08.0573915Z created by github.com/digitalocean/go-libvirt/socket.(*Socket).Connect
2022-08-04T22:21:08.0574366Z 	/home/runner/work/go-libvirt/go-libvirt/socket/socket.go:151 +0x608
2022-08-04T22:21:08.0574554Z 
2022-08-04T22:21:08.0574668Z goroutine 153 [chan receive, 9 minutes]:
2022-08-04T22:21:08.0575087Z github.com/digitalocean/go-libvirt.(*Libvirt).waitAndDisconnect(0xc00020c180)
2022-08-04T22:21:08.0575551Z 	/home/runner/work/go-libvirt/go-libvirt/libvirt.go:690 +0x5b
2022-08-04T22:21:08.0575974Z created by github.com/digitalocean/go-libvirt.(*Libvirt).ConnectToURI
2022-08-04T22:21:08.0576401Z 	/home/runner/work/go-libvirt/go-libvirt/libvirt.go:187 +0xba
2022-08-04T22:21:08.0576774Z FAIL	github.com/digitalocean/go-libvirt	600.029s
2022-08-04T22:21:08.0577197Z ?   	github.com/digitalocean/go-libvirt/internal/constants	[no test files]
2022-08-04T22:21:08.0577616Z ok  	github.com/digitalocean/go-libvirt/internal/event	1.371s
2022-08-04T22:21:08.0578037Z ok  	github.com/digitalocean/go-libvirt/internal/go-xdr/xdr2	0.031s
2022-08-04T22:21:08.0578466Z ?   	github.com/digitalocean/go-libvirt/internal/lvgen	[no test files]
2022-08-04T22:21:08.0579091Z ?   	github.com/digitalocean/go-libvirt/internal/lvgen/gen	[no test files]
2022-08-04T22:21:08.0579513Z ?   	github.com/digitalocean/go-libvirt/libvirttest	[no test files]
2022-08-04T22:21:08.0579902Z ok  	github.com/digitalocean/go-libvirt/socket	0.022s
2022-08-04T22:21:08.0580311Z ?   	github.com/digitalocean/go-libvirt/socket/dialers	[no test files]
2022-08-04T22:21:08.0580614Z FAIL

[king-jam]

I ran into this recently. There is a thread safety issue with coordinating Disconnect and Connect.

If the connection is closed either through Disconnect or the connection being closed by a lower level AND an immediate call to Connect occurs (say for something that wants to auto reconnect), then there is a period of time where a new go routine will get created https://github.com/digitalocean/go-libvirt/blob/master/libvirt.go#LL297C18-L297C18 while the old is still in place and then we will overwrite the channel that is in use here - https://github.com/digitalocean/go-libvirt/blob/master/libvirt.go#L305.

It appears the use of a channel close as a barrier creates the race condition as the connection could go down and be re-established without a Disconnect completing. A sync.Mutex for graceful shutdown may need to be in place to guard against a Connect/Dial to an actively disconnecting instance.

This design appears to existing in socket as well.