I have the following Go-related packages installed: dev-lang/go-1.4.2 dev-go/godep-0_p20150520 dev-go/go-net-1.4.2_p20150520 dev-go/go-text-1.4.2_p20150506 dev-go/go-tools-1.4.2_p20150602 Now write a program that uses golang.org/x/net/context. For testing purposes, the following program suffices: package main import "golang.org/x/net/context" func main() { _ = context.Background() } Put this in $GOPATH/src/whatever/whatever.go, with $GOPATH an otherwise empty directory. This works, using the context package provided by dev-go/go-net ("go install whatever" results in a binary that successfully does nothing). Making $GOPATH/src/whatever an empty git repo and running "godep save" (or "godep save -r") there results in: godep: directory "/usr/lib/go/src" is not using a known version control system Which I believe translates to "godep requires all non-stdlib dependencies to live in version controlled directories", which is not the case for the system-wide golang.org/x/net/context. Trying to give godep a version controlled dependency to work with by running "go get golang.org/x/net/context" results in: go install golang.org/x/net/context: open /usr/lib/go/pkg/linux_amd64/golang.org/x/net/context.a: permission denied and "go get -u golang.org/x/net/context" results in: package golang.org/x/net/context: directory "/usr/lib/go/src/golang.org/x/net/context" is not using a known version control system which I believe translates to "go get is trying to install or update the system copy". Alright, let's manually install a local copy. Looks like the canonical source is in https://go.googlesource.com/net ("go get" finds this by retrieving https://golang.org/x/net/context and looking at the go-import meta tag), so git clone that into $GOPATH/src/golang.org/x/net. This still doesn't work (same errors from godep). "go install" produces some intriguing error messages: % go install golang.org/x/net/context go install golang.org/x/net/context: open /usr/lib/go/pkg/linux_amd64/golang.org/x/net/context.a: permission denied % go install ./src/golang.org/x/net/context go install: no install location for /tmp/go/src/golang.org/x/net/context: hidden by /usr/lib/go/src/golang.org/x/net/context % go build -x golang.org/x/net/context WORK=/tmp/go-build268794770 mkdir -p $WORK/golang.org/x/net/context/_obj/ mkdir -p $WORK/golang.org/x/net/ cd /usr/lib/go/src/golang.org/x/net/context /usr/lib/go/pkg/tool/linux_amd64/6g -o $WORK/golang.org/x/net/context.a -trimpath $WORK -p golang.org/x/net/context -complete -D _/usr/lib/go/src/golang.org/x/net/context -I $WORK -pack ./context.go Looks like Go insists on using the system copy before the local one? This is annoying because (as described on bug 500452) go-tools really has to be installed system-wide, as that's where "go tool" looks for things, and go-tools depends on go-net. So I can have working godep or working godoc/"go tool cover" and friends, but not both at the same time. How should this conflict be resolved? Is Go preferring GOROOT over GOPATH an upstream bug?
(In reply to Marien Zwart from comment #0) > How should this conflict be resolved? After go-tools is installed, you are free to uninstall go-net because it is only a build-time dependency. > Is Go preferring GOROOT over GOPATH an upstream bug? The problem disappears if I do this as root: touch /usr/lib/go/pkg/linux_amd64/golang.org/x/net/context.a That fits the description of this upstream issue: https://github.com/golang/go/issues/5064
Maybe the ebuild should be tweaked to fiddle with the timestamps of the *.a files relative to the *.go files...
This also fixes it: touch -r /usr/lib/go/src/golang.org/x/net/context/context.go \ /usr/lib/go/pkg/linux_amd64/golang.org/x/net/context.a And if I bump the mtime of context.go, then it's broken again. So, apparently it compares the mtimes of these two files.
I've added a go-net-1.4.2_p20150604 ebuild with the following fix: --- go-net-1.4.2_p20150520.ebuild +++ go-net-1.4.2_p20150604.ebuild @@ -50,5 +50,6 @@ src_install() { insinto /usr/lib/go find "${WORKDIR}"/{pkg,src} -name '.git*' -exec rm -rf {} \; 2>/dev/null + insopts -m0644 -p # preserve timestamps for bug 551486 doins -r "${WORKDIR}"/{pkg,src} } Does this solve your issues?
Ooh, well spotted! This makes a difference, but doesn't solve my actual problem. With this change in place, "go install golang.org/x/net/context" successfully does nothing. It no longer unsuccessfully tries to update /usr/lib/go/pkg/linux_amd64/golang.org/x/net/context.a, but it doesn't build context.a from $GOPATH/src/... either (it ignores the source present there). The same is true for "go get golang.org/x/net/context". "go get -u golang.org/x/net/context" still fails, "go install ./src/golang.org/x/net/context" after manually checking out the code still fails, and godep save still fails, all with the same error messages. Uninstalling go-net does work around it (I tend to keep build deps installed and up to date, hadn't thought of that workaround). This smells as if Go's support for non-stdlib packages in GOROOT, or at least having the same package in GOROOT and GOPATH, is a bit spotty at best (or it really is intentionally preferring GOROOT over GOPATH, which seems backwards). If you prefer to close this and have me file/find an upstream bug, I totally understand. Go itself does at least consider "stdlib" and "installed in GOROOT" different things: cmd/go/pkg only considers things in GOROOT without a "." in the name stdlib packages (which is also why godep is trying to process the package if it's not present in GOPATH). So presumably some support is supposed to be there...
(In reply to Marien Zwart from comment #5) > "go get -u golang.org/x/net/context" still fails, "go install > ./src/golang.org/x/net/context" after manually checking out the code still > fails, and godep save still fails, all with the same error messages. Yeah, I get this error: package golang.org/x/net/context: directory "/usr/lib/go/src/golang.org/x/net/context" is not using a known version control system > Uninstalling go-net does work around it (I tend to keep build deps installed > and up to date, hadn't thought of that workaround). That is consistent with the "try deleting that first" suggestion here: https://code.google.com/p/go-wiki/wiki/InstallTroubleshooting#Why_does_go_get_work_for_some_packages_and_report_permission_den > This smells as if Go's support for non-stdlib packages in GOROOT, or at > least having the same package in GOROOT and GOPATH, is a bit spotty at best > (or it really is intentionally preferring GOROOT over GOPATH, which seems > backwards). If you prefer to close this and have me file/find an upstream > bug, I totally understand. Let's keep this open until we find out whether or not it is considered "correct" to install things in GOROOT, and what alternatives might be available for packagers. > Go itself does at least consider "stdlib" and "installed in GOROOT" > different things: cmd/go/pkg only considers things in GOROOT without a "." > in the name stdlib packages (which is also why godep is trying to process > the package if it's not present in GOPATH). So presumably some support is > supposed to be there... Okay, that's interesting...
(In reply to Zac Medico from comment #6) > (In reply to Marien Zwart from comment #5) > > "go get -u golang.org/x/net/context" still fails, "go install > > ./src/golang.org/x/net/context" after manually checking out the code still > > fails, and godep save still fails, all with the same error messages. > > Yeah, I get this error: > package golang.org/x/net/context: directory > "/usr/lib/go/src/golang.org/x/net/context" is not using a known version > control system > > > Uninstalling go-net does work around it (I tend to keep build deps installed > > and up to date, hadn't thought of that workaround). > > That is consistent with the "try deleting that first" suggestion here: > > https://code.google.com/p/go-wiki/wiki/ > InstallTroubleshooting#Why_does_go_get_work_for_some_packages_and_report_perm > ission_den Ah, and that wiki page also agrees "$GOROOT ... is always checked first" (we don't have to set GOROOT explicitly, "go env" confirms it's /usr/lib/go). It also mentions GOPATH can be a colon-separated list of packages. So if dev-lang/go set a default GOPATH of /usr/lib/go-dist (or similar, I thought about the name for all of 2 seconds), organized as a workspace that packages in dev-go/ install into, that would be more flexible. I could then create my own workspace and put /usr/lib/go-dist behind it in GOPATH. Or it could leave GOPATH unset by default, with ebuilds setting it to pick up the system packages they depend on (and the option for users to do the same). Judging from http://pkg-go.alioth.debian.org/packaging.html we're not the only ones who ran into this: according to that page Debian's policy for Go libraries is to only install the source into /usr/share/gocode/src, and only for the purpose of building other Debian packages. When building Go-based Debian packages, they then symlink those into their builddir. http://anonscm.debian.org/cgit/collab-maint/dh-golang.git/tree/lib/Debian/Debhelper/Buildsystem/golang.pm#n91 explains why they cannot put /usr/share/gocode on GOPATH when building: because Go will then attempt to install compiled library dependencies into it. If we're building those (like dev-go/ stuff currently does) that problem would not apply to us. I poked around in the Go source tree a bit and didn't really see advice for distributors/packagers.
@William: As dev-lang/go maintainer, your thoughts on all of this would be appreciated.
(In reply to Zac Medico from comment #8) > @William: As dev-lang/go maintainer, your thoughts on all of this would be > appreciated. I just read "go help gopath". My interpretation of it is that GOPATH is to be used for things that are not part of the standard go tree. But, if you look at how "go install" behaves, it looks like all of the golang.org/x/* repositories are split out, but upstream wants them installed in $GOROOT. I am working on updating the dev-go/* live ebuilds based on the eclasses I have been working on. I will get those rewritten this weekend. You gave me permission to commit the go-tools live ebuild directly, are you ok with me committing the others? There is another question we should think about for all go packages besides dev-lang/go. That is whether we install the pkg directories at all since the *.a files are only used at build time.
(In reply to William Hubbs from comment #9) > You gave me permission to commit the go-tools live ebuild directly, are you > ok with me committing the others? Yes, please do. > There is another question we should think about for all go packages besides > dev-lang/go. That is whether we install the pkg directories at all since the > *.a files are only used at build time. If they are installed for a library, are they not used when building packages which depend on the library?
(In reply to William Hubbs from comment #9) > (In reply to Zac Medico from comment #8) > > @William: As dev-lang/go maintainer, your thoughts on all of this would be > > appreciated. > > I just read "go help gopath". My interpretation of it is that GOPATH is to > be used for things that are not part of the standard go tree. But, if you > look at how "go install" behaves, it looks like all of the golang.org/x/* > repositories are split out, but upstream wants them installed in $GOROOT. How can you tell upstream wants them installed in $GOROOT? Notice "go get" installs them into GOPATH if that is set, and they're not already present in $GOROOT, just like any other non-stdlib package. go-tools was special, last I checked, but is the other golang.org/x stuff? > I am working on updating the dev-go/* live ebuilds based on the eclasses I > have been working on. I will get those rewritten this weekend. > > You gave me permission to commit the go-tools live ebuild directly, are you > ok with me committing the others? > > There is another question we should think about for all go packages besides > dev-lang/go. That is whether we install the pkg directories at all since the > *.a files are only used at build time. You might run into the same issue pointed out in http://anonscm.debian.org/cgit/collab-maint/dh-golang.git/tree/lib/Debian/Debhelper/Buildsystem/golang.pm#n91 : Go trying to install the missing .a (into $GOROOT) if you build a dependency, and the source is present in $GOROOT. I haven't tested this, though.
(In reply to Zac Medico from comment #10) > (In reply to William Hubbs from comment #9) > > There is another question we should think about for all go packages besides > > dev-lang/go. That is whether we install the pkg directories at all since the > > *.a files are only used at build time. > > If they are installed for a library, are they not used when building > packages which depend on the library? This is exactly why I thought a discussion is a good thing. :-) The *.a files are static objects, so they are linked when a binary uses them. The rule is that the sorce code is where compatibility is guaranteed between different versions of go1, *not* the *.a level, so when the next point release of Go comes out, all *.a objects from external Go packages might have to be rebuilt. I thought about subslotting the external packages based on the go version, but since go-generated binaries are static, the rebuild of the *.a objects doesn't really need to happen until a binary needs them, so I don't know if subslotting is the answer (In reply to Marien Zwart from comment #11) > (In reply to William Hubbs from comment #9) > > (In reply to Zac Medico from comment #8) > > > @William: As dev-lang/go maintainer, your thoughts on all of this would be > > > appreciated. > > > > I just read "go help gopath". My interpretation of it is that GOPATH is to > > be used for things that are not part of the standard go tree. But, if you > > look at how "go install" behaves, it looks like all of the golang.org/x/* > > repositories are split out, but upstream wants them installed in $GOROOT. > > How can you tell upstream wants them installed in $GOROOT? Notice "go get" > installs them into GOPATH if that is set, and they're not already present in > $GOROOT, just like any other non-stdlib package. > go-tools was special, last I checked, but is the other golang.org/x stuff? Sure, "go get" mandates that GOPATH is set and places any package you download into the first directory in GOPATH. I was looking at "go install". This is the command that seems to act differently for the golang.org/x stuff than it does for any other packages. > > There is another question we should think about for all go packages besides > > dev-lang/go. That is whether we install the pkg directories at all since the > > *.a files are only used at build time. > > You might run into the same issue pointed out in > http://anonscm.debian.org/cgit/collab-maint/dh-golang.git/tree/lib/Debian/ > Debhelper/Buildsystem/golang.pm#n91 : Go trying to install the missing .a > (into $GOROOT) if you build a dependency, and the source is present in > $GOROOT. I haven't tested this, though. Again, this is why I was curious about this, see my response to zmedico above about these *.a objects.
(In reply to William Hubbs from comment #12) > I thought about subslotting the external packages based on the go version, > but since go-generated binaries are static, the rebuild of the *.a objects > doesn't really need to happen until a binary needs them, so I don't know if > subslotting is the answer If we use subslotting, then having the slot-operator := deps in DEPEND rather that RDEPEND can be used to to distinguish static linking from dynamic linking. We can use this information to modify the logic that the package manager uses to trigger rebuilds. There can be an option to delay rebuilds of statically linked packages until something that depends on them needs to be built.
(In reply to Zac Medico from comment #13) > (In reply to William Hubbs from comment #12) > > I thought about subslotting the external packages based on the go version, > > but since go-generated binaries are static, the rebuild of the *.a objects > > doesn't really need to happen until a binary needs them, so I don't know if > > subslotting is the answer > > If we use subslotting, then having the slot-operator := deps in DEPEND > rather that RDEPEND can be used to to distinguish static linking from > dynamic linking. We can use this information to modify the logic that the > package manager uses to trigger rebuilds. There can be an option to delay > rebuilds of statically linked packages until something that depends on them > needs to be built. What happens currently if := is in DEPEND rather than RDEPEND? If that works as it is, I suppose the answer would be to do two things: In dev-lang/go: SLOT="0/${PV}" Then in golang-vcs.eclass and golang-build.eclass: DEPEND=">=dev-lang/go-1.4.2:=" correct?
(In reply to William Hubbs from comment #14) > What happens currently if := is in DEPEND rather than RDEPEND? Currently it depends on the --with-bdeps=<y|n> option. If := is only in DEPEND, then you won't see rebuilds triggered unless you specify --with-bdeps=y, since --with-bdeps=n is the default. > If that works as it is, I suppose the answer would be to do two things: > > In dev-lang/go: > > SLOT="0/${PV}" > > Then in golang-vcs.eclass and golang-build.eclass: > > DEPEND=">=dev-lang/go-1.4.2:=" > > correct? Yes.
(In reply to Zac Medico from comment #15) > (In reply to William Hubbs from comment #14) > > What happens currently if := is in DEPEND rather than RDEPEND? > > Currently it depends on the --with-bdeps=<y|n> option. If := is only in > DEPEND, then you won't see rebuilds triggered unless you specify > --with-bdeps=y, since --with-bdeps=n is the default. That makes sense since we are talking about build time dependencies right? I guess the question should be whether the default should change or not,. > > If that works as it is, I suppose the answer would be to do two things: > > > > In dev-lang/go: > > > > SLOT="0/${PV}" > > > > Then in golang-vcs.eclass and golang-build.eclass: > > > > DEPEND=">=dev-lang/go-1.4.2:=" > > > > correct? > > Yes. Thinking about this, I don't think it will be necessary to add the subslot operator in golang-vcs, because that doesn't build anything. (In reply to William Hubbs from comment #12) > (In reply to Marien Zwart from comment #11) > > (In reply to William Hubbs from comment #9) > > > (In reply to Zac Medico from comment #8) > > > > @William: As dev-lang/go maintainer, your thoughts on all of this would be > > > > appreciated. > > > > > > I just read "go help gopath". My interpretation of it is that GOPATH is to > > > be used for things that are not part of the standard go tree. But, if you > > > look at how "go install" behaves, it looks like all of the golang.org/x/* > > > repositories are split out, but upstream wants them installed in $GOROOT. > > > > How can you tell upstream wants them installed in $GOROOT? Notice "go get" > > installs them into GOPATH if that is set, and they're not already present in > > $GOROOT, just like any other non-stdlib package. > > go-tools was special, last I checked, but is the other golang.org/x stuff? > > Sure, "go get" mandates that GOPATH is set and places any package you > download into the first directory in GOPATH. I was looking at "go install". > This is the command that seems to act differently for the golang.org/x stuff > than it does for any other packages. You may be right; this may just apply to go-tools; I'll check it out as I work through the dev-go category.
(In reply to William Hubbs from comment #16) > (In reply to Zac Medico from comment #15) > > (In reply to William Hubbs from comment #14) > > > What happens currently if := is in DEPEND rather than RDEPEND? > > > > Currently it depends on the --with-bdeps=<y|n> option. If := is only in > > DEPEND, then you won't see rebuilds triggered unless you specify > > --with-bdeps=y, since --with-bdeps=n is the default. > > That makes sense since we are talking about build time dependencies right? I > guess the question should be whether the default should change or not,. Right. It's conceivable that we could have an option that is somewhere in between, which searches for installed packages with the := operator in DEPEND and rebuilds them if they were built against an older subslot.
(In reply to Marien Zwart from comment #7) > So if dev-lang/go set a default GOPATH of /usr/lib/go-dist (or similar, I > thought about the name for all of 2 seconds), organized as a workspace that > packages in dev-go/ install into, that would be more flexible. I could then > create my own workspace and put /usr/lib/go-dist behind it in GOPATH. Or it > could leave GOPATH unset by default, with ebuilds setting it to pick up the > system packages they depend on (and the option for users to do the same). I'm thinking something like this is going to be the way to go -- a system-wide "work space", say, /usr/lib/go-Gentoo, which would contain all pkg and src files from all installed go packages other than the Go standard library. /usr/lib/go will be GOROOT and will contain only the standard go tree. I will be able to use my eclasses to set up GOPATH correctly for ebuilds, and install the files correctly. Users would have to set up GOPATH the way they want. > Judging from http://pkg-go.alioth.debian.org/packaging.html we're not the > only ones who ran into this: according to that page Debian's policy for Go > libraries is to only install the source into /usr/share/gocode/src, and only > for the purpose of building other Debian packages. When building Go-based > Debian packages, they then symlink those into their builddir. > http://anonscm.debian.org/cgit/collab-maint/dh-golang.git/tree/lib/Debian/ > Debhelper/Buildsystem/golang.pm#n91 explains why they cannot put > /usr/share/gocode on GOPATH when building: because Go will then attempt to > install compiled library dependencies into it. If we're building those (like > dev-go/ stuff currently does) that problem would not apply to us. > > I poked around in the Go source tree a bit and didn't really see advice for > distributors/packagers. You are correct, there isn't a lot of advice upstream for distributors/packagers. Most of their installation instructions seem to be geared toward user-level installs, e.g. /home/<yourname>/go.
(In reply to William Hubbs from comment #18) > I'm thinking something like this is going to be the way to go -- a > system-wide "work space", say, /usr/lib/go-Gentoo, which would contain all > pkg and src files from all installed go packages other than the Go standard > library. We're using /usr/lib/go-gentoo now, so let's close this.
