| Summary: | net-analyzer/netdata-go-plugin - Additional plugin for Netdata written in Go | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | EK <ek> |
| Component: | New packages | Assignee: | Craig Andrews <candrews> |
| Status: | RESOLVED FIXED | ||
| Severity: | enhancement | CC: | aparicio99, arkamar, candrews, genzilla, jstein, mail, michiel, skobkin-ru, srcshelton, steffen.weber |
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| URL: | https://github.com/netdata/go.d.plugin | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Attachments: |
proposed unified ebuild
patch for bad module checksum ebuild for 0.31.2 version netdata-go-plugins-bin ebuild |
||
|
Description
EK
2021-02-26 18:32:02 UTC
The go plugin is separate from netdata. The netdata installer script downloads and installs it: https://github.com/netdata/netdata/blob/v1.30.0/netdata-installer.sh#L1485 And that page, https://github.com/netdata/go.d.plugin says: "go.d.plugin is a Netdata external plugin." so I think this plugin should be a new Gentoo package. Currently all plugins are being rewritten in go, not everything is complete for the current release. So for the upcomming netdata release this would be a great addition. (In reply to Craig Andrews from comment #1) > The go plugin is separate from netdata. > > The netdata installer script downloads and installs it: > https://github.com/netdata/netdata/blob/v1.30.0/netdata-installer.sh#L1485 > > And that page, https://github.com/netdata/go.d.plugin says: > "go.d.plugin is a Netdata external plugin." > > so I think this plugin should be a new Gentoo package. The word "external" here is misleading. The go.d plugin is *not* separate from netdata. This misunderstanding is a consequence of Netdata's choice of terminology, described here: https://learn.netdata.cloud/docs/collect/how-collectors-work#collector-architecture-and-terminology Specifically, from the above docs, Netdata's distinction between "internal" and "external" plugins is defined as such: * External plugins gather metrics from external processes, such as a webserver or database, and run as independent processes that communicate with the Netdata daemon via pipes. * Internal plugins gather metrics from /proc, /sys, and other Linux kernel sources. They are written in C, and run as threads within the Netdata daemon. Whether this unfortunate choice of nomenclature aligns in any way with *our* expectation of what "internal" and "external" mean from a package maintainership perspective was evidently not considered by the authors of that documentation. To repeat: this does *not* mean the plugin is separate from Netdata in the sense that it is in any way external to the project; rather it means only that the plugin monitors things outside of Netdata and the kernel. Indeed it is otherwise explicitly stated to be shipped with Netdata (cf. https://github.com/netdata/go.d.plugin#install, "Go.d.plugin is shipped with Netdata."), and this statement is consistent with Netdata's packaging on most other distributions, but Gentoo's packaging currently violates that expectation without an alternative. Inclusion of package options should in general be controlled by USE flags, not by fragmentation of the project into multiple packages. Modular package behavior is what USE flags are for. In this case, the prevalence of plugin-specific packages across other distributions, either for the go or python plugins, is inconsistent. Which way is easier or makes more sense is a philosophical point and up to the Gentoo maintainers to decide, but given these bits are meant to be distributed together, I suggest they should stay together unless there is some game-breaking reason the plugins for Netdata can't be handled in the same ebuild as Netdata itself. It is worth pointing out that, despite running afoul of all the same distinctions, the python.d plugin and its collectors are installed by the current ebuild. For it to provide one plugin and not the other is an inconsistency I suspect was not intentional. Any impression that the go.d plugin collectors are still incomplete, experimental, or otherwise unfit for use is also, at this time, incorrect. Several of the go collectors are now superior to their python counterparts. If I can write the ebuild bits for the go plugin, I will submit them here. Created attachment 762928 [details]
proposed unified ebuild
Created attachment 762929 [details, diff]
patch for bad module checksum
Proposals attached. But first, storytime. :x I have discovered a small rabbit hole waiting for the person who attempts to write an ebuild for the Netdata go plugin. The details are not strictly relevant to this bug, but I would not have been able to complete my attempt without solving the problem I am about to describe. The go.d.plugin project uses Go modules. Not surprising, and not a problem. For this the ebuild must inherit the go-module eclass, and there is a small dance one can do with the GO_OPTIONAL environment variable if one wishes to conceal its behavior behind a USE flag. All well and good, until you get down to src_compile and Go wants to verify all those modules fetched by the ebuild's proxy. One of the modules fails its checksum verification: ``` go: downloading github.com/ilyam8/hashstructure v1.1.0 verifying github.com/ilyam8/hashstructure@v1.1.0: checksum mismatch downloaded: h1:o3hpiGa1yergDO4AlUVBJGNvCEeaD+Ynu0OWYxee0iA= go.sum: h1:N8t8hzzKLf2Da87XgC/DBYqXUmSbclgx+2cZxS5/klU= ``` This is unexpected -- mostly because it doesn't happen if you try the build yourself outside of Portage in an ordinary git checkout of the go.d.plugin repository. The build isn't hard; you just CD into the top folder and run `make`, which in turn does `go mod download` and `go build` plus a couple other things. The rabbit hole shows up when you get to exploring the difference between what happens when `go mod download` fetches the modules on its own and what happens when Portage pre-fetches them. Spoiler alert: one of them makes a mistake, and it isn't Portage. Starting with the expectation -- this is the code that was supposed to be fetched: https://github.com/ilyam8/hashstructure/blob/v1.1.0/hashstructure.go In my ordinary checkout, after doing the build successfully, I inspect my Go module cache. I find there the code for the hashstructure module mentioned in the verification error, and I compare it to what is in the repository. I find that the code installed in my cache matches an *earlier* tag: https://github.com/ilyam8/hashstructure/blob/v1.0.0/hashstructure.go I do `go clean -modcache` and run the build again. It validates all the modules again and succeeds again with the wrong code. I use a different machine; I have a Chromebook with a Linux container on it. Same thing. I conclude something is consistently serving the wrong bits. A stale cache somewhere. By this time, I have already inspected the modules downloaded by Portage for the go-module eclass. I found the code inside matched the expected commit tagged with "v1.1.0" -- meaning that Portage fetched the right bits where Go on its own fetched the wrong bits. The "downloaded" Hash1 sum above represents the true checksum of the desired content. So I have my ebuild patch that checksum into the go.sum file of the go.d.plugin tree, and I find the ebuild runs to completion. I confess I don't know why this is happening. While I am aware that certain versions of Go once upon a time generated inconsistent hashes, I am confident this is not that problem. For one thing, these checksums were generated by much later versions of Go; and for another, both hashes are stable. I don't well understand what Portage does differently inside of the go-module eclass that permits it to obtain the expected code, either, unless it is ultimately obtaining those sources from more stable repositories than what `go mod download` would ordinarily select. I know that this bug requests a separate ebuild for "netdata-go-plugins," but in light of my previous comment I implemented the enhancement within a monolithic Netdata ebuild and control the feature selection with the "go" USE flag, following the pattern of some official ebuilds. If a separate package is still the desired outcome, I can do that, but I don't think it really adds any value to complicate it with a dependency relationship. Unless maybe you don't want to rebuild the go.d plugin every time, or you want to update it separately -- but keep in mind the packaging file in the main repository at https://github.com/netdata/netdata/packaging/go.d.version has got the version pin in it so the end-user probably shouldn't be mix-and-matching. I have only done the versioned ebuild, not the live ebuild, because you have to include the EGO_SUM array and it has to match line for line what is in the go.d.plugin repository's go.sum file, which can change at any time. A live ebuild would just use git-r3 for both repositories, but I'm not sure that's a useful pattern because of the need to keep that array updated. Please advise. Noticing now I forgot to update the LICENSE array (or check if it needs updating) for all the modules pulled in. Fixable on the next pass. I agree with Craig and I think the go.d plugin should be a new package. It is released independently to netdata and releases are not synchronized. That would make version bumping and potential stabilization of netdata package unnecessarily complicated. So, I suggest to create a new package and use optfeature eclass in netdata to advertise it. Created attachment 763066 [details]
ebuild for 0.31.2 version
I suggest this new ebuild.
please note that there are also binary releases: https://github.com/netdata/go.d.plugin/releases/tag/v0.31.2 It may be worthwhile to create a virtual that can be satisfied by either a binary or a source ebuild. Any progress on that? I've just realized that Netdata can's collect a lot of metrics on my home server running Gentoo. For example SNMP metrics collection described here: https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/snmp isn't available despite the fact that it's not considered something which should be installed separately by Netdata documentation. Created attachment 802228 [details]
netdata-go-plugins-bin ebuild
My attempt at a netdata-go-plugins-bin ebuild that downloads the pre-compiled go plugins.
Thanks Steffen, seems to be working fine This package provides only one go.d.plugin, net-analyzer/netdata-go-plugin seems to be more appropriate name. The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=20498248615c62dd5c378526bf52e9785055226c commit 20498248615c62dd5c378526bf52e9785055226c Author: Petr Vaněk <arkamar@gentoo.org> AuthorDate: 2024-01-05 10:34:57 +0000 Commit: Petr Vaněk <arkamar@gentoo.org> CommitDate: 2024-01-05 10:53:31 +0000 net-analyzer/netdata-go-plugin: new package, add 0.58.0 Closes: https://bugs.gentoo.org/773112 Signed-off-by: Petr Vaněk <arkamar@gentoo.org> net-analyzer/netdata-go-plugin/Manifest | 2 + net-analyzer/netdata-go-plugin/metadata.xml | 11 +++++ .../netdata-go-plugin-0.58.0.ebuild | 51 ++++++++++++++++++++++ 3 files changed, 64 insertions(+) The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=060e2e411ddd1311bddf854d50970382a00cc631 commit 060e2e411ddd1311bddf854d50970382a00cc631 Author: Petr Vaněk <arkamar@gentoo.org> AuthorDate: 2024-01-05 20:08:28 +0000 Commit: Petr Vaněk <arkamar@gentoo.org> CommitDate: 2024-01-05 20:23:56 +0000 net-analyzer/netdata: advertise net-analyzer/netdata-go-plugin Bug: https://bugs.gentoo.org/773112 Signed-off-by: Petr Vaněk <arkamar@gentoo.org> net-analyzer/netdata/netdata-1.44.0.ebuild | 4 +++- net-analyzer/netdata/netdata-9999.ebuild | 6 ++++-- 2 files changed, 7 insertions(+), 3 deletions(-) |
