I see a couple obvious benefits to doing this: distributed and crowdsourced backups, and the ability for users to browse the wiki offline. Wikipedia already does this of course, so I think it would be good for us to do this too.
Wiki project has had the policy since before my involvement to not perform database dumps (although I think I may have formalized the position at the following link): https://wiki.gentoo.org/wiki/Project:Wiki#Can_I_get_a_dump_of_the_Gentoo_wiki_database.3F There are quite a few considerations if we are to make this a possibility: 1. Historically there were a few 'competing' Gentoo wikis - obviously all of which except for ours were unofficial. Our wiki team didn't want to create more confusion by making it easy for an unofficial, unaffiliated third party to stand up unofficial instances of our wiki, which has a very real possibly to be misleading to our community members. At this point in history, it is much more difficult to get confused. Our wiki has been established in search engine history and the unofficial wikis have all been offline for years. Personally, although I was not the project lead at the time the decision was made to not publish database dumps I still hold a concern that unaffiliated third parties will stand up instances that may be confusing for our community. I'd like other's thoughts on this matter. My caution here is as soon as we publish a single database dump, there is no going back. If unofficial clones appear, we'll want to help them stay up to date with continuous releases, and that sounds like a lot of bandwidth and hosting requirements, which leads into my next point... 2. Does infra have the bandwidth to distribute these database dumps? If we do enable database dumps, what would be the release cadence and how would we distribute them? Dump once per month, once per quarter, one per year? Create an package and publish it out to distfiles/? 3. If we published the wiki, should we also publish the Bugzilla database and/or the packages.g.o databases as well? Perhaps this is comparing apples to oranges, but I find it likely **someone** in the community would want us to follow suit for consistency, although these would certainly be less useful. I'm probably missing some important considerations, so I'd like to rope in more more Infra options. The above reasons are a few quick things off the top of my head. In short - I'd like to make this a reality, but I have concerns over abuse and the cost/benefit ratio seems heavier on the cost side... That's hard to say because Gentoo has a very transparent project structure; we're not trying to hide anything by not releasing the database - it's just that it may not be best to do so.
Looping in infra@ as some of these points are definitely infra questions.
(In reply to John Helmert III from comment #2) > Looping in infra@ as some of these points are definitely infra questions. So the general problem is that dumps contain private information (such as PII) and we want to avoid disseminating that information. Things like who signed up when, from where, hashed and salted passwords, etc. In general I'd advocate for a crawling-based approach (like we have for bugzilla), where users are granted a scoped API token to read what they are allowed to read, and can crawl the site as normal. Specifically for packages.gentoo.org; its not a stateful database. Users who "want a copy" can just download and execute the code and they will get the same data (in their copy of the database) after it syncs up from external sources. -A
I think the PII problem is the most difficult non-technical problem, User:* namespace contains most of it. Effort cost is the largest problem overall: simply dumping SQL is not suitable for the reasons other provided. Is there a meaningful export format and pre-existing automation? Then as you say, we have have to maintain those exports, ensure the tooling keeps working etc. The literal hosting of it isn't difficult per-se. How does Wikipedia handle this (because I know they publish exports)?
I did this recently for work. I believe that the appropriate tool is built into Mediawiki - DumpBackup.php https://www.mediawiki.org/wiki/Manual:DumpBackup.php tl;dr: > XML Dumps contain the content of a wiki (wiki pages with all their revisions), without the site-related data. A XML dump does not create a full backup of the wiki database, the dump does not contain user accounts, images, edit logs, etc. This appears to be the format (and likely tool) that Wikipedia uses. It should be trivial to implement and maintain dumps so I wholeheartedly support the idea!
This is related to bug 671696.
