Composer (http://getcomposer.org) is becoming the defacto method to handle project specific dependancies in PHP. It is very useful to have it available globally on the system if you work with many projects and are often installing or trying out other people's code which depends on composer. I have an ebuild in my overlay which I am happy to proxy maintain if it gets included the tree: https://github.com/jamiel/gentoo-overlay/blob/master/dev-php/composer/composer-1.0.0_alpha5-r2.ebuild
Bumped to alpha6. https://github.com/jamiel/gentoo-overlay/tree/master/dev-php/composer
Created attachment 351124 [details] ebuild for composer 1.0.0-alpha7
Any reason this is on hold?
Doing mv "${DISTDIR}/${A}" within src_unpack() is risky. In regular portage installations distdir is likely going to be 0755 root:portage and mv will fail with Permission denied. cp as a simple workaround seems to work fine, as I think portage will clean everything up later anyway.
(In reply to Leho Kraav (:macmaN @lkraav) from comment #3) > Any reason this is on hold? Simply nobody has picked it up yet. If someone is interesting in proxy-maintaining it, perhaps the proxy-maintainers team can accommodate that.
For the record, the reason the PHP team did not pick this up is due to our opinion that composer is a method for installing packages per project. It is self contained, the install method is one single command and it is keeping itself up to date. Having it in portage does not give anyone any benefits, as we see it.
Created attachment 368340 [details] composer-1.0.0_alpha8 Hi, here my cent. Composer is a package manager/tool for web applications, but could be important for a gentoo systemist have a way to install this package through portage instead install it manually and share it through server users. Why doesn't supply this tool to web developer through portage ? For example: composer it is used inside laravel framework. However, on my overlay (geaaru-overlay on githup) it is available composer-1.0.0_alpha8 that resolve problems with permission and create a link composer to real version, so could be possible in the future manage multiple version through eselect. Regards, G.
@ole Markus with: I probably should do more research before opening my mouth, but how does composer differ to npm in the nodejs ecosystem? Its a global tool but mostly used for a local/per project purpose. I would personally like to see the tool within gentoo-x86 for convenience (composer <foo> vs php <path/to/composer.phar> <foo>). Also, the ebuild should really check for the CLI sapi.
If we're going to add it, it would be nice if we could have its dependencies unbundled. But as far as I've looked into it, this would mean that'd we have to patch the bootstrap.php and write a custom autoload.php (or composer/autoload_*.php and possibly installed.json).
(In reply to Johan Bergström from comment #8) I agree with Johan. Suppose you have multiple PHP projects - will you keep a composer.phar in each of them? Will you then update each? Or do you want to download it each time you build a project? If you have it globally, you just install it once, it's a tagged release and not just git master. It happens from time to time that you need to downgrade it because, well, it's git master, it can be broken ;)
I've been running with system-level install for months without a single issue. Convenience of this is fantastic.
I would be happy to install it globally if it didn't bundle all of its dependencies, but Tiziano is right: it would be a huge pain to unbundle them (and almost every project that uses composer has the same problem).
Hey there, I maintained composer on my gentoo overlay (https://github.com/GuillaumeSeren/gentoo-overlay/blob/master/dev-php/composer/composer-1.2.1.ebuild) for a while, I think I could be great to have it merged into the main tree. I have also be running my overlay version, on several server and it is working well, I have open a PR on the gentoo tree here https://github.com/gentoo/gentoo/pull/2692 Let me know if anything need to be upgraded. Guillaume.
(In reply to Guillaume Seren from comment #13) > Hey there, > I maintained composer on my gentoo overlay > (https://github.com/GuillaumeSeren/gentoo-overlay/blob/master/dev-php/ > composer/composer-1.2.1.ebuild) for a while, > I think I could be great to have it merged into the main tree. Before this can go in the tree, it needs to have its dependencies unbundled (see my comment #12). The upstream releases still bundle everything, even if they try to hide that fact by zipping everything up into a phar archive. That leaves you with security vulnerabilities (bugs in libraries) that will never get fixed, and more plainly just defeats the purpose of the package manager. If you're going to be responsible for checking up on every dependency of composer yourself for all time, then you might as well just download the phar and run it -- having portage install it doesn't help you. To see what I mean, $ wget -q https://getcomposer.org/download/1.2.1/composer.phar $ php -r '$phar = new Phar("composer.phar"); $phar->extractTo("./");' $ ls vendor/ You will see a bunch of third-party libraries that were copy/pasted into the phar archive. They're also very difficult to unbundle because of the way composer works. Anything that uses composer is going to take a massive amount of work to get into the main tree. If you really want to take on the job, you'll have to mimic what Fedora and Debian do. You need to package all of its dependencies first, but doing so requires you to write an autoloader for those dependencies and to hack around the existing autoloaders that e.g. composer expects.
Thanks for the information. Perma-unstable or ** keyword, while being tree, to me sounds better than not being in tree.
(In reply to Michael Orlitzky from comment #14) > having portage install it doesn't help you. Since the subject was quite old (4 years) I expected it to be somehow different, but I understand and agree to your point. If we go this way we need to package all dependencies and also it will probably create a different (at least in term of hash) version of composer, so I think it is more a upstream problem than gentoo. To me having it in portage is convenient because, I don't need to let the composer.phar in my projects repositories, and also and I can upgrade it with my systems. > Anything that uses composer is going to take a massive > amount of work to get into the main tree. I agree but packaging every lib in https://packagist.org, seems not the best option, it is quite the same problem, with npm. I think it is a upstream problem in term of developer of the website and tools that are used by them to build it, yes there is a lot of security problem but if you can package the tools, it is easier for admin and dev to test / upgrade / maintain those applications. So I think instead of letting dev install any old/modified version of a tool is not good the best admin to provide it, that way it can be trusted a bit more. > If you really want to take on the > job, you'll have to mimic what Fedora and Debian do. You need to package all > of its dependencies first, but doing so requires you to write an autoloader > for those dependencies and to hack around the existing autoloaders that e.g. > composer expects. Look like a big chunk of work, mainly on packaging the dependencies and following them before each upgrade, but still if it *really* better I could package them as there are only 5 of them (console, filesystem, finder, polyfill-mbstring, process). I am wondering, what will be the benefits of such works because those libs wont likely be used by other packages ? Guillaume.
(In reply to Leho Kraav (:macmaN @lkraav) from comment #15) > Thanks for the information. > > Perma-unstable or ** keyword, while being tree, to me sounds better than not > being in tree. There's no formal policy against it, but you will be hard-pressed to find someone to commit a *new* package with known QA violations that will be masked forever =) (In reply to Guillaume Seren from comment #16) > > To me having it in portage is convenient because, I don't need to let the > composer.phar in my projects repositories, and also and I can upgrade it > with my systems. I sympathize with the convenience argument, but Gentoo is used by tens of thousands of people, most of whom are less technical than the people who e.g. read bugzilla. The things that go in the official ::gentoo tree are supposed to be safe for those people to use, and therefore need to meet a much higher bar for quality than overlay packages do. > > Anything that uses composer is going to take a massive > > amount of work to get into the main tree. > I agree but packaging every lib in https://packagist.org, seems not the > best option, it is quite the same problem, with npm. Indeed, it's worse. Even if you package the dependencies of something that uses composer, it won't work straight away. In addition, you need to undo the composer integration. But yes, it's a similar problem to why no npm packages are in the tree (upstream doesn't care). > I think it is a upstream problem in term of developer of the website and > tools that are used by them to build it, yes there is a lot of security > problem but if you can package the tools, > it is easier for admin and dev to test / upgrade / maintain those > applications. > So I think instead of letting dev install any old/modified version of a tool > is not good the best admin to provide it, that way it can be trusted a bit > more. I agree 100%... on our servers, I store global copies of these tools in /usr/local/bin, and manage them with our configuration management tool (e.g. puppet, chef, etc.). It's not quite as easy as it would be with portage, but it eliminates the duplication. > I am wondering, what will be the benefits of such works because those libs > wont likely be used by other packages ? It's a bit of a chicken-and-egg problem. There are other PHP applications that could use those libraries, but we always have this problem: the first application that needs some library requires you to package the library before the application. Once the libraries are there, the other applications that use them become easier to package. Eventually you get to a point where most of the common libraries are already packaged, and most of your work can be spent on the applications that you actually care about. In this case it's a little more annoying than normal, because those libraries require composer (and you first have to undo that), but nevertheless someone will still have to suck it up and package the libraries before we can package the apps that use them =)
I too agree that it would be better to add this to portage even not in perfect shape. Like binary packages, it would never go stable. Since our devs use this a lot, I could proxy maintain this.
(In reply to Michael Orlitzky from comment #17) > (In reply to Leho Kraav (:macmaN @lkraav) from comment #15) > > Thanks for the information. > > > > Perma-unstable or ** keyword, while being tree, to me sounds better than not > > being in tree. > > There's no formal policy against it, but you will be hard-pressed to find > someone to commit a *new* package with known QA violations that will be > masked forever =) > > > (In reply to Guillaume Seren from comment #16) > > > > To me having it in portage is convenient because, I don't need to let the > > composer.phar in my projects repositories, and also and I can upgrade it > > with my systems. > > I sympathize with the convenience argument, but Gentoo is used by tens of > thousands of people, most of whom are less technical than the people who > e.g. read bugzilla. The things that go in the official ::gentoo tree are > supposed to be safe for those people to use, and therefore need to meet a > much higher bar for quality than overlay packages do. > > > > > Anything that uses composer is going to take a massive > > > amount of work to get into the main tree. > > I agree but packaging every lib in https://packagist.org, seems not the > > best option, it is quite the same problem, with npm. > > Indeed, it's worse. Even if you package the dependencies of something that > uses composer, it won't work straight away. In addition, you need to undo > the composer integration. But yes, it's a similar problem to why no npm > packages are in the tree (upstream doesn't care). > > > > I think it is a upstream problem in term of developer of the website and > > tools that are used by them to build it, yes there is a lot of security > > problem but if you can package the tools, > > it is easier for admin and dev to test / upgrade / maintain those > > applications. > > So I think instead of letting dev install any old/modified version of a tool > > is not good the best admin to provide it, that way it can be trusted a bit > > more. > > I agree 100%... on our servers, I store global copies of these tools in > /usr/local/bin, and manage them with our configuration management tool (e.g. > puppet, chef, etc.). It's not quite as easy as it would be with portage, but > it eliminates the duplication. > > > > I am wondering, what will be the benefits of such works because those libs > > wont likely be used by other packages ? > > It's a bit of a chicken-and-egg problem. There are other PHP applications > that could use those libraries, but we always have this problem: the first > application that needs some library requires you to package the library > before the application. Once the libraries are there, the other applications > that use them become easier to package. Eventually you get to a point where > most of the common libraries are already packaged, and most of your work can > be spent on the applications that you actually care about. > > In this case it's a little more annoying than normal, because those > libraries require composer (and you first have to undo that), but > nevertheless someone will still have to suck it up and package the libraries > before we can package the apps that use them =)
First sorry for the double post, just a missclick. (In reply to Michael Orlitzky from comment #17) > > Perma-unstable or ** keyword, while being tree, to me sounds better than not > > being in tree. > There's no formal policy against it, but you will be hard-pressed to find > someone to commit a *new* package with known QA violations that will be > masked forever =) If it can fit to the main tree, even in unstable form I guess it still should be better than not having it at all and maintaining per-project versions. > (In reply to Guillaume Seren from comment #16) > I sympathize with the convenience argument, but Gentoo is used by tens of > thousands of people, most of whom are less technical than the people who > e.g. read bugzilla. The things that go in the official ::gentoo tree are > supposed to be safe for those people to use, and therefore need to meet a > much higher bar for quality than overlay packages do. Yes I understand that convenience is not the only point in the problem. > > I agree but packaging every lib in https://packagist.org, seems not the > > best option, it is quite the same problem, with npm. > Indeed, it's worse. Even if you package the dependencies of something that > uses composer, it won't work straight away. In addition, you need to undo > the composer integration. But yes, it's a similar problem to why no npm > packages are in the tree (upstream doesn't care). Yes it is also the same problem with pypy packages, cpan, gems, luarock, and probably lots I don't know, they are like user libs, and have there own packaging system, most of the time each user of a system can install want he want but I can distinct those that can be used for production system, like composer, npm, so for them it could be great to have them in the tree. > > So I think instead of letting dev install any old/modified version of a tool > > is not good the best admin to provide it, that way it can be trusted a bit > > more. > I agree 100%... on our servers, I store global copies of these tools in > /usr/local/bin, and manage them with our configuration management tool (e.g. > puppet, chef, etc.). It's not quite as easy as it would be with portage, but > it eliminates the duplication. Yes it is an option, that's why I have maintained that in my overlay, but I am pretty sure I am not the only one who need that on his production server :). > > I am wondering, what will be the benefits of such works because those libs > > wont likely be used by other packages ? > It's a bit of a chicken-and-egg problem. There are other PHP applications > that could use those libraries, but we always have this problem: the first > application that needs some library requires you to package the library > before the application. Once the libraries are there, the other applications > that use them become easier to package. Eventually you get to a point where > most of the common libraries are already packaged, and most of your work can > be spent on the applications that you actually care about. > > In this case it's a little more annoying than normal, because those > libraries require composer (and you first have to undo that), but > nevertheless someone will still have to suck it up and package the libraries > before we can package the apps that use them =) Yes I agree that those libs could end be used by some other php package, so it is better to package them *but* it also mean to generate a different phar file and that could introduce new bug(s) so maybe if we need that we should start a small fork to maintain it, and this problem occur to all others phar applications (php_codesniffer also for example). I didn't have the time to check the Debian/Fedora solution of that problem, but maybe we can package the phar application, as upstream provide it as unstable, then try to develop a solution for stable, like that dev can have the upstream solution and we can try to make a viable solution for stable in the second time. Honestly I like the 'package everything' solution, could be great and it can be a not so big task after all, but I do not really like forking (or creating a new version) the project to package it, maybe we can provide a more generic solution that still use the upstream phar *and* package the dependencies, we could patch the original phar and change the path to load the deps we packaged, not sure if it is possible, but this way we limit the change to original phar file to the minimum, what do you think ? Guillaume.
(In reply to Guillaume Seren from comment #20) > > Honestly I like the 'package everything' solution, could be great and it can > be a not so big task after all, > but I do not really like forking (or creating a new version) the project to > package it, > maybe we can provide a more generic solution that still use the upstream > phar *and* package the dependencies, > we could patch the original phar and change the path to load the deps we > packaged, not sure if it is possible, > but this way we limit the change to original phar file to the minimum, what > do you think ? The phar file is just a zip of the source code, so there's no need to work with it directly -- just work with the source. You can re-zip it afterwards if you want, but at that point, you might as well execute the textual PHP script rather than the "binary" phar. For example, we symlink the PHP_CodeSniffer scripts under /usr/bin and they execute normally. However, patching the source is still going to be very difficult, and will have to be done for every composer-using dependency in your dependency tree. I've collected some of these ideas on the wiki at https://wiki.gentoo.org/wiki/Project:PHP/The_Composer_problem and now we just need somebody to do the work =) I'm sorry I'm not being very cooperative in that regard, but I have 275207278 other things on my to-do list, and adding back a bunch of "require" statements to projects that have pointlessly removed them won't be near the top until it becomes life-threatening.
Hey thank you Michael for taking time to read my reply. (In reply to Michael Orlitzky from comment #21) > The phar file is just a zip of the source code, so there's no need to work > with it directly -- just work with the source. You can re-zip it afterwards > if you want, but at that point, you might as well execute the textual PHP > script rather than the "binary" phar. For example, we symlink the > PHP_CodeSniffer scripts under /usr/bin and they execute normally. I understand why they do it, in the first place to keep composer easy to install anywhere and I also understand the need for gentoo to stay down to sources (this is why I use gentoo, to be close to the sources and so to the upstream). > However, patching the source is still going to be very difficult, and will > have to be done for every composer-using dependency in your dependency tree. > I've collected some of these ideas on the wiki at > > https://wiki.gentoo.org/wiki/Project:PHP/The_Composer_problem I've finally found it and read it, it is helpful (thank you for writing it up), and also the fedora package itself: http://pkgs.fedoraproject.org/cgit/rpms/composer.git/ > and now we just need somebody to do the work =) Yes I am trying to play with it I agree on not using the phar as source, and start from the github look more clear, I will keep you in touch. > I'm sorry I'm not being very cooperative in that regard, but I have > 275207278 other things on my to-do list, and adding back a bunch of > "require" statements to projects that have pointlessly removed them won't be > near the top until it becomes life-threatening. Yes I understand that, and I am willing to put some time to do it. Guillaume.
So I have been doing some more tests 1 As you mention you can mimic the fedora way of packaging, and so use their autoloader (https://github.com/php-fedora/autoloader). 2 Most of libs on packagist did not provide autoloader so, it will be better to generate one using the autoloader, we still can generate / use specific autoloader if we really need. So the number of package for composer itself, will be defined by dependencies in the composer.json, like: "require": { "php": "^5.3.2 || ^7.0", "justinrainbow/json-schema": "^1.6 || ^2.0 || ^3.0 || ^4.0", "composer/ca-bundle": "^1.0", "composer/semver": "^1.0", "composer/spdx-licenses": "^1.0", "seld/jsonlint": "^1.4", "symfony/console": "^2.7 || ^3.0", "symfony/finder": "^2.7 || ^3.0", "symfony/process": "^2.7 || ^3.0", "symfony/filesystem": "^2.7 || ^3.0", "seld/phar-utils": "^1.0", "seld/cli-prompt": "^1.0", "psr/log": "^1.0" } After doing some tests, I think packaging the fedora/autoloader first has it will be used by all other libs, then finish with composer itself. Guillaume
(In reply to Guillaume Seren from comment #23) > So I have been doing some more tests > > 1 As you mention you can mimic the fedora way of packaging, and so use their > autoloader (https://github.com/php-fedora/autoloader). > Ah, this didn't exist a few months ago. It looks like they've factored out most of the custom code in their RPMs. Previously, each package had its own autoloader that was a lot more involved. Now, each package still has its own autoload.php, but that file just calls out to the Fedora autoloader. That should make things easier. I updated our wiki page to mention their autoloader, and to suggest that using it is probably the easiest thing to do. Fedora should already have packaged the dependencies of Composer, too. That means they'll already have written the autoload.php for each package. If god is merciful, then we should be able to just take that autoload.php, stick it in $FILESDIR, and then install it to /usr/share/php/, under a subdirectory corresponding to the package name (e.g. Composer/CaBundle). I think the directory structure is important, so we can copy Fedora's. Fedora uses the same PHP include path as us -- at least for now -- so everything should be looking in the right place.
Hey, (In reply to Michael Orlitzky from comment #24) > Ah, this didn't exist a few months ago. It looks like they've factored out > most of the custom code in their RPMs. Previously, each package had its own > autoloader that was a lot more involved. Now, each package still has its own > autoload.php, but that file just calls out to the Fedora autoloader. Yes I discovered that by reading their packages, and how they solved the solution, so I already packaged it and it works, look: https://github.com/GuillaumeSeren/gentoo-overlay/blob/master/dev-php/autoloader/autoloader-0.2.1.ebuild > That should make things easier. I updated our wiki page to mention their > autoloader, and to suggest that using it is probably the easiest thing to do. Ok cool we agree it is better to use that and give our feedback to them than making a competitive solution (if no real reason to). > Fedora should already have packaged the dependencies of Composer, too. That > means they'll already have written the autoload.php for each package. Yes all the package can be checked here: http://pkgs.fedoraproject.org/cgit/rpms/composer.git/tree/composer.spec > if god > is merciful, then we should be able to just take that autoload.php, stick it > in $FILESDIR, and then install it to /usr/share/php/, under a subdirectory > corresponding to the package name (e.g. Composer/CaBundle). Yes I think we install all the needed libs in /usr/share/php/libname. > I think the > directory structure is important, so we can copy Fedora's. Fedora uses the > same PHP include path as us -- at least for now -- so everything should be > looking in the right place. Yes
Hey there, after working on the *new* package for some time with the the feedback of Michael, and the fedora work, it is finally done ! You can try it from my overlay: https://github.com/GuillaumeSeren/gentoo-overlay And I opened a PR to the main gentoo tree on github: https://github.com/gentoo/gentoo/pull/2905 If you can try it, and report some feedback it could help ! Guillaume
Thanks for all the work,that's a huge PR. Great you made it real. Will test and report back.
Hi there, I have continue to work on composer and it is going well, we still searching for feedback and reviews, to be sure of the quality before merging it, please test and report if you can. https://github.com/gentoo/gentoo/pull/2905 Guillaume
Thanks to Guillaume, dev-php/composer is now in our main ::gentoo tree! Please give it a try.
Thanks Guillaume, Michael (and others), it works great. Thanks for bringing composer to Gentoo!
In one of my latest syncs i've seen composer popping up in portage and i wanted to say thank you, as well! It's great to have it in portage even if i dislike it with all my heart! To have your package management system be patched to be working from several Linux distributions says allot! PHP folks love it and that's the reason i need it from time to time. I haven't yet understand everything you've patched but i've seen this "Composer Problem" page gone in the Gentoo Wiki. Does this mean we can have latest PHP software added into portage too? Like Drush?
(In reply to zasire from comment #31) > > I haven't yet understand everything you've patched but i've seen this > "Composer Problem" page gone in the Gentoo Wiki. Does this mean we can have > latest PHP software added into portage too? Like Drush? I removed that page because now we have a way to do it: https://wiki.gentoo.org/wiki/Composer_packaging There aren't any insurmountable obstacles left to packaging something like Drush (which now uses Composer), it would merely require a lot of work to add every dependency of a dependency of a dependency... and to create autoloaders for each of them.
I see. Maybe you should put a link on this old wiki page or redirect it to this new one. I've seen it on quite a number of sites in recent month when searching for Gentoo and composer. But whatever you do i am happy there finally is a solution to this problem. Thanks allot for all the work!
(In reply to zasire from comment #33) > I see. Maybe you should put a link on this old wiki page or redirect it to > this new one. I've seen it on quite a number of sites in recent month when > searching for Gentoo and composer. It should redirect now. Some of the old information is there under the intro and "packaging hurdles," but all of my half-baked solution proposals have now been replaced with the one that actually works.
