Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications. Mesos can run Hadoop, Jenkins, Spark, Aurora, and other applications on a dynamically shared pool of nodes. Reproducible: Always
Created attachment 377452 [details] mesos-0.18.0.ebuild
>> # Copyright 1999-2006 Gentoo Foundation Please, fix year >> KEYWORDS="~alpha amd64 ~arm ~hppa ~ia64 ~mips ~ppc ~ppc64 ~s390 ~sh ~sparc ~x86 ~amd64-fbsd ~sparc-fbsd ~x86-fbsd" Have you tested it on all these arches? >> RDEPEND="" Not needed if empty The need of "${S}/build" looks strange for me, but I will look at package build system. )
(In reply to Jauhien Piatlicki from comment #2) >> Please, fix year ok >> Have you tested it on all these arches? No just amd64. >> Not needed if empty ok
(In reply to Axel Etcheverry from comment #3) > >> Have you tested it on all these arches? > > No just amd64. So your ebuild must have one keyword: ~amd64. Read this: http://devmanual.gentoo.org/keywording/
(In reply to Jauhien Piatlicki from comment #4) > (In reply to Axel Etcheverry from comment #3) > > > >> Have you tested it on all these arches? > > > > No just amd64. > > So your ebuild must have one keyword: ~amd64. Read this: > http://devmanual.gentoo.org/keywording/ ok thx ;)
Created attachment 377456 [details] mesos-0.18.0-r1.ebuild
>> S="${WORKDIR}/${P}" >> ECONF_SOURCE="${S}" Not needed >> src_prepare() { >> mkdir "${S}/build" || die >> } Not needed >> cd "${S}/build" Not needed >> src_install() { >> cd "${S}/build" >> emake DESTDIR="${D}" install || die "emake install failed" >> } Not needed Also, make sure to read http://wiki.gentoo.org/wiki/Project:Python/Eclasses and http://www.gentoo.org/proj/en/java/java-devel.xml as you have python and java stuff there. Have you tested, that parallel build fails? What's the reason?
Also I see lots of bundled 3rd party stuff, try to unbundle it. If it is not in Gentoo tree, create separate ebuilds.
Look at this configure option for example: --without-included-zookeeper excludes building and using the included ZooKeeper package in lieu of a system installed version (note, however, that no attempt is made to find the package and explicitly setting CPPFLAGS and LDFLAGS as appropriate may be necessary) And look at bug #318029
>> Also I see lots of bundled 3rd party stuff, try to unbundle it. If it is not in Gentoo tree, create separate ebuilds. ok how I do to extract the 3rd party? >> Have you tested, that parallel build fails? What's the reason? yes, exceeded memory... > x86_64-pc-linux-gnu-g++: internal compiler error: Killed (program cc1plus) > Please submit a full bug report, > with preprocessed source if appropriate. > See <http://bugs.gentoo.org/> for instructions. > make[2]: *** [master/libmesos_no_3rdparty_la-http.lo] Error 1 I change my memory (1GB to 4GB) for fix this bug (I'm on vagrant VM) >>>> S="${WORKDIR}/${P}" >>>> ECONF_SOURCE="${S}" >> >> Not needed >> >> src_prepare() { >> mkdir "${S}/build" || die >> } >> >> Not needed >> >>>> cd "${S}/build" >> >> Not needed >>>> src_install() { >>>> cd "${S}/build" >>>> emake DESTDIR="${D}" install || die "emake install failed" >>>> } >> >> Not needed I followed the official documentation, it's not a problem if I run the ./configure and make in the ${S} directory?
(In reply to Axel Etcheverry from comment #10) > > ok how I do to extract the 3rd party? > Look at 3rdparty folder and look how build system of your package uses this 3rd party software. E.g. for zookeeper you have a special option that allows you to use the system-wide one. > > I followed the official documentation, it's not a problem if I run the > ./configure and make in the ${S} directory? If it still compiles and installs it's ok. Just test it. If you'll need out of source build, have a look at https://devmanual.gentoo.org/eclass-reference/autotools-utils.eclass/index.html I think it should be ok with building in the source, but you should test it.
(In reply to Jauhien Piatlicki from comment #11) > (In reply to Axel Etcheverry from comment #10) > > > > > ok how I do to extract the 3rd party? > > > > Look at 3rdparty folder and look how build system of your package uses this > 3rd party software. E.g. for zookeeper you have a special option that allows > you to use the system-wide one. > It's really difficult to extract the 3rd party.
When you'll have ebuild less or more prepared, you can ask for review on sunrise irc channel: #gentoo-sunrise. Also you can ask for commit access to the sunrise overlay: https://wiki.gentoo.org/wiki/Project:Sunrise/How_to_commit If you do not like sunrise overlay, I still recommend you ask there for review, then I can help you with proxy-maintainment: https://wiki.gentoo.org/wiki/Project:Proxy_Maintainers
Any progress with the packages?
OK, so I hacked an ebuild for mesos-0.20.0 not unlike the previous version. It needs to be tested: # Copyright 1999-2014 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 # $Header: $ ## hack by James aug 2014 EAPI=5 inherit autotools eutils flag-o-matic multilib MY_PV=${PV/_/} DESCRIPTION="fast cluster manager for distributed applications" HOMEPAGE="http://mesos.apache.org/" SRC_URI="http://www.apache.org/dist/mesos/${PV}/${P}.tar.gz http://mir2.ovh.net/ftp.apache.org/dist/mesos/${PV}/${P}.tar.gz" LICENSE="Apache-2.0" KEYWORDS="~amd64" IUSE="java python" SLOT="0" DEPEND="net-misc/curl dev-libs/cyrus-sasl python? ( dev-lang/python dev-python/boto ) java? ( virtual/jdk ) dev-java/maven-bin " RDEPEND="python? ( dev-lang/python ) >=virtual/jdk-1.6 dev-java/maven-bin ${DEPEND}" S="${WORKDIR}/${P}" ECONF_SOURCE="${S}" src_prepare() { mkdir "${S}/build" || die } src_configure() { cd "${S}/build" econf \ $(use_enable python) \ $(use_enable java) } src_compile() { cd "${S}/build" emake -j1 V=1 } src_install() { cd "${S}/build" emake DESTDIR="${D}" install || die "emake install failed" }
Created attachment 385282 [details] mesos-0.20.0.ebuild Apache-mesos
Created attachment 385316 [details] mesos-0.20.0.ebuil fixed download source Feedback from testers is most welcome.
Thank you James for your ebuild, I test it and I will make a feedback.
(In reply to James Horton from comment #17) > Created attachment 385316 [details] > mesos-0.20.0.ebuil fixed download source > > Feedback from testers is most welcome. Thx for your ebuild, I tested on my Gentoo amd64 it's ok
Is there any outstanding actions on this bug that need any assistance? I see we have an ebuild and a couple of tests. Are we looking to add this to the tree or anything?
Hello, I'm new to ebuild writing, but this mesos one looked easy. It installs on my system. Further testing would be keen. I'll do the version bump is a week or so. I'm in the cc list, but I am not getting the notices when comments are posted to this bug? I checked my junk make folder and nothing in there either. Strange I get many cc notices from other bugs. James
(In reply to Alex Brandt from comment #20) > Is there any outstanding actions on this bug that need any assistance? I > see we have an ebuild and a couple of tests. Are we looking to add this to > the tree or anything? It looks like the main "problem", as pointed out by Jauhien, is that upstream bundles 3rd party libs which are used by the ebuild and this violates QA. TBH I can't see this ebuild going live in tree until nothing has been tried to get rid of those 3rd party libs. Not to mention any possible static lib that would come out of this build (didn't check about that).
> It looks like the main "problem", as pointed out by Jauhien, is that upstream > bundles 3rd party libs which are used by the ebuild and this violates QA. > TBH I can't see this ebuild going live in tree until nothing has been tried to > get rid of those 3rd party libs. Not to mention any possible static lib that > > would come out of this build (didn't check about that). Um, I'm a wee bit confused with this. Looking at the mesos ebuild I only see these as the possible offenders: dev-libs/cyrus-sasl java? ( virtual/jdk ) dev-java/maven-bin I used icedtea for the openjdk on my system. I can replace maven-bin with maven dev-libs/cyrus-sasl is in the tree. So is this what you are stating as problems? Can you be explicit or tell me how to run tools to determine which libs or codes offend the QA? I'm new to putting ebuild together. OK? James
(In reply to James Horton from comment #23) > > It looks like the main "problem", as pointed out by Jauhien, is that upstream > bundles 3rd party libs which are used by the ebuild and this violates QA. > > Um, I'm a wee bit confused with this. Looking at the mesos ebuild I only > see these as the possible offenders: Ultrabug was talking about the 3rdparty directory in the mesos code: https://github.com/apache/mesos/tree/master/3rdparty This is a collection of things that need to potentially be patched out, dropped from the source tree, and linked and installed from ebuilds.
This is what that link shows: libprocess Makefile.am distribute-0.6.26.tar.gz Renamed 'third_party' to '3rdparty'. leveldb.patch leveldb.tar.gz pip-1.5.6.tar.gz versions.am Install Python libraries during make install. wheel-0.24.0.tar.gz Install Python libraries during make install. zookeeper-3.4.5.patch zookeeper-3.4.5.tar.gz leveldb, app-misc/pip, Makefile.am, versions.am and wheel (python libs) are all in portage. So just change the mesos ebuild to use what is in the portage tree, versus a raw download from the net? That would leave zookeeper, libprocess as the only remaining issues? zookeeper is in ultrabug's reop: sys-cluster/zookeeper [1] Available versions: ~3.4.6^mb [1] "ultrabug" /var/lib/layman/ultrabug libprocess just needs to be remove as it is distro centric. Is this the sort of actions necessary? Any you are saying this can all be achieved via extra code in the mesos.0.20.0.ebuild? Comments and guidance are most appreciated. (I'm still not getting updates despite getting other BGO bug updates ; automatically). James
(In reply to James Horton from comment #25) > leveldb, app-misc/pip, Makefile.am, versions.am and wheel (python libs) are > all in portage. So just change the mesos ebuild to use what is in the > portage tree, versus a raw > download from the net? Not quite that easy. You need to update the build system to not try to link against the built-in versions in that directory. This typically requires a patch (that should be sent upstream if possible). Once you can delete that directory as part of src_prepare() and it still builds, you're probably on the right track. > zookeeper is in ultrabug's reop: > sys-cluster/zookeeper [1] > Available versions: ~3.4.6^mb > [1] "ultrabug" /var/lib/layman/ultrabug Before mesos can be added to the tree this ebuild would need to be added to the tree as well. > libprocess just needs to be remove as it is distro centric. Can you prove that with a successful build and execution? > Is this the sort of actions necessary? Any you are saying this can all be > achieved via extra code in the mesos.0.20.0.ebuild? Some, but I highly suspect that a patch will be required as well. Feel free to ask for guidance in email, IRC, or check devmanual (http://devmanual.gentoo.org/). This bug is probably not the best forum to discuss how to write ebuilds and we should continue this in another medium. Hope that gets you started and feel free to ask about anything that's unclear.
Created attachment 399778 [details] mesos.0.22.0.ebuild Fixed many problems. zookeeper is still not in the portage tree. An upstream patch needs to be filed. I have not done this before so some guidance is needed. Please test ebuild.
Need guidance on upstream patch filing for libprocess.
looking at last ebuild (mesos-0.22.0.ebuild) the job of de-bundling is far away of being done. Actually Fedora guys are advancing in that regard, so if you are familiar with rpm spec files you can take a look at: http://pkgs.fedoraproject.org/cgit/mesos.git/tree/ I guess I have to check this soon (I don't know how soon yet) as maybe I will need to prepare custom CoreOS image which includes mesos+marathon+chronos.
Created attachment 412652 [details] mesos-0.24.0.ebuild The apache-mesos project has a funny way of not publishing candiate tarballs, with correct minor and revision numbers. I'm working to sort this out. This ebuild compiles just fine. cd /usr/bin/ and run 'ls mesos*' to see the scripts and such. I'm going to try to pre configure the essential things needed to get mesos set up in the ebuild. What flags to support and some basic documentation on the final configuration options is on going work that could use suggestions. For now you'll have to look at the generic documentation on configuring mesos. Ultimately, optional HDFS support will be a flag option. Many are interested in running mesos on top of btrfs and cephfs as the distributed file system. James
I think mesos-1.0 has been organized such that those 3rd party builds are no longer a problem. Everything previously listed seems to be in the protage tree now:: CLEAN_EXTRACTED = $(BOOST) $(GLOG) $(GMOCK) $(GPERFTOOLS) $(HTTP_PARSER) $(LIBEV) $(PICOJSON) $(PROTOBUF) Verification from someone else would be appreciated. https://github.com/apache/mesos/blob/master/3rdparty/libprocess/3rdparty/Makefile.am#L70 Some guidance before I attempt to update the mesos ebuild would be appreciated. James
Easy way to confirm would be to ensure the bundled flag is disabled during configure and then run the ebuild and see if mesos works when it's finished. Seems like mesos has an easy way to disable the bundled dependencies and use what's in the environment already baked in. It shouldn't be too bad to get this working as an accepted ebuild (without bundling). Let us know if you have any specific questions.
@alex:: Huh. I like what you are saying. I cannot find a bundled flag, nor does it make sense. OK, no problem on this coarse of action. Perhaps it can be explicitly pointed out to me? I presume the bundled flag was missed by me as the ebuild I put together, mesos-0.24.0.ebuild has only (2) flags:: java and python. What do I need to look at, to enhance my comprehension into an actionable state? James
@james, to start out, manually extract the source and run ./configure --help inside of it and studying the output. Then try to understand the existing ebuilds by looking at the devmanual. (If you haven't done so already, start by reading https://devmanual.gentoo.org/quickstart/index.html .)
