Jump to content

NEWS
  • WELCOME TO CRYPTOCURRENCYTALK!
  • We've upgraded the website with a lot of new features!
  • 3 NEW THEMES! Click on the + Themes button above!
  • Notice about SOCIAL LINKS: UPDATE YOUR SOCIAL LINKS
  • New video section, add your videos: VIDEO SECTION
  • Advertising will be available shortly, hold tight.
  • If you have any issues with the new site, please submit a support ticket: SUPPORT
C.M

[Group Research] Build Automation & Package support

What distros need immediate support?  

13 members have voted

  1. 1. What distros need immediate support?

    • Ubuntu
      8
    • Fedora/RHEL/CentOS
      4
    • *.deb debian (non-distro specific)
      9
    • *.rpm (non-distro specific)
      5
    • Other (Slackware, Docker, etc).
      2
    • Chocolatey (Windows)
      0
    • *BSD
      1
  2. 2. What distros have you successfully compiled gridcoin research on? (Not packages)

    • Ubuntu/*Buntu
      9
    • Fedora/RHEL/CentOS
      0
    • *BSD
      0
    • Slackware
      1
    • Debian
      3
    • Docker
      0


Recommended Posts

Hey guys,

 

I think it's totally worth some investigation into automating the build process and package creation for multiple operating systems & their respective repositories.

 

Creating packages and hosting them on an official repo would make updating on linux as easy as adding the official gridcoin repo (a single line addition to a file), then typing either 'apt-get upgrade' or 'yum -y upgrade', etc. If node maintainers schedule this command, we could always have the latest clients running on the network with minimal effort.

 

I believe there's two routes for deploying packages - either we create our own repo and have it hosted on *.gridcoin.us, or we submit the packages to official 3rd party software repositories for *.deb and *.rpm - I've listed a few of these below.

 

Ubuntu/Debian based (*.deb):

Repo how to:

https://help.ubuntu....ty/AptGet/Howto

https://help.ubuntu....tories/Personal

https://help.ubuntu....sitories/Ubuntu

http://packaging.ubuntu.com/html/

 

Build automation:

http://askubuntu.com/questions/161155/how-to-create-a-ppa-for-c-program/161212#161212

http://askubuntu.com/questions/52475/how-to-prepare-auto-updating-ppa/52478#52478

 

Repos:

https://launchpad.net/

https://launchpad.net/gridcoin(Not 100% verified as working yet)

 

Fedora/RHEL based (*.rpm):

Repo how to:

http://yum.baseurl.org/

http://www.unixmen.c...y-in-fedora-19/

https://fedoraproject.org/wiki/Join_the_package_collection_maintainers

 

Build automation:

https://fedoraproject.org/wiki/Koji

 

Repos:

http://rpmfusion.org/

http://freshrpms.net/

http://rpm.livna.org/

 

OpenSuse (*.rpm & non-os specific):

https://build.opensuse.org/

https://en.opensuse.org//openSUSE:Build_Service

 

 

Windows:

https://chocolatey.org/

https://github.com/c.../CreatePackages

https://github.com/chocolatey/chocolatey/wiki/AutomaticPackages

http://coapp.org/index.html

 

Other:

https://www.docker.com/

http://docs.docker.com/docker-hub/builds/

https://coreos.com/blog/rocket/

 

I'd be up for creating new install guides on the website for all of the above, and will help investigate the required processes above.

 

IMO, priority is getting the .deb and .rpm support sorted out, as windows already has compiled msi's and a working embedded updater & docker/rocket are just for a fun side project.

 

Does anyone have any experience in the above, or know of extra reading material that should be included above?

 

Regards,

CM.

Edited by C.M

^ Smash that upvote button! ;D

Follow me on:

Github   Twitter   Steemit   SoundCloud (Hangouts)

 

Share this post


Link to post
Share on other sites

Kia Ora CM

I'm well down the track exploring the PPA route and have been PMing to Rob about it. I'm on holiday on the boat for another 7 days, then looking forward to compiling my first few binaries on Launchpad.

I had asked Rob who might be interested in forming "team gridcoin" to manage the repo. I'm keen.

Cellphone coverage for me is sporadic, but happy to talk about it.

Share this post


Link to post
Share on other sites

Cross-posted the following posts from the announcement thread for easier tracking of this discussion:

 

 

TyphooN, on 28 Dec 2014 - 3:37 PM, said:

I don't think it would take much effort to setup something to auto compile gridcoin via cron every x minutes for .deb and .rpm (only when new code is downloaded) and then as scp them to a hosted location.  I can volunteer to do this if I can be given a location to scp the files.  I wouldn't have the time today to set this up, but some time in the near future if hosting is provided.  .deb and .rpm will cover a lot of popular distros, but more formats or repos could be added over time.

 

Edit: As far as setting up yum/apt repos goes.. that also doesn't require much effort and I already have experience with both.

Thanks for volunteering to do that; I realize having someone do a regular build of linux is more realistic since we have a lot of demand and support on linux compared to mac.  Where this would really help us is for the plain vanilla average linux user.  If we can still set this up, they will not have to ask for a key to run a linux node.  I think we can set up an escalation system where I give the key to power users only or exchanges, and someone like Typhoon (or whoever ultimately maintains the build script) can make a linux distro for the main end user base.

 

Rob

 

 

Rob Halförd - (Gridcoin), on 28 Dec 2014 - 3:30 PM, said:

I am considering RPMs and distros, but given the track record of not one person successfully volunteering to compile linux on a Mac more than once per year, I dont think creating 4 binaries daily is actually realistic.  Until I actually see the results appearing and building and md5'ing and linking and running.  (In contrast, SeP and Sec have been building daily).

 

Next, we have to have to a mechanism in the coin to allow exchanges to build the coin and check in the source and make it opensource anyway; otherwise exchanges will not run it; so this step is pretty much necessary either way.  What you are pointing out is the difference between me hiding future code releases from everyone and not hiding the source (as checking it in reveals the source that the config manager will be using to build the rpms every night).

 

Anyway, my main point is, in order for gridcoin to succeed I feel the last resort will have to be binary distros as people will always wonder whats in the code, and with binary distros you do have to support a much wider range of platforms than 3 - its more like 2 compiled versions per flavor; more like 15 distros (thats why most of these ftp sites have a full page of binaries).  If we miss a distro, the user cant run the coin at all.

 

So for this step Im homogenizing the key requirement for exchanges or end linux users down to an organization name + public key.  If the key is abused or compromised we can ban it in the next release.  If the whole system is compromised we will either have to fix the system or move to the distros.  But my effort now is to keep the source open at this point.

 

Rob

 

I like the idea, but I also understand Rob's very true observation.

 

Yes I know it can be automated, I have package repos for employees and myself, I also oversee the management of others internal ones for companies I have contracts with.

The point is that you need to decide what the minimum common denominator you are willing to support (which is the least amount of distros) and then create Package Dev machine (virtual or not) that are pure clean vanilla, but also fully updated with all the dev tools needed and keep them that way.

Need to know how to properly clean build and package works.

Be willing to report, support and fix any problem in the various steps, keeping good proper practice.

Most can be scripted, but still need some maintenance and attention... basically knowledge and time or money.

 

If we can maybe get a dedicated serious and reliable package maintainer we could give it a shot.

 

also I don't think we need the windows packaging system Chocolatey, since windows is already packaged with the installer; I would vote as you for the .deb and .rpm, and maybe .tgz.

The only thing is the choice of distro to support: I would go first on the serious users first, therefore CentOS and straight Debian maybe CoreOS since those are the one that are the most used on headless servers and clouds by pros and then Slackware, Ubuntu, Fedora, OpenSUSE for the second tiers.

 

I am trying to make it easy and still appeal to the most and disgruntle none: CenOS is basically RedHat Enterprise without the price and strings, which is where all the .rpm distros came from (fedora is only the free community edition of the RedHat home/workstation so no really server material); Debian on the other side of the server spectrum is the serious choice, where all the other .deb distros come from, including Ubuntu, not the other way around... even if Ubuntu LTS Server could be considered, it has less of an professional user help base to go to, compared to Debian, if there are any problems.

 

For .tgz you then could add Slackware (yes probably very publicized, but many very very important servers run on it, when you need ultimate flexibility and uninterrupted up time in the 500+ days, that is where you go, like... guess what... world DNSes, infrastructure, routing, firewall, etc.)... Slackware is one of the very first and the best distro... but you really need to know what you are doing and where... Basically that is if you are really serious and an hardcore Unix admin... so with no offense,  I would leave Slackware last, when we get the other two working.

 
P.S.: I also vote to keep the source open at the same time... the packaging is only as a service to make it easy for the linux users community, maybe a marketing pitch point and once trusted, for the exchanges; Especially with hash and signature... I would put mine if package is built and system is maintained by me.

 

CM's reply to the above post

 

You're correct that even after working on an automated process for building and deploying packages, there'll need to be several dedicated maintainers to oversee the process. We've already seemingly got a dedicated userbase who host the gridcoin client on vps nodes - since we're talking about reducing the amount of repetitive work, we could all investigate this and work it all out together?

 

I only added chocolatey becase i found it the other day (thought it was kinda neat), and the opensuse links were for a package builder that works across multiple operating systems. Do you think it'd be easy to write a powershell script to do a scheduled clean update of the gridcoin.msi for windows?

 

You're correct that we should prioritize .deb and .rpm - we'll hit the majority of users with such support.

 

Perhaps it's worth a quick community vote/audit regarding what the most popular operating systems are with gridcoin users (does this data exist?).

 

Regarding fedora, it was the case that it was home/workstation in fedora 20 & below, but they've moved to a workstation/server/cloud format with fedora 21 (https://getfedora.org/) - they're trying their hand at docker virtualization.

 

I'm very interested in docker containers - there's some fierce competition heating up between https://coreos.com/and http://www.projectatomic.io/ - I'm considering trying to get project atomic + boinc/gridcoin running for maximum safety when running the work units from BOINC projects, but that's way down the road - .deb & .rpm for sure priority.

Edited by C.M

^ Smash that upvote button! ;D

Follow me on:

Github   Twitter   Steemit   SoundCloud (Hangouts)

 

Share this post


Link to post
Share on other sites

Kia Ora CM

I'm well down the track exploring the PPA route and have been PMing to Rob about it. I'm on holiday on the boat for another 7 days, then looking forward to compiling my first few binaries on Launchpad.

I had asked Rob who might be interested in forming "team gridcoin" to manage the repo. I'm keen.

Cellphone coverage for me is sporadic, but happy to talk about it.

Hey,

 

Spelucher has taken the initiative to create a launchpad entry for gridcoin:

https://cryptocointalk.com/topic/1331-new-coin-launch-announcement-grc-gridcoin/page-604#entry157535

https://launchpad.net/gridcoin

 

We need to verify that it's all setup correctly, and that an user can install the ppa via terminal & update successfully.

 

We need to now work on a recipie for launchpad to create the ppa correctly - this includes things like using the correct boost version and whether or not we include flags for creating the QT (gui) version.

https://help.launchpad.net/Packaging/PPA/BuildingASourcePackage(good info!).

https://help.launchpad.net/Packaging/SourceBuilds/GettingStarted

https://help.launchpad.net/Packaging/SourceBuilds/Recipes

https://help.launchpad.net/Packaging/SourceBuilds/BzrBuilder

 

http://blog.launchpad.net/cool-new-stuff/source-package-recipes

http://gridcoin.us/Dev/compilelinux.htm

 

Something we need to check is how gridcoin will handle being updated via this route - will the user need to remove some appdata files (peer.dat/blockchain/etc?) or will it be a clean update?

 

Launchpad covers x64 & x86 for ubuntu, however we still need to cover .deb & .rpm for maximum distro coverage.

 

Regards, CM.

Edited by C.M

^ Smash that upvote button! ;D

Follow me on:

Github   Twitter   Steemit   SoundCloud (Hangouts)

 

Share this post


Link to post
Share on other sites

I have some experience building packages on the Open Build Service. However last time I tried to create a package for Gridcoin I gave up because I found it ridiculously hard to handle the version number, which is currently not tagged on Github. If someone has a cool idea how to do it I could provide RPMs.

 

Update: If you want to view my current PoC for RPMs, see https://build.opensuse.org/package/show/home:theMarix/gridcoin.

Edited by Marix

Packaged Gridcoin for openSUSE and Fedora: http://software.opensuse.org/search?q=gridcoin&search_devel=false&search_unsupported=true

Like my packages? Why not send some mBTC to 19GmPiHt3irPJpRCndt9uWujYmxFA13buU or some GRC to SGZTxq5K78KgW1msFWZJr2QjYHynhiy58p.

Share this post


Link to post
Share on other sites

I have some experience building packages on the Open Build Service. However last time I tried to create a package for Gridcoin I gave up because I found it ridiculously hard to handle the version number, which is currently not tagged on Github. If someone has a cool idea how to do it I could provide RPMs.

 

Update: If you want to view my current PoC for RPMs, see https://build.opensuse.org/package/show/home:theMarix/gridcoin.

Whilst the version numbering doesn't follow a 100% formulaic pattern, this link has the commits (which each have a version number):

https://github.com/gridcoin/Gridcoin-Research/commits/master

 

Is it possible to try to check for a new commit, and build a package for it rather than looking for a new version number? Since version numbering in the past has included letters and special characters.

 

Have you managed to compile gridcoin on opensuse/*.rpm distro successfully before?

 

I've only attempted to compile the classic gridcoin on ubuntu in the past - maybe it'd be worth attempting to compile it on a few different distros to see if there's any missing compile steps for different distros... I'll have a look at this in the next couple days.

 

Edit: Added a poll question regarding what distros/OSes people have successfully compiled grc research on in the past.

Edited by C.M

^ Smash that upvote button! ;D

Follow me on:

Github   Twitter   Steemit   SoundCloud (Hangouts)

 

Share this post


Link to post
Share on other sites

Whilst the version numbering doesn't follow a 100% formulaic pattern, this link has the commits (which each have a version number):

https://github.com/gridcoin/Gridcoin-Research/commits/master

 

Is it possible to try to check for a new commit, and build a package for it rather than looking for a new version number? Since version numbering in the past has included letters and special characters.

 

Have you managed to compile gridcoin on opensuse/*.rpm distro successfully before?

 

I've only attempted to compile the classic gridcoin on ubuntu in the past - maybe it'd be worth attempting to compile it on a few different distros to see if there's any missing compile steps for different distros... I'll have a look at this in the next couple days.

 

Edit: Added a poll question regarding what distros/OSes people have successfully compiled grc research on in the past.

 

Yes, I have successfully built on openSUSE before. I also have a working set-up to build the RPM, it's just inconvenient currently, as I manually have to adjust the commit-hash and version number for every update.


Packaged Gridcoin for openSUSE and Fedora: http://software.opensuse.org/search?q=gridcoin&search_devel=false&search_unsupported=true

Like my packages? Why not send some mBTC to 19GmPiHt3irPJpRCndt9uWujYmxFA13buU or some GRC to SGZTxq5K78KgW1msFWZJr2QjYHynhiy58p.

Share this post


Link to post
Share on other sites

Yes, I have successfully built on openSUSE before. I also have a working set-up to build the RPM, it's just inconvenient currently, as I manually have to adjust the commit-hash and version number for every update.

Right, there should totally be the ability to grab the latest github commit via an api that github provides - i'll help look into this. I'll edit this post with my findings.


^ Smash that upvote button! ;D

Follow me on:

Github   Twitter   Steemit   SoundCloud (Hangouts)

 

Share this post


Link to post
Share on other sites

The problem is less getting the latest commit from Github. This can be done using so called services of the open build service. Either file download service or VCS service work fine for this. My problem is more to automatically figure out the version number to add to the RPM in that care. There seems to be a version extraction service, but I have not found sufficient documentation to be able to make it parse the version from clientversion.h. I guess I will have to look at the source as soon as I find some spare time.


Packaged Gridcoin for openSUSE and Fedora: http://software.opensuse.org/search?q=gridcoin&search_devel=false&search_unsupported=true

Like my packages? Why not send some mBTC to 19GmPiHt3irPJpRCndt9uWujYmxFA13buU or some GRC to SGZTxq5K78KgW1msFWZJr2QjYHynhiy58p.

Share this post


Link to post
Share on other sites

Sorry, accidentally double posted because of a bass connection.

Edited by Marix

Packaged Gridcoin for openSUSE and Fedora: http://software.opensuse.org/search?q=gridcoin&search_devel=false&search_unsupported=true

Like my packages? Why not send some mBTC to 19GmPiHt3irPJpRCndt9uWujYmxFA13buU or some GRC to SGZTxq5K78KgW1msFWZJr2QjYHynhiy58p.

Share this post


Link to post
Share on other sites

The problem is less getting the latest commit from Github. This can be done using so called services of the open build service. Either file download service or VCS service work fine for this. My problem is more to automatically figure out the version number to add to the RPM in that care. There seems to be a version extraction service, but I have not found sufficient documentation to be able to make it parse the version from clientversion.h. I guess I will have to look at the source as soon as I find some spare time.

Yo,

 

Checked out clientversion.h: https://github.com/gridcoin/Gridcoin-Research/blob/194370fdfdf74fb5c342709c66d7a2988fd885b0/src/clientversion.h

I'd say there's nothing we could work off of in that file to deduce the current commit version.

 

File 'gridcoinstake.pro' https://github.com/gridcoin/Gridcoin-Research/blob/d897c1cc58a1e9563e993136437b2cfd8c168990/gridcoinstake.pro

This file does have a client version, but it's not up to date. File states the version is '3.1.0.1' and the latest commit version is '3.3.2.2' (https://github.com/gridcoin/Gridcoin-Research/commits/master).

 

So I mean, maybe we could extract the latest commit version from the gridcoinstake.pro for a version number, but IMO, it'd probably be easier to use the github api to grab the latest commit version (https://github.com/gridcoin/Gridcoin-Research/commits/master)...


^ Smash that upvote button! ;D

Follow me on:

Github   Twitter   Steemit   SoundCloud (Hangouts)

 

Share this post


Link to post
Share on other sites

The version is completely available in client version.h. It is spread over four variables.

 

The problem with git commit IDs is that these are not numerically increasing and this not s suited for RPM package versioning.


Packaged Gridcoin for openSUSE and Fedora: http://software.opensuse.org/search?q=gridcoin&search_devel=false&search_unsupported=true

Like my packages? Why not send some mBTC to 19GmPiHt3irPJpRCndt9uWujYmxFA13buU or some GRC to SGZTxq5K78KgW1msFWZJr2QjYHynhiy58p.

Share this post


Link to post
Share on other sites

 

The version is completely available in client version.h. It is spread over four variables.

 

The problem with git commit IDs is that these are not numerically increasing and this not s suited for RPM package versioning.

 

 

Ah, gotcha now.

I'll help look into this.. there's got to be some kind of automatic way of doing this.

Any chance of posting some links for more info? : )


^ Smash that upvote button! ;D

Follow me on:

Github   Twitter   Steemit   SoundCloud (Hangouts)

 

Share this post


Link to post
Share on other sites

Looked at the source and it seems the version extraction features are too limited. Have got the package tidied up now, though. Once the release rate decreases the manual version parsing should only be a minor inconvenience and the current set-up should be quite feasible.


Packaged Gridcoin for openSUSE and Fedora: http://software.opensuse.org/search?q=gridcoin&search_devel=false&search_unsupported=true

Like my packages? Why not send some mBTC to 19GmPiHt3irPJpRCndt9uWujYmxFA13buU or some GRC to SGZTxq5K78KgW1msFWZJr2QjYHynhiy58p.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


  • Recently Browsing   0 members

    No registered users viewing this page.

×

Important Information

By using CRYPTOCURRENCYTALK.COM, you agree to our Terms of Use.