RPM builds and version control

[Elf]

@hammy was soliciting input on how to version RPM builds, given that a version of a build:

• Uses a specific version of source that originates elsewhere (usually in tarball form)
• Uses RPM spec files that originate elsewhere, but applies modifications to them to result in the final spec file to be used
• Generates an SRPM that has everything needed to perform the build, but is too big to fit in version control
• Finally, generates a binary RPM that is installable by the end user

Hopefully @hammy can let me know if I have misunderstood the problem in any way? To my understanding, the basic conundrum was how to manage versioning and version control of these things, given that some are too large to fit in git (or not necessarily appropriate for git).

My observation is that a lot of these fit into the category of binary artifacts. Namely the original source tarballs, the resultant SRPMs, and the resultant binary RPMs. They have independent but related versions, namely an upstream source revision and a build revision.

I would suggest a scheme for dealing with them as follows:

1. All binary artifacts (original source tarballs, built SRPMs, binary RPMs) are versioned and stored in a separate artifact repository and not in source control
   • The artifact repository can be the SGUG S3 bucket (with CloudFront CDN distribution), which should be fast, exceptionally durable, and easy to mirror for anyone interested
   • Upstream packages:
     • Should still be stored (cached) in our artifact repository just in case they disappear from upstream later
     • Retain their original upstream version number
   • SRPMs and binary RPMs:
     • Treated the same way as each other even though the SRPMs may be used as an intermediate product in a build process
     • Versioned according to the upstream version plus a build version (and distinguished by target IRIX version / architecture, etc.)
     • Stored in a yum repository compatible format
2. Build versioning (for SRPMs/RPMs)
   • Distinguish between snapshot and release with a suffix
     • Snapshot: a development build between official releases, not meant to be stable or relied on by users other than in a testing capacity
     • Release: a finalized release (whatever that means to someone, but ideally in a deliverable state, stable, and tested)
     • Store them in different yum repository roots so that the users won't accidentally install snapshot versions?
   • Use date / time stamp versioning of snapshot artifacts to avoid having to synchronize version numbers with multiple developers (e.g. a Major.Minor.YYYYMMDDHHMMSS format in UTC)
   • Use major / minor / incremental versioning of releases, as releases should be coordinated by all involved developers
3. Build process
   • I think the key part of the above with regards to not maintaining SRPMs in version control is that everything necessary to create the SRPM, and to proceed thereon to a binary RPM, should be version controlled
   • Spec files should either be version controlled directly, or alternatively a set of patch files used to create the spec file along with a script to fetch the spec file and make that happen as part of an automated build process
   • The build scripts themselves should be versioned
   • The build scripts should perform some semblance of:
     • Downloading the upstream sources from the artifact repository
     • Using (or creating and using) the spec file to generate the SRPM
     • Executing the build process of turning an SRPM into a binary RPM
     • Optionally (if credentials are present) uploading the resultant SRPMs and binary RPMs to the destination yum repository, although ideally in an end state this would be handled only by a CI system

Given the above process, there is no need to keep any SRPMs in source control given that the mechanism for generating (and then using) the SRPMs is source controlled and should be fully deterministic. It should be noted that this is similar to how many other modern development lifecycles work with external dependencies (e.g. libraries in a Maven repository for a Java project).

Again let me know if this addresses the original concern @hammy; or anyone else is of course free to comment as well.

 

[massiverobot]

Regarding storage of the SRPMS and S3- it will be another bootstrap process to get python working, then some aws_cli tooling and then the ability for our system to manage artifacts in a S3 system. I wonder if trying to use a system like Nexus first may be easier. Not sure, but it is something i'll investigate. Ultimately I'd def want things in S3 but not sure how it would really work from within IRIX.

 

[Elf]

It depends where one moves that management of the artifact repository. Conceivably a simple web service could be constructed to both authenticate the uploader and accept new versions via regular HTTP POST. That wouldn't be too hard to script against given the availability of curl on the IRIX side, I would think. That would completely remove the need for aws cli and related friends, and would help enforce the repo structure.

(I'd volunteer to write that web service if nobody else wants to)

 

[hammy]

@Elf - yep, you've got it.

If it helps a little - I've found that each "package" (project/piece of software - something with a .spec) basically falls into four categories:

1. Easy port - using the existing fedora .spec file, perhaps tweaked for irix a little, maybe with a patch or two.
2. Complex port - using the existing fedora .spec file, sometimes with a different version, tarball or heavily modified the .spec.
3. Incompatible existing fedora package - incompatible or cannot currently be built on IRIX - e.g. libjpeg-turbo needs cmake so we switch to the libjpeg from didbs.
4. New - a package for which no fedora .spec exists (e.g. libdicl, the IRIX compatibility library).

I've found it's useful to keep the original .srpm around, since it may contain things that aren't in the modified .spec - and if things aren't referenced in the build, they don't appear in the output .srpm.

Note: I'm talking about the "bootstrap" building here - but keeping the original fedora .srpm and artifacts around continues to be useful if we wish to "activate" functionality commented out from the original.

Some diagrams may be useful (examples only, possibly don't match up with how things are in "didbsng").

Yellow == Useful that we archive these, but not crucial for a build
Green == We need these versioned and tracked, probably in GIT
Orange == These are needed for a build, but they can't be stuffed in GIT
Red == These are our output build artifacts

(1) Existing spec, minor changes:

How I build:

• Install original fedora .srpm
• Copy the SOURCES/* from the didbsng package into ~/rpmbuild/SOURCES/
• Copy the SPECS/* from the didbsng package into ~/rpmbuild/SPECS/
• rpmbuild -ba PACKAGE.spec --nocheck

(2) Existing spec, major changes:

How I build:

• Install original fedora .srpm
• Copy the SOURCES/* from the didbsng package into ~/rpmbuild/SOURCES/
• Copy the SPECS/* from the didbsng package into ~/rpmbuild/SPECS/
• Copy the needed extra binary from didbs (which has the right .tar.gz
• rpmbuild -ba PACKAGE.spec --nocheck

(3) Existing .spec incompatible with IRIX:

How I build:

• Copy the SOURCES/* from the didbsng package into ~/rpmbuild/SOURCES/
• Copy the SPECS/* from the didbsng package into ~/rpmbuild/SPECS/
• Copy the needed extra binary from didbs (which has the right .tar.gz
• rpmbuild -ba PACKAGE.spec --nocheck

(4) No existing .spec, new package:

How I build:

• Copy the SOURCES/* from the didbsng package into ~/rpmbuild/SOURCES/
• Copy the SPECS/* from the didbsng package into ~/rpmbuild/SPECS/
• Copy the needed extra binary from didbs (which has the right .tar.gz
• rpmbuild -ba PACKAGE.spec --nocheck

 

[hammy]

One other thing that might be useful to know - rpm has an idea of "architecture" - which for us is "mips".

This would be quite the ball-ache to change - and currently I see no way to differentiate between mips3 and mips4 using this.

It's possible that separate trees using the rpm "dist" could be used.

e.g. didbsng3 and didbsng4 as variations on the dist - but it's not yet clear to me if that enforces installing the right RPMs specific for the target.

TLDR: I've no idea if/how we build mips3/mips4 and keep things "correct".

 

[Elf]

Good writeup! The visualizations are helpful.

How has it been maintaining patches in source control for you? It isn't a process I have a lot of familiarity with since I usually use source control to apply changes against full sources. It seems like the workflow might be somewhat cumbersome if not figured out properly.

Regarding architecture, I think one is able to specify it (although I admit to not having a working knowledge of this). Some of the basics are in Maximum RPM Ch. 19.

According to Ch. 19 ( http://ftp.rpm.org/max-rpm/s1-rpm-multi-build-install-detection.html ):

The --buildarch and --buildos options can be used to set the build-time architecture and operating system rather than relying on RPM's automatic detection capabilities. These options are added to a normal RPM build command.

One can also incorporate architecture specific flags into the build process: http://ftp.rpm.org/max-rpm/s1-rpm-multi-optflags.html

Eventually you would need to update rpmrc to map compatible architectures. From common practice, I think while different architectures built for the same OS may be in the same repository, packages for different OSes (in our case, IRIX 6.5, 6.2, 5.4, etc.) would be separated into different yum repositories. One could separate architecture along mips3/mips4 lines or conceivably even get as specific as IP level, if the appropriate rpmrc mappings are in place. But mips3/mips4 seems most straightforward for now.

 

[hammy]

Q: What's it like maintaining patches in source control?
A: Usual "it depends" applies.

e.g.

1. For patches to apply to packages themselves, in didbs (and didbsng too) - I store and version the "sgifixes" for a package as the diff against the base package. Thus its a case of "untar, apply patch, configure and build". This works great for these package release diffs and the workflow seems fine.
   
   Since probably the initial port / set of fixes is done outside of RPM, this seems a sensible way to continue going forwards.
   
   
2. For RPM ".specs" I work a little differently - here I'm versioning and storing the .spec directly in source control rather than diffs against the fedora .spec. For me it doesn't make sense these are stored as "diffs" or patches - while I'm working on these I like to be able to see the "whole picture" - and on top of that there are packages for which no existing .spec exists, or the .spec for IRIX is so wildly different from the fedora/other one it doesn't make sense to use diffs.
   

Of course this is just the workflow I'm using right now while trying to get a base environment bootstrapped - and all the ordering of builds and working out what to copy where, what to patch, what tarballs are needed is done by hand. We have a free hand to come up with some other method or workflow.

The only constraints that we have are related to the heavy artifacts the we use / produce - i.e. avoiding storing 200MB tarballs in GIT.

RE: mips3/mips4 - ah - I've not dived deeply in there at all to look into that stuff - I leave this investigation / proposal as "out of scope" for the moment .-)

 

[hammy]

Edit: didbsng has served it's purpose and it's time to be retired. A new collaborative project "sgug-rse" is in progress with a newer selfhosting snapshot.

More soon.

For anyone interested, there is now an alpha quality preview of the didbsng environment (precursor/proving ground before we go /usr/sgug) downloadable from github here:

Note: This really is alpha quality, quite a few things are broken (no ca-certs, no editors, gdb missing, no debug info)

Follow the destructions available from the project readme:

Use the "didbsngshell" mentioned there to start a shell using only the self-hosted tools.

 

[onre]

1. For patches to apply to packages themselves, in didbs (and didbsng too) - I store and version the "sgifixes" for a package as the diff against the base package. Thus its a case of "untar, apply patch, configure and build". This works great for these package release diffs and the workflow seems fine.
   
   Since probably the initial port / set of fixes is done outside of RPM, this seems a sensible way to continue going forwards.

Reading this + staring at the diagrams leaves me with this question:

How does this work if the package has a bunch of patches from Fedora? For example, pcre2 has sixteen patches and none of them patch the one thing that affects IRIX, so patching those patches doesn't make sense. In this particular case patching the "base package", which I assume to be the source package before applying any Fedora patches, would work, but I'm pretty sure that won't be the case every time.

 

[onre]

Also, should the initial port really even happen outside of RPM? If we begin the journey in rpmsphere (six consonants in a row), we get the aforementioned Fedora patches "for free" and possibly avoid duplicate work. Also it feels like a rather handy way of working, now that I've actually tried it.

 

[hammy]

Reading this + staring at the diagrams leaves me with this question:

How does this work if the package has a bunch of patches from Fedora? For example, pcre2 has sixteen patches and none of them patch the one thing that affects IRIX, so patching those patches doesn't make sense. In this particular case patching the "base package", which I assume to be the source package before applying any Fedora patches, would work, but I'm pretty sure that won't be the case every time.

Well it depends on the package of course. Some of the stuff I've worked with have a bunch of extra stuff added by the fedora patches that do things and have dependencies that just add to the complexity of any port. e.g. systemd, kerberos, linux-isms etc.

I've tried to leave in place as much fedora patching/fixes as is possible - it's nice to profit from their work + fixes.

Also, should the initial port really even happen outside of RPM? If we begin the journey in rpmsphere (six consonants in a row), we get the aforementioned Fedora patches "for free" and possibly avoid duplicate work. Also it feels like a rather handy way of working, now that I've actually tried it.

It's a style/taste thing I guess. I find it easier to do a port outside of RPM to reduce the number of moving parts. There's no "right way", that's just what has worked for me so far. Maybe once we have debug symbols + gdb I'll cave to just "doing it via RPM" .-)

 

[onre]

The need for artifact repository is becoming real, I suppose. Alternatively we need some way of specifying which exact source is needed with a spec file. It's all better once we have the initial SRPM of a package, but even that should be stored somewhere and Github is not that somewhere.

 

[Elf]

Probably time to look at a forum authenticated upload mechanism for ports.sgi.sh