RPM builds and version control

[Elf]

Edit: With apologies, I no longer wish to have involvement with SGUG or SGI communities in general,
and have also chosen to remove all of my content. Many things have changed since I co-founded, named, and ultimately
then left SGUG. There are many good people around, to whom I apologize for frustrating by removing these things, and
also many petty people that over the years whittled down both the enjoyment as well as sense of obligation I used to
feel to anyone else regarding what was ultimately just a hobby. Unfortunately one of the latter now writes the rules
and so it is time for me to take my things and go.

This message will replace all of my previous forum posts because deleting threads that I started would have removed
other peoples' posts.

 

[massiverobot]

Regarding storage of the SRPMS and S3- it will be another bootstrap process to get python working, then some aws_cli tooling and then the ability for our system to manage artifacts in a S3 system. I wonder if trying to use a system like Nexus first may be easier. Not sure, but it is something i'll investigate. Ultimately I'd def want things in S3 but not sure how it would really work from within IRIX.

 

[Elf]

Edit: With apologies, I no longer wish to have involvement with SGUG or SGI communities in general,
and have also chosen to remove all of my content. Many things have changed since I co-founded, named, and ultimately
then left SGUG. There are many good people around, to whom I apologize for frustrating by removing these things, and
also many petty people that over the years whittled down both the enjoyment as well as sense of obligation I used to
feel to anyone else regarding what was ultimately just a hobby. Unfortunately one of the latter now writes the rules
and so it is time for me to take my things and go.

This message will replace all of my previous forum posts because deleting threads that I started would have removed
other peoples' posts.

 

[hammy]

@Elf - yep, you've got it.

If it helps a little - I've found that each "package" (project/piece of software - something with a .spec) basically falls into four categories:

1. Easy port - using the existing fedora .spec file, perhaps tweaked for irix a little, maybe with a patch or two.
2. Complex port - using the existing fedora .spec file, sometimes with a different version, tarball or heavily modified the .spec.
3. Incompatible existing fedora package - incompatible or cannot currently be built on IRIX - e.g. libjpeg-turbo needs cmake so we switch to the libjpeg from didbs.
4. New - a package for which no fedora .spec exists (e.g. libdicl, the IRIX compatibility library).

I've found it's useful to keep the original .srpm around, since it may contain things that aren't in the modified .spec - and if things aren't referenced in the build, they don't appear in the output .srpm.

Note: I'm talking about the "bootstrap" building here - but keeping the original fedora .srpm and artifacts around continues to be useful if we wish to "activate" functionality commented out from the original.

Some diagrams may be useful (examples only, possibly don't match up with how things are in "didbsng").

Yellow == Useful that we archive these, but not crucial for a build
Green == We need these versioned and tracked, probably in GIT
Orange == These are needed for a build, but they can't be stuffed in GIT
Red == These are our output build artifacts

(1) Existing spec, minor changes:

How I build:

• Install original fedora .srpm
• Copy the SOURCES/* from the didbsng package into ~/rpmbuild/SOURCES/
• Copy the SPECS/* from the didbsng package into ~/rpmbuild/SPECS/
• rpmbuild -ba PACKAGE.spec --nocheck

(2) Existing spec, major changes:

How I build:

• Install original fedora .srpm
• Copy the SOURCES/* from the didbsng package into ~/rpmbuild/SOURCES/
• Copy the SPECS/* from the didbsng package into ~/rpmbuild/SPECS/
• Copy the needed extra binary from didbs (which has the right .tar.gz
• rpmbuild -ba PACKAGE.spec --nocheck

(3) Existing .spec incompatible with IRIX:

How I build:

• Copy the SOURCES/* from the didbsng package into ~/rpmbuild/SOURCES/
• Copy the SPECS/* from the didbsng package into ~/rpmbuild/SPECS/
• Copy the needed extra binary from didbs (which has the right .tar.gz
• rpmbuild -ba PACKAGE.spec --nocheck

(4) No existing .spec, new package:

How I build:

• Copy the SOURCES/* from the didbsng package into ~/rpmbuild/SOURCES/
• Copy the SPECS/* from the didbsng package into ~/rpmbuild/SPECS/
• Copy the needed extra binary from didbs (which has the right .tar.gz
• rpmbuild -ba PACKAGE.spec --nocheck

 

[hammy]

One other thing that might be useful to know - rpm has an idea of "architecture" - which for us is "mips".

This would be quite the ball-ache to change - and currently I see no way to differentiate between mips3 and mips4 using this.

It's possible that separate trees using the rpm "dist" could be used.

e.g. didbsng3 and didbsng4 as variations on the dist - but it's not yet clear to me if that enforces installing the right RPMs specific for the target.

TLDR: I've no idea if/how we build mips3/mips4 and keep things "correct".

 

[Elf]

Edit: With apologies, I no longer wish to have involvement with SGUG or SGI communities in general,
and have also chosen to remove all of my content. Many things have changed since I co-founded, named, and ultimately
then left SGUG. There are many good people around, to whom I apologize for frustrating by removing these things, and
also many petty people that over the years whittled down both the enjoyment as well as sense of obligation I used to
feel to anyone else regarding what was ultimately just a hobby. Unfortunately one of the latter now writes the rules
and so it is time for me to take my things and go.

This message will replace all of my previous forum posts because deleting threads that I started would have removed
other peoples' posts.

 

[hammy]

Q: What's it like maintaining patches in source control?
A: Usual "it depends" applies.

e.g.

1. For patches to apply to packages themselves, in didbs (and didbsng too) - I store and version the "sgifixes" for a package as the diff against the base package. Thus its a case of "untar, apply patch, configure and build". This works great for these package release diffs and the workflow seems fine.
   
   Since probably the initial port / set of fixes is done outside of RPM, this seems a sensible way to continue going forwards.
   
   
2. For RPM ".specs" I work a little differently - here I'm versioning and storing the .spec directly in source control rather than diffs against the fedora .spec. For me it doesn't make sense these are stored as "diffs" or patches - while I'm working on these I like to be able to see the "whole picture" - and on top of that there are packages for which no existing .spec exists, or the .spec for IRIX is so wildly different from the fedora/other one it doesn't make sense to use diffs.
   

Of course this is just the workflow I'm using right now while trying to get a base environment bootstrapped - and all the ordering of builds and working out what to copy where, what to patch, what tarballs are needed is done by hand. We have a free hand to come up with some other method or workflow.

The only constraints that we have are related to the heavy artifacts the we use / produce - i.e. avoiding storing 200MB tarballs in GIT.

RE: mips3/mips4 - ah - I've not dived deeply in there at all to look into that stuff - I leave this investigation / proposal as "out of scope" for the moment .-)

 

[hammy]

Edit: didbsng has served it's purpose and it's time to be retired. A new collaborative project "sgug-rse" is in progress with a newer selfhosting snapshot.

More soon.

For anyone interested, there is now an alpha quality preview of the didbsng environment (precursor/proving ground before we go /usr/sgug) downloadable from github here:

Note: This really is alpha quality, quite a few things are broken (no ca-certs, no editors, gdb missing, no debug info)

Follow the destructions available from the project readme:

Use the "didbsngshell" mentioned there to start a shell using only the self-hosted tools.

 

[onre]

1. For patches to apply to packages themselves, in didbs (and didbsng too) - I store and version the "sgifixes" for a package as the diff against the base package. Thus its a case of "untar, apply patch, configure and build". This works great for these package release diffs and the workflow seems fine.
   
   Since probably the initial port / set of fixes is done outside of RPM, this seems a sensible way to continue going forwards.

Reading this + staring at the diagrams leaves me with this question:

How does this work if the package has a bunch of patches from Fedora? For example, pcre2 has sixteen patches and none of them patch the one thing that affects IRIX, so patching those patches doesn't make sense. In this particular case patching the "base package", which I assume to be the source package before applying any Fedora patches, would work, but I'm pretty sure that won't be the case every time.

 

[onre]

Also, should the initial port really even happen outside of RPM? If we begin the journey in rpmsphere (six consonants in a row), we get the aforementioned Fedora patches "for free" and possibly avoid duplicate work. Also it feels like a rather handy way of working, now that I've actually tried it.

 

[hammy]

Reading this + staring at the diagrams leaves me with this question:

How does this work if the package has a bunch of patches from Fedora? For example, pcre2 has sixteen patches and none of them patch the one thing that affects IRIX, so patching those patches doesn't make sense. In this particular case patching the "base package", which I assume to be the source package before applying any Fedora patches, would work, but I'm pretty sure that won't be the case every time.

Well it depends on the package of course. Some of the stuff I've worked with have a bunch of extra stuff added by the fedora patches that do things and have dependencies that just add to the complexity of any port. e.g. systemd, kerberos, linux-isms etc.

I've tried to leave in place as much fedora patching/fixes as is possible - it's nice to profit from their work + fixes.

Also, should the initial port really even happen outside of RPM? If we begin the journey in rpmsphere (six consonants in a row), we get the aforementioned Fedora patches "for free" and possibly avoid duplicate work. Also it feels like a rather handy way of working, now that I've actually tried it.

It's a style/taste thing I guess. I find it easier to do a port outside of RPM to reduce the number of moving parts. There's no "right way", that's just what has worked for me so far. Maybe once we have debug symbols + gdb I'll cave to just "doing it via RPM" .-)

 

[onre]

The need for artifact repository is becoming real, I suppose. Alternatively we need some way of specifying which exact source is needed with a spec file. It's all better once we have the initial SRPM of a package, but even that should be stored somewhere and Github is not that somewhere.

 

[Elf]

Edit: With apologies, I no longer wish to have involvement with SGUG or SGI communities in general,
and have also chosen to remove all of my content. Many things have changed since I co-founded, named, and ultimately
then left SGUG. There are many good people around, to whom I apologize for frustrating by removing these things, and
also many petty people that over the years whittled down both the enjoyment as well as sense of obligation I used to
feel to anyone else regarding what was ultimately just a hobby. Unfortunately one of the latter now writes the rules
and so it is time for me to take my things and go.

This message will replace all of my previous forum posts because deleting threads that I started would have removed
other peoples' posts.