During the course of trying to make the most efficient RSE bootstrap method, I came to the conclusion that distributing it as a tardist was the only way to get it quickly and easily installed on an un-blessed (with DIDBS) SGI. Using FTP to retrieve (ftp.irix.fun) and inst to install removes most barriers for any SGI enthusiast using IRIX 6.5.22/30. All you need is avtive connection to the Internet to fetch and then install RSE.
But first, I needed that tardist. How to create one?
Existing Resources
There is this classic page: https://vanalboom.org/node/6
However, reading that- it's first, super convoluted, and 2nd it's targeted towards a specific software package being built from source. I had other designs. I wanted to create a tardist of dozens of packages, already installed and built.
There is also this techpub: 007-2503-003.pdf
While it gives a seemingly nice overview of Software Packager, it also does not delve into any details on how to do something other than an existing small software package. And it makes no mention of any error conditions from gendist, nor does it mention symlinks...
Doing It Yourself
During the course of events around creating (what I thought was) a simple "tardist of an existing directory of files..." I have solved a few things that I have not seen documented around on the Internet, and I want to document here for others to (enjoy) use.
I created a gist to document what I did to generate the tardist. The gist is here:
I'll be going over and refining the process again in order to create a self-hoster for RSE B6 and at that time I may find the time to clean this up. But until then there is a workable process that can be followed.
Below is the process part, copied from my gist. The python script is in the gist:
You need to pre-process the idb file and create a file which has all the symlinks from the filesystem.
In this scenario we have a hierarchy of files on our sgi. You could just tarball them up and move them to a new system and extract. But in this case we want to use inst to install, and therefore gendist to create a tardist.
So there are two directories involved with creating a tardist: gendist calls them source base and target base.
For this case, the source base (sbase) is /usr/sgug_raw_selfhoster and the target base is /usr/sgug.
This means the files to use to create the tardist exist in /usr/sgug_raw_selfhoster and when inst installs them I want them to go into /usr/sgug. Be aware of those paths in the example below and change as necessary.
1. Run Software Packager - point at the top of your SBASE. Add all the files into one sw.base for ease of use. Set the target up to be usr/sgug (no leading /). Save the IDB and SPEC files somewhere you can edit them. The rest will be done by hand. Quit Software Packager.
This sample made a tardist with over 11,000 files in it. I ran Software Packager on a Fuel w/ 4G ram and 900mhz CPU and it takes a bit of time. If you have a slower system be prepared for SP to take some time to move between stages. I didn't change or add any defaults to the files beyond tagging them and setting the target base.
Do NOT name the package anything fancy, do not use any dashes in the name.
Sample:
$ cat rse-b5-self-hoster.spec
product rse
id "R S E -- SGUG"
image sw
id "RSE Bootstrap Software"
version 1.5
order 9999
subsys base default
id "Base Software"
replaces self
exp rse.sw.base
endsubsys
endimage
endproduct
You were warned. Use a single name for the exp! I say this as I lost about 10 hours (cumlitave, over about 4 days) of my life by naming the exp "rse-self-hoster.sw.base" - and had errors in gendist that I couldn't figure out till I just had an ah-ha moment and simplified the name. Please use a single word with no punctuation in it.
In this sample the IDB file is named: rse-b5-self-hoster.idb
A file called fs_link_input_file.txt is created and used to map symlinks
2. Sort the idb file properly:
$ sort -k 5 rse-b5-self-hoster.idb > rse-b5-self-hoster.idb_sorted
$ cp rse-b5-self-hoster.idb rse-b5-self-hoster.idb_orig
$ cp rse-b5-self-hoster.idb_sorted rse-b5-self-hoster.idb
3. Generate symlink input file. You need this as input to the python script to show it where the symlinks link to
$ find /usr/sgug_raw_selfhoster -type l -ls | awk '{print $11 " " $13}' > fs_link_input_file.txt
4. Fix this file using sed, as we want the paths in it to match our TBASE (not SBASE)
$ sed 's/sgug_raw_selfhoster/sgug/g' fs_link_input_file.txt > fs_link_input_file2.txt
The file should look like this:
$ cat fs_link_input_file2.txt
...
/usr/sgug/lib32/libdicl-0.1.so.0 libdicl-0.1.so.0.0.0
/usr/sgug/lib32/libffi.so.6 libffi.so.6.0.4
/usr/sgug/lib32/libbz2.so.1 libbz2.so.1.0.8
/usr/sgug/lib32/libbz2.so.1.0 libbz2.so.1.0.8
/usr/sgug/lib32/liblzma.so.5 liblzma.so.5.2.4
/usr/sgug/lib32/libgcc_s.so libgcc_s.so.1
/usr/sgug/lib32/libgcc_s.so.1 libgcc_s-9-20190812.so.1
/usr/sgug/lib32/libz.so.1 libz.so.1.2.11
...
It should have the symlink a space and the file it's linked to.
5. Run the python script (below)
$ python ./fix_symlinks2.py > rse-b5-self-hoster.idb_fixed
6. Create the tardist
$ /usr/sbin/gendist -sbase /usr/sgug_raw_selfhoster -idb /usr/people/dillera/rse-b5-self-hoster.idb_fixed -spec /usr/people/dillera/rse-b5-self-hoster.spec -dist /usr/people/dillera/dist -verbose -nostrip
Use -dryrun to see details if there are errors.
7. Make a tar file
~ $ cd dist/
~/dist $ tar cvf ../rse.tardist *
rse
rse.idb
rse.sw
Now you have a tardist named "res.tardist" and you can distribute it and install it via inst.
Here I am installing it on a pristine 6.5.30 install:
* Warning - the first part [Unpacking tardist file...] will take a long time. Be prepared to wait 10+min. It took that long on my origin2k.
origin 55# inst -a -f ./sgug.rse.b5-boostrap.tardist
Unpacking tardist file into temporary distribution directory /var/tmp/tardista006Yk
Unpacking tardist file .. 100% Done.
Reading product descriptions .. 13%
Reading /var/inst/hist
Reading product descriptions .. 25%
Setting distribution to /var/tmp/tardista006Yk
Reading product descriptions .. 100% Done.
Reading fileset information .. 8%
Pre-installation check .. 16%
Checking space requirements .. 24%
Pre-installation check completed
Installing/removing files .. 24%
Installing new versions of selected rse.sw subsystems
Installing/removing files .. 94%
Removing orphaned directories
Running exit-commands .. 100% Done.
Installations and removals were successful.
Requickstarting ELF files (see rqsall(1)) .. 100% Done.
once it's installed you want to set the env correctly in csh:
# setenv LD_LIBRARYN32_PATH /usr/sgug/lib32:/usr/lib32:/lib32:/usr/lib:/lib
# setenv PATH /usr/sgug/bin:/usr/sbin:/usr/bsd:/sbin:/usr/bin:/usr/bin/X11::/usr/etc
And now you can access curl, rpm and other tools in RSE self-hoster.
Now you may want to go out and use RPM to install other packages that are part of the RSE b5.
[edits are corrections for spelling mistakes, adding new sections at top]
But first, I needed that tardist. How to create one?
Existing Resources
There is this classic page: https://vanalboom.org/node/6
However, reading that- it's first, super convoluted, and 2nd it's targeted towards a specific software package being built from source. I had other designs. I wanted to create a tardist of dozens of packages, already installed and built.
There is also this techpub: 007-2503-003.pdf
While it gives a seemingly nice overview of Software Packager, it also does not delve into any details on how to do something other than an existing small software package. And it makes no mention of any error conditions from gendist, nor does it mention symlinks...
Doing It Yourself
During the course of events around creating (what I thought was) a simple "tardist of an existing directory of files..." I have solved a few things that I have not seen documented around on the Internet, and I want to document here for others to (enjoy) use.
I created a gist to document what I did to generate the tardist. The gist is here:
This python script fixes symlinks in idm files on IRIX
This python script fixes symlinks in idm files on IRIX - fix_symlinks2.py
gist.github.com
I'll be going over and refining the process again in order to create a self-hoster for RSE B6 and at that time I may find the time to clean this up. But until then there is a workable process that can be followed.
Below is the process part, copied from my gist. The python script is in the gist:
You need to pre-process the idb file and create a file which has all the symlinks from the filesystem.
In this scenario we have a hierarchy of files on our sgi. You could just tarball them up and move them to a new system and extract. But in this case we want to use inst to install, and therefore gendist to create a tardist.
So there are two directories involved with creating a tardist: gendist calls them source base and target base.
For this case, the source base (sbase) is /usr/sgug_raw_selfhoster and the target base is /usr/sgug.
This means the files to use to create the tardist exist in /usr/sgug_raw_selfhoster and when inst installs them I want them to go into /usr/sgug. Be aware of those paths in the example below and change as necessary.
1. Run Software Packager - point at the top of your SBASE. Add all the files into one sw.base for ease of use. Set the target up to be usr/sgug (no leading /). Save the IDB and SPEC files somewhere you can edit them. The rest will be done by hand. Quit Software Packager.
This sample made a tardist with over 11,000 files in it. I ran Software Packager on a Fuel w/ 4G ram and 900mhz CPU and it takes a bit of time. If you have a slower system be prepared for SP to take some time to move between stages. I didn't change or add any defaults to the files beyond tagging them and setting the target base.
Do NOT name the package anything fancy, do not use any dashes in the name.
Sample:
$ cat rse-b5-self-hoster.spec
product rse
id "R S E -- SGUG"
image sw
id "RSE Bootstrap Software"
version 1.5
order 9999
subsys base default
id "Base Software"
replaces self
exp rse.sw.base
endsubsys
endimage
endproduct
You were warned. Use a single name for the exp! I say this as I lost about 10 hours (cumlitave, over about 4 days) of my life by naming the exp "rse-self-hoster.sw.base" - and had errors in gendist that I couldn't figure out till I just had an ah-ha moment and simplified the name. Please use a single word with no punctuation in it.
In this sample the IDB file is named: rse-b5-self-hoster.idb
A file called fs_link_input_file.txt is created and used to map symlinks
2. Sort the idb file properly:
$ sort -k 5 rse-b5-self-hoster.idb > rse-b5-self-hoster.idb_sorted
$ cp rse-b5-self-hoster.idb rse-b5-self-hoster.idb_orig
$ cp rse-b5-self-hoster.idb_sorted rse-b5-self-hoster.idb
3. Generate symlink input file. You need this as input to the python script to show it where the symlinks link to
$ find /usr/sgug_raw_selfhoster -type l -ls | awk '{print $11 " " $13}' > fs_link_input_file.txt
4. Fix this file using sed, as we want the paths in it to match our TBASE (not SBASE)
$ sed 's/sgug_raw_selfhoster/sgug/g' fs_link_input_file.txt > fs_link_input_file2.txt
The file should look like this:
$ cat fs_link_input_file2.txt
...
/usr/sgug/lib32/libdicl-0.1.so.0 libdicl-0.1.so.0.0.0
/usr/sgug/lib32/libffi.so.6 libffi.so.6.0.4
/usr/sgug/lib32/libbz2.so.1 libbz2.so.1.0.8
/usr/sgug/lib32/libbz2.so.1.0 libbz2.so.1.0.8
/usr/sgug/lib32/liblzma.so.5 liblzma.so.5.2.4
/usr/sgug/lib32/libgcc_s.so libgcc_s.so.1
/usr/sgug/lib32/libgcc_s.so.1 libgcc_s-9-20190812.so.1
/usr/sgug/lib32/libz.so.1 libz.so.1.2.11
...
It should have the symlink a space and the file it's linked to.
5. Run the python script (below)
$ python ./fix_symlinks2.py > rse-b5-self-hoster.idb_fixed
6. Create the tardist
$ /usr/sbin/gendist -sbase /usr/sgug_raw_selfhoster -idb /usr/people/dillera/rse-b5-self-hoster.idb_fixed -spec /usr/people/dillera/rse-b5-self-hoster.spec -dist /usr/people/dillera/dist -verbose -nostrip
Use -dryrun to see details if there are errors.
7. Make a tar file
~ $ cd dist/
~/dist $ tar cvf ../rse.tardist *
rse
rse.idb
rse.sw
Now you have a tardist named "res.tardist" and you can distribute it and install it via inst.
Here I am installing it on a pristine 6.5.30 install:
* Warning - the first part [Unpacking tardist file...] will take a long time. Be prepared to wait 10+min. It took that long on my origin2k.
origin 55# inst -a -f ./sgug.rse.b5-boostrap.tardist
Unpacking tardist file into temporary distribution directory /var/tmp/tardista006Yk
Unpacking tardist file .. 100% Done.
Reading product descriptions .. 13%
Reading /var/inst/hist
Reading product descriptions .. 25%
Setting distribution to /var/tmp/tardista006Yk
Reading product descriptions .. 100% Done.
Reading fileset information .. 8%
Pre-installation check .. 16%
Checking space requirements .. 24%
Pre-installation check completed
Installing/removing files .. 24%
Installing new versions of selected rse.sw subsystems
Installing/removing files .. 94%
Removing orphaned directories
Running exit-commands .. 100% Done.
Installations and removals were successful.
Requickstarting ELF files (see rqsall(1)) .. 100% Done.
once it's installed you want to set the env correctly in csh:
# setenv LD_LIBRARYN32_PATH /usr/sgug/lib32:/usr/lib32:/lib32:/usr/lib:/lib
# setenv PATH /usr/sgug/bin:/usr/sbin:/usr/bsd:/sbin:/usr/bin:/usr/bin/X11::/usr/etc
And now you can access curl, rpm and other tools in RSE self-hoster.
Now you may want to go out and use RPM to install other packages that are part of the RSE b5.
[edits are corrections for spelling mistakes, adding new sections at top]
Last edited: