Making Tardists the hard way

massiverobot

irix detailer
Feb 8, 2019
96
61
18
Philly
twitter.com
During the course of trying to make the most efficient RSE bootstrap method, I came to the conclusion that distributing it as a tardist was the only way to get it quickly and easily installed on an un-blessed (with DIDBS) SGI. Using FTP to retrieve (ftp.irix.fun) and inst to install removes most barriers for any SGI enthusiast using IRIX 6.5.22/30. All you need is avtive connection to the Internet to fetch and then install RSE.

But first, I needed that tardist. How to create one?

Existing Resources
There is this classic page: https://vanalboom.org/node/6
However, reading that- it's first, super convoluted, and 2nd it's targeted towards a specific software package being built from source. I had other designs. I wanted to create a tardist of dozens of packages, already installed and built.

There is also this techpub: 007-2503-003.pdf
While it gives a seemingly nice overview of Software Packager, it also does not delve into any details on how to do something other than an existing small software package. And it makes no mention of any error conditions from gendist, nor does it mention symlinks...

Doing It Yourself
During the course of events around creating (what I thought was) a simple "tardist of an existing directory of files..." I have solved a few things that I have not seen documented around on the Internet, and I want to document here for others to (enjoy) use.

I created a gist to document what I did to generate the tardist. The gist is here:

I'll be going over and refining the process again in order to create a self-hoster for RSE B6 and at that time I may find the time to clean this up. But until then there is a workable process that can be followed.



Below is the process part, copied from my gist. The python script is in the gist:

You need to pre-process the idb file and create a file which has all the symlinks from the filesystem.
In this scenario we have a hierarchy of files on our sgi. You could just tarball them up and move them to a new system and extract. But in this case we want to use inst to install, and therefore gendist to create a tardist.

So there are two directories involved with creating a tardist: gendist calls them source base and target base.

For this case, the source base (sbase) is /usr/sgug_raw_selfhoster and the target base is /usr/sgug.

This means the files to use to create the tardist exist in /usr/sgug_raw_selfhoster and when inst installs them I want them to go into /usr/sgug. Be aware of those paths in the example below and change as necessary.

1. Run Software Packager - point at the top of your SBASE. Add all the files into one sw.base for ease of use. Set the target up to be usr/sgug (no leading /). Save the IDB and SPEC files somewhere you can edit them. The rest will be done by hand. Quit Software Packager.

This sample made a tardist with over 11,000 files in it. I ran Software Packager on a Fuel w/ 4G ram and 900mhz CPU and it takes a bit of time. If you have a slower system be prepared for SP to take some time to move between stages. I didn't change or add any defaults to the files beyond tagging them and setting the target base.

Do NOT name the package anything fancy, do not use any dashes in the name.

Sample:
$ cat rse-b5-self-hoster.spec
product rse
id "R S E -- SGUG"
image sw
id "RSE Bootstrap Software"
version 1.5
order 9999
subsys base default
id "Base Software"
replaces self
exp rse.sw.base
endsubsys
endimage
endproduct


You were warned. Use a single name for the exp! I say this as I lost about 10 hours (cumlitave, over about 4 days) of my life by naming the exp "rse-self-hoster.sw.base" - and had errors in gendist that I couldn't figure out till I just had an ah-ha moment and simplified the name. Please use a single word with no punctuation in it.


In this sample the IDB file is named: rse-b5-self-hoster.idb
A file called fs_link_input_file.txt is created and used to map symlinks

2. Sort the idb file properly:
$ sort -k 5 rse-b5-self-hoster.idb > rse-b5-self-hoster.idb_sorted
$ cp rse-b5-self-hoster.idb rse-b5-self-hoster.idb_orig
$ cp rse-b5-self-hoster.idb_sorted rse-b5-self-hoster.idb


3. Generate symlink input file. You need this as input to the python script to show it where the symlinks link to
$ find /usr/sgug_raw_selfhoster -type l -ls | awk '{print $11 " " $13}' > fs_link_input_file.txt

4. Fix this file using sed, as we want the paths in it to match our TBASE (not SBASE)
$ sed 's/sgug_raw_selfhoster/sgug/g' fs_link_input_file.txt > fs_link_input_file2.txt

The file should look like this:

$ cat fs_link_input_file2.txt
...
/usr/sgug/lib32/libdicl-0.1.so.0 libdicl-0.1.so.0.0.0
/usr/sgug/lib32/libffi.so.6 libffi.so.6.0.4
/usr/sgug/lib32/libbz2.so.1 libbz2.so.1.0.8
/usr/sgug/lib32/libbz2.so.1.0 libbz2.so.1.0.8
/usr/sgug/lib32/liblzma.so.5 liblzma.so.5.2.4
/usr/sgug/lib32/libgcc_s.so libgcc_s.so.1
/usr/sgug/lib32/libgcc_s.so.1 libgcc_s-9-20190812.so.1
/usr/sgug/lib32/libz.so.1 libz.so.1.2.11
...


It should have the symlink a space and the file it's linked to.


5. Run the python script (below)
$ python ./fix_symlinks2.py > rse-b5-self-hoster.idb_fixed

6. Create the tardist
$ /usr/sbin/gendist -sbase /usr/sgug_raw_selfhoster -idb /usr/people/dillera/rse-b5-self-hoster.idb_fixed -spec /usr/people/dillera/rse-b5-self-hoster.spec -dist /usr/people/dillera/dist -verbose -nostrip

Use -dryrun to see details if there are errors.

7. Make a tar file
~ $ cd dist/
~/dist $ tar cvf ../rse.tardist *
rse
rse.idb
rse.sw


Now you have a tardist named "res.tardist" and you can distribute it and install it via inst.

Here I am installing it on a pristine 6.5.30 install:

* Warning - the first part [Unpacking tardist file...] will take a long time. Be prepared to wait 10+min. It took that long on my origin2k.


origin 55# inst -a -f ./sgug.rse.b5-boostrap.tardist
Unpacking tardist file into temporary distribution directory /var/tmp/tardista006Yk
Unpacking tardist file .. 100% Done.
Reading product descriptions .. 13%
Reading /var/inst/hist
Reading product descriptions .. 25%
Setting distribution to /var/tmp/tardista006Yk
Reading product descriptions .. 100% Done.
Reading fileset information .. 8%
Pre-installation check .. 16%
Checking space requirements .. 24%
Pre-installation check completed
Installing/removing files .. 24%
Installing new versions of selected rse.sw subsystems
Installing/removing files .. 94%
Removing orphaned directories
Running exit-commands .. 100% Done.
Installations and removals were successful.
Requickstarting ELF files (see rqsall(1)) .. 100% Done.



once it's installed you want to set the env correctly in csh:


# setenv LD_LIBRARYN32_PATH /usr/sgug/lib32:/usr/lib32:/lib32:/usr/lib:/lib
# setenv PATH /usr/sgug/bin:/usr/sbin:/usr/bsd:/sbin:/usr/bin:/usr/bin/X11::/usr/etc


And now you can access curl, rpm and other tools in RSE self-hoster.

Now you may want to go out and use RPM to install other packages that are part of the RSE b5.


[edits are corrections for spelling mistakes, adding new sections at top]
 
Last edited:

hammy

Active member
Jun 1, 2019
107
61
28
UK
Good job! It's one thing to talk about "we should just do X" and quite another to actually deliver something!

Do you have any idea how tricky it is to add the appropriate patches as dependencies? (must support 6.5.22/6.5.30 and IRIX/IRIX64)

I know it's not simple, it's perhaps something we can think about for the future (see https://github.com/sgidevnet/sgug-rse/issues/51)

(Borat voice) - Very nice
 

About us

  • Silicon Graphics User Group (SGUG) is a community for users, developers, and admirers of Silicon Graphics (SGI) products. We aim to be a friendly hobbyist community for discussing all aspects of SGIs, including use, software development, the IRIX Operating System, and troubleshooting, as well as facilitating hardware exchange.

User Menu