Often there is no available binary distribution or the binary fails to run for obscure reasons. In these circumstances one has to download the source code, compile and link on your local machine, and set the environment so that the resulting executables or libraries can be located by the operating system. In this lesson, we will download and build two packages. The first, ncftp-3.2.2 is a nice back end to FTP (file transfer protocol) - a protocol for moving files over the internet. A predecessor program, simply ftp, exists on your computer, but it is not as nice to use, and besides ncftp is a nice simple package for your first compile. The second,

g++-4.2.3, is a very recent version of the same C++ compiler that comes bundled on your linux box. We need this newer version to ensure compatibility between builds of LIGO software here and with the developers at Caltech.

DOWNLOADING AND UNPACKING NCFTP

The source for ncftp is available at http://www.ncftp.com/download/ . Scroll down the page until you see the most recent version of the source code. Currently this is NcFTP Client 3.2.2. On the link, click with the right mouse button (right click), and select save link as. There are two versions of the distribution, called ncftp-3.2.2-src.tar.gz and ncftp-3.2.2-src.tar.bz2 respectively. You can get either. Save to your home directory, then cd to your home directory using cd ~. In this directory, create a subdirectory called utilities by typing mkdir utilities. Then move the binary into this subdirectory using mv ncftp-3.2.2-src.tar.gz utilities/. . Now cd into the utilities directory with cd utilities. Once in this directory you are ready to unpack the source. Hopefully this pedestrian set of instructions will have taught any of you who don’t already know how to move around your directory tree using command line instructions. I will be less pedestrian with these things from now on.

You have just downloaded a compressed archive of the source for the ncftp client.

What you do next depends on whether you downloaded the version ending in .gz or that ending in .bz2. For the .bz2 version, you need to type bunzip2 <compressed archive name>. For the .gz version, you could type gzip -d <compressed archive name>. One thing you may or may not have noticed by now is a handy feature of the bash shell - file name completion. If you type gzip -d libftp and you don’t really want to type in the rest of that long winded name, just hit tab and as long as the file you want is the only one in the directory starting this way, the shell will complete the file name for you. If it can’t figure out what to do, the computer will beep at you instead. gzip and bzip are two utilities for compressing and decompressing  archives. An archive itself is essentially a directory tree packed up into a single file. What you should be left with after decompression of the .bz2 or .gz file is a file whose name ends in .tar . The final stage is to use the tar program to unpack the archive. This is done with tar -xf <archive name>. Once you have done this you should be able to see a directory called ncftp-3.2.2 (or whatever version you downloaded) in your utilities directory. Finally, cd into this directory.

Note that there are always shortcuts in unix. The entire decompression process can be handled (for the .gz compressed archive) with

gzip -cd <compressed archive name> | tar xf -

There is certainly an equivalent command for a bzip compressed archive.

Or if you have a reasonably modern version of tar (another free software utility) you can use

tar -zxf <compressed archive name>

because modern versions of tar know about gzip compression. Note that sun solaris 10 machines come with a tar that does NOT know about gzip compression. Generally speaking archives that have been compressed with gzip are usually called archive_name.tar.gz or archive_name.tgz . Incidentally, in WINDOWS or on a mac you can simply double click on the icon for the compressed archive and, at least for gzip compressed archives, the OS knows to run WINZIP or stuffit, which can handle the decompression. However, the free WINZIP and stuffit decompressors will ONLY decompress, whereas the unix based gzip and bzip will both compress and uncompress. And in addition, because gzip, bzip and tar are command line utilities you can automate the archiving of directory trees by incorporating these commands into shell scripts or even compiled programs.

BUILDING AND INSTALLING NCFTP

The FREE SOFTWARE FOUNDATION is an outfit founded by Richard Stallman to write and release free software as source code for the benefit of humanity, and notably for physicists who are probably a subset of humanity and certainly often in need of free software. He is an interesting guy and this is an interesting organization. In fact ncftp is NOT a free software foundation (FSF) product, but it does come in the form of a package built using the toolset developed by FSF for building software from source. The first sign that you are dealing with such a package is that when you ls in the top directory you see a file called configure. In fact, configure is usually a script autogenerated by a program called autoconf. You can run configure in the shell. It’s job is to determine the properties of your computer that are relevant to the build and setting up the apparatus within the package to compile, link and install the software in the appropriate way. Until later you don’t need to know how it works. I suggest you type one of the following - the first if you are on a standalone linux system where you have root privileges, the second if you plan to install ncftp in some non-standard directory in which you have write access:

$ ./configure --prefix=/usr

Configure should produce a great number of lines of response, each of which says it is checking something, and giving a response, usually yes or no. What is going on is the script is testing your system in different ways, and using the responses to create a file called a Makefile that will then instruct the compiler how to build and install the package. However, if configure discovers that some aspect of your system makes it unable to build or install the code, it will exit, hopefully with a helpful message telling you what is wrong. After a few minutes, configure will get done and you will see the prompt again. If you don’t see any error messages, you can go on to the next step. Use ls, and see if a file called Makefile has appeared. The simplest thing to do next is just to type make, and see the compilation unfold before your eyes in the terminal window. However, if something goes wrong, the text that tells you what happened may be lost. To get around this, execute make in a new shell, redirecting the output of make to a file. Like this:

$ bash -c ‘make > ncftpmake.log 2>&1’ &

This rather complicated command deserves some explanation. bash -c means, use the bash interpreter to interpret a string, given in single quotes after the -c argument. The text inside the quotes means, run make and send whatever would normally appear in the terminal (the standard output) to a file called ncftpmake.log. The 2>&1 means that if anything is sent to the standard error stream (used by the operating system for recording error messages), send this output to the same place that standard output got sent to. The final & means that this entire command is launched in a new shell independent of the shell of the terminal window in which the command was typed. So when you hit return after this command, you simply get the prompt back again, along with a number in square brackets which is called the PROCESS ID of the shell in which the command is executing. If you now type ls, you will see that a file ncftpmake.log has appeared. As compilation progresses, you can follow its progress by typing

tail -f ncftpmake.log . tail -f is yet another unix command that in this form displays the last portion of a text file, continuously updating it if the file is being written to. To leave tail -f, type ctrl-c. This does NOT kill the make command, just the tail command that is displaying its output.

For ncftp, the build process will be quite short - in fact by the time you get through reading the above paragraph it may have finished, in which case tail -f will simply show you the static end of the file. Assuming everything goes well, the last line should begin with something like ‘leaving (name of a directory)’, or ‘nothing to be done for all’ or something of that sort. This means it’s finished. Or you may have some horrific error message, in which case you need to figure out what went wrong. More about how to do this in a later lesson.

$ ./configure --prefix=<directory name>

In the latter case, <directory name> is an existing directory. You could use /home/<username>/myunixutilties, for example, if you had previously typed mkdir ~/myunixutilities

If you intend to install in /usr (the standard place for system-wide executables, libraries, include files, manual pages, etc), you need to become root. Type su and enter the root password. Be careful when you are root; rm -f / destroys the entire filesystem - five little characters and you would need to start over.

Whether or not you are installing in /usr, your next command is make install. This should install the executable for ncftp and any libraries that it needs in the directory specified after

--prefix= at the configure stage. If you are root, use exit to stop being root after installation.

Finally, you need to set up the OS to locate ncftp when you invoke the command. If you installed in /usr, you don’t need to do anything; the OS always looks there by default. Otherwise, you need to set an environment variable called path. First, type echo $PATH to find out what PATH is already set to. It may be that it doesn’t exist or it is empty. If so, you can set it to point to the location of the binary for ncftp. For example, if you installed in

/home/<username>/myunixutilities, you would type

$ export PATH=/home/<username>/myunixutilities/bin

If, however, PATH already contains some directories, you need to add the path to the ncftp executable to the existing PATH. You could do this with

$ export PATH=/home/<username>/myunixutilities/bin:$PATH

This means that linux will search all the directories in the PATH environment variable first, with the ones earlier on the list taking precedence over the later ones, then finally it will search the places it ‘usually’ looks for executables such as /usr/bin. Note that if there are several copies of ncftp lying around and you care which one is used, it is critical WHERE in the path environment variable the directory containing the desired version is located.

After you have modified PATH (or not if you installed in /usr), you should be able to type

which ncftp and get the full directory location of the ncftp that you installed. Now it is time

to try running it. For example, to go the the free software foundation web site, use

ncftp ftp.gnu.org . You should find yourself transported to the remote location, as long as your computer is on the internet. For now, just type quit to get back out and refuse to enter a bookmark. We will next use this software to download gcc/g++ compilers for building other more gravitational wave based applications and libraries.

Notice the quotes are SINGLE and FORWARD FACING. Different quotes do different things.

Finally, if you installed in /home/<username>/myunixutilities, you will need to add the same command (export PATH...etc) to the .bashrc file in your home directory.

BUILDING AND INSTALLING GCC/G++ VERSION 4.2.3

We next build from source and install the same C and C++ compilers used by the developers of the DMT at Caltech. If you have already done this, then you may disregard this section. To get the compilers, use

$ ncftp ftp.gnu.org

(once you are logged in) cd pub/gnu/gcc/gcc-4.2.3

(to get the c compiler) get gcc-core-4.2.3.tar.bz2

(to get the c++ compiler) get gcc-g++-4.2.3.tar.bz2

(to get out again) quit

(at this point you are asked if you want to save a

bookmark for the site. I suggest you use the bookmark

gcc. Next time you need to get to this repository you

can just type ncftp gcc and avoid all the other messing

about.

Move both the compressed archives to the utilities directory, then uncompress first the core

archive and then the g++ archive using bunzip2 followed by tar as for the ncftp build.

It is probably best NOT to install these compilers in /usr. This is because the compilers resident on the system, and particularly the libraries associated with these compilers, may be needed by other programs. If you install the new compilers in the same place, there is a possibility that these existing components may get overwritten. I doubt it, but I am not sure, so best to be on the safe side and install these compilers in some other write-accessible directory. Perhaps /home/<username>/myunixutilities would be a good choice.

You should now be able to build and install the compilers using exactly the same sequence of steps as for ncftp. For interest, note that the build proceeds using a ‘bootstrap compile’. First, an existing c compiler is found on the system. This does not have to be gcc, and it can be quite primative. Next this compiler is used to build a preliminary version of gcc then this compiler is used to re-compile the same source. This is repeated three times at which point test determines whether the latest compiler version and the previous one are functionally identical. If they are, the latest gcc compiler is used to build a g++ compiler directly from source.

FURTHER PRACTICE AND PREREQUISITES FOR ROOT

The following programs should be build from source and installed on your system in preparation for your first taste of ROOT. If they are already on your system, then don’t worry.

SVN client - SVN is short for subversion, and it is a version control program. Version control is a method by which many programmers working on the same code can work with the same source distribution in a controlled manner. Many software projects use version control to check out code, including DMT and root. The DMT uses cvs which I feel sure will be on your system (unless you run solaris-10). Subversion will probably be there too - type which svn to find out. if it is not, download the source from here. You need the latest source code compressed archive, which is currently 1.5.2 . Incidentally a compressed archive created using tar followed by a compression program such as gzip or bzip2 is often called a tarball.

Emacs - if you haven’t already got this on your linux box, you will need it. This is the text editor that I recommend you to use for actually writing your code and shell scripts. It has an initial learning curve that is fairly steep, but it is quite powerful and has lots of useful features. You can download it from ftp.gnu.org - the subdirectory is pub/gnu/emacs. Get the latest version, avoiding xemacs as it doesn’t add much to the functionality of emacs. The same build method applies, and I recommend that it is installed in /usr.

Back to courses index