Preface
IO

The purpose of this package is to allow efficient and flexible input/output operations from &GAP;. This is achieved by providing bindings to the low-level I/O functions in the C-library. On top of this an implementation of buffered I/O in the &GAP; language is provided. Further, a framework for serialisation of arbitrary &GAP; objects is implemented. Finally, an implementation of the client side of the HTTP protocol is included in the package.

This package allows to use file based I/O, access to links and file systems, pipes, sockets, and the UDP and TCP/IP protocols.

By default the `IO` package is not automatically loaded by &GAP; when it is installed. You must load the package with LoadPackage("IO"); before its functions become available.

Please, send me an e-mail (neunhoef@mcs.st-and.ac.uk) if you have any questions, remarks, suggestions, etc. concerning this package. Also, I would like to hear about applications of this package.

Max Neunhöffer
Installation of the `IO`-package

To get the newest version of this &GAP; 4 package download one of the archive files io-x.x.tar.gz io-x.x.zoo io-x.x.tar.bz2 io-x.x.zip and unpack it using gunzip io-x.x.tar.gz; tar xvf io-x.x.tar respectively unzoo -x io-x.x.zoo and so on.

Do this in a directory called pkg, preferably (but not necessarily) in the pkg subdirectory of your &GAP; 4 installation. It creates a subdirectory called io.

To install this package do

cd io ./configure [path]

where path is a path to the main &GAP; root directory (if not given the default ../.. is assumed).

Afterwards call make to compile a binary file.

If you installed &GAP; on several architectures, you must execute this configure/make step on each of the architectures immediately after configuring &GAP; itself on this architecture.

The package will not work without this step.

If you installed the package in another pkg directory than the standard pkg directory in your &GAP; 4 installation, then you have to add the path to the directory containing your pkg directory to &GAP;'s list of directories. This can be done by starting &GAP; with the -l command line option followed by the name of the directory and a semicolon. Then your directory is prepended to the list of directories searched. Otherwise the package is not found by &GAP;. Of course, you can add this option to your &GAP; startup script.

Static linking

This might be interesting for M&dollar; Windows users, as dynamic loading of binary modules does not work there.

You can also create a new statically linked gap binary as follows: Go into the main &GAP; directory and then into bin/BINDIR. Here BINDIR means the directory containing the gap executable after compiling gap. This directory also contains the &GAP; compiler script gac. Assuming IO in the standard location you can then say

./gac -o gap-static -p "-DIOSTATIC" -P "-static" ../../pkg/io/src/io.c

Then copy your gap start script to, say, gapbig and change the references to the &GAP; binary to gap-static.

If you want to install more than one package with a C-part like this package, you can still create a statically linked &GAP; executable by combining all the compile and link options and all the .c files as in the ./gac command above. For the IO package, you have to add -DIOSTATIC to the string of the -p option and the file ../../pkg/io/src/io.c somewhere on the command line.

Recompiling the documentation

Recompiling the documentation is possible by the command gap makedoc.g in the io directory. But this should not be necessary.

Functions directly available from the C library

The following functions from the C library are made available as &GAP; functions:

accept
, bind, chdir, chmod, chown, close, closedir, connect, creat, dup, dup2, execv, execve, execvp, exit, fchmod, fchown, fcntl, fork, fstat, gethostbyname, getpid, getppid, getsockopt, kill, lchown, link, listen, lseek, lstat, mkdir, mkfifo, mknod, open, opendir, pipe, read, readdir, readlink, recv, recvfrom, rename, rewinddir, rmdir, seekdir, select, send, sendto, setsockopt, socket, stat, symlink, telldir, unlink, write.

Use the man command in your shell to get information about these functions.

For each of these functions there is a corresponding &GAP; global function with the prefix IO&uscore; before its name. Apart from minor differences (see below) they take exactly the same arguments as their C counterparts. Strings must be specified as &GAP; strings and integers as &GAP; immediate integers. Return values are in general the same as for the C counterparts. However, an error condition is indicated by the value fail instead of -1, and if the result can only be success or failure, true indicates success.

All errors are reported via the function.

In the C library a lot of integers are defined as macros in header files. All the necessary values for the above functions are bound to their name in the global IO record.

Warning:
Existence of many of these functions and constants is platform dependent. The compilation process checks existence and this leads to the situation that on the &GAP; levels the functions and constants are there or not. If you want to develop platform independent &GAP; code using this package, then you have to check for existence of the functions and constants you need.

Differences in arguments - an overview

The open function has to be called with three arguments. The version with two arguments is not available on the &GAP; level.

The read function takes four arguments: fd is an integer file descriptor, st is a &GAP; string, offset is an offset within this string (zero based), and count is the maximal number of bytes to read. The data is read and stored into the string st, starting at position offset+1. The string st is made long enough, such that count bytes would fit into it, beginning at position offset+1. The number of bytes read is returned or fail in case of an error.

The write function is similar, it also takes four arguments: fd is an integer file descriptor, st is a &GAP; string, offset is an offset within this string (zero based), and count is the number of bytes to write, starting from position offset+1 in the string st. The number of bytes written is returned, or a fail in case of an error.

The opendir function only returns true or fail.

The readdir function takes no argument. It reads the directory that was specified in the last call to opendir. It just returns a string, which is the name of a file or subdirectory in the corresponding directory. It returns false after the last file name in the directory or fail in case of an error.

The closedir function takes no argument. It should be called after readdir returned false or fail to avoid excessive use of file descriptors.

The functions stat, fstat, and lstat only take one argument and return a &GAP; record that has the same entries as a struct stat.

The function socket can optionally take a string as third argument. In that case it automatically calls getprotobyname to look up the protocol name.

The functions bind and connect take only one string argument as address field, because the string already encodes the length.

There are two convenience functions and to create such addresses. The first takes two arguments addr and port, where addr is a string of length 4, containing the 4 bytes of the IP address and port is a port number as &GAP; integer. The function takes the same arguments, but the first can be a string containing an IP address in dot notation like 137.226.152.77.

The setsockopt function has no argument optlen. The length of the string optval is taken.

The select function works as the function UNIXSelect in the &GAP; library.

As of now, the file locking mechanisms of fcntl using struct flock are not yet implemented on the &GAP; level.
The low-level functions in detail

Nearly all of this functions return an integer result in the C library. On the &GAP; level this is either returned as a non-negative integer in case of success or as fail in case of an error (where on the C level -1 would be returned). If the integer can only be 0 for no error this is changed to true on the &GAP; level.

an integer or fail
Accepts an incoming network connection. For details see man 2 accept. The argument addr can be made with and contains its length such that no third argument is necessary.
an integer or fail
Binds a local address to a socket. For details see man 2 bind. The argument my&uscore;addr can be made with and contains its length such that no third argument is necessary.
true
or fail
Changes the current working directory. For details see man 2 chdir.
true
or fail
Changes the mode of a file. For details see man 2 chmod.
true
or fail
Sets owner and/or group of file. For details see man 2 chown.
true
or fail
Closes a file descriptor. For details see man 2 close.
true
or fail
Closes a directory. For details see man 3 closedir. Has no arguments, because we only have one DIR struct in the C part.
true
or fail
Connects to a remote socket. For details see man 2 connect. The argument serv&uscore;addr can be made with and contains its length such that no third argument is necessary.
an integer or fail
Creates a new file. For details see man 2 creat.
an integer or fail
Duplicates a file descriptor. For details see man 2 dup.
true
or fail
Duplicates a file descriptor to a new one. For details see man 2 dup2.
fail
or does not return
Replaces the process with another process. For details see man 3 execv. The argument argv is a list of strings. The called program does not have to be the first argument in this list.
fail
or does not return
Replaces the process with another process. For details see man 3 execve. The arguments argv and envp are both lists of strings. The called program does not have to be the first argument in argv. The list envp can be made with from a record acquired from and modified later.
fail
or does not return
Replaces the process with another process. For details see man 3 execvp. The argument argv is a list of strings. The called program does not have to be the first argument in this list.
Stops process immediately with return code status. For details see man 2 exit. The argument status must be an integer. Does not return.
true
or fail
Changes mode of an opened file. For details see man 2 fchmod.
true
or fail
Changes owner and/or group of an opened file. For details see man 2 fchown.
an integer or fail
Does various things to control the behaviour of a file descriptor. For details see man 2 fcntl.
an integer or fail
Forks off a child process, which is an identical copy. For details see man 2 fork. Note that if you want to use the function to wait or check for the termination of child processes, you have to activate the SIGCHLD handler for this package beforehand by using the function . Note further that after that you cannot use the function any more, since its SIGCHLD handler does not work any more. To switch back to that functionality use the function .
a record or fail
Returns the file meta data for an opened file. For details see man 2 fstat. A &GAP; record is returned with the same entries than a struct stat.
a record or fail
Return host information by name. For details see man 3 gethostbyname. A &GAP; record is returned with all the relevant information about the host.
an integer
Returns the process ID of the current process as an integer. For details see man 2 getpid.
an integer
Returns the process ID of the parent of the current process as an integer. For details see man 2 getppid.
true
or false
Get a socket option. For details see man 2 getsockopt. Note that the argument optval carries its length around, such that no 5th argument is necessary.
true
or fail
Sends the signal sig to the process with process ID pid. For details see man 2 kill. The signal numbers available can be found in the global IO record with names like SIGTERM.
true
or false
Changes owner and/or group of a file not following links. For details see man 2 lchown.
true
or false
Create a hard link. For details see man 2 link.
true
or false
Switch a socket to listening. For details see man 2 listen.
an integer or fail
Seeks within an open file. For details see man 2 lseek.
a record or fail
Returns the file meta data for a file not following links. For details see man 2 lstat. A &GAP; record is returned with the same entries than a struct stat.
true
or false
Creates a directory. For details see man 2 mkdir.
true
or false
Creates a FIFO special file (a named pipe). For details see man 3 mkfifo.
true
or false
Create a special or ordinary file. For details see man 2 mknod.
an integer or fail
Open and possibly create a file or device. For details see man 2 open. Only the variant with 3 arguments can be used.
true
or false
Opens a directory. For details see man 3 opendir. Note that only true is returned if everything is OK, since only one DIR struct is stored on the C level and thus only one directory can be open at any time.
a record or fail
Create a pair of file descriptors with a pipe between them. For details see man 2 pipe. Note that no arguments are needed. The result is either fail in case of an error or a record with two components toread and towrite bound to the two filedescriptors for reading and writing respectively.
an integer or fail
Reads from file descriptor. For details see man 2 read. Note that there is one more argument offset to specify at which position in the string st the read data should be stored. Note that offset zero means at the beginning of the string, which is position 1 in &GAP;. The number of bytes read or fail in case of an error is returned.
a string or fail or false
Reads from a directory. For details see man 2 readdir. Note that no argument is required as we have only one DIR struct on the C level. If the directory is read completely false is returned, and otherwise a string. An error is indicated by fail.
an integer or fail
Reads the value of a symbolic link. For details see man 2 readlink. buf is modified. The new length of buf is returned or fail in case of an error.
an integer or fail
Receives data from a socket. For details see man 2 recv. Note the additional argument offset which plays the same role as for the function.
an integer or fail
Receives data from a socket with given address. For details see man 2 recvfrom. Note the additional argument offset which plays the same role as for the function. The argument addr can be made with and contains its length such that no 7th argument is necessary.
true
or false
Renames a file or moves it. For details see man 2 rename.
true
or fail
Rewinds a directory. For details see man 2 rewinddir. Note that no argument is required as we have only one DIR struct on the C level. Returns fail only, if no prior command has been called.
true
or fail
Removes an empty directory. For details see man 2 rmdir.
true
or fail
Sets the position of the next readdir call. For details see man 3 seekdir. Note that no second argument is required as we have only one DIR struct on the C level.
an integer or fail
Used for I/O multiplexing. For details see man 2 select. inlist, outlist and exclist are lists of filedescriptors, which are modified. If the corresponding file descriptor is not yet ready, it is replaced by fail.
an integer or fail
Sends data to a socket. For details see man 2 send. Note that the additional argument offset specifies the position of the data to send within the string st. It is zero based, meaning that zero indicates the start of the string, which is position 1 in &GAP;.
an integer or fail
Sends data to a socket. For details see man 2 sendto. Note that the additional argument offset specifies the position of the data to send within the string st. It is zero based, meaning that zero indicates the start of the string, which is position 1 in &GAP;. The argument addr can be made with and contains its length such that no 7th argument is necessary.
true
or fail
Sets a socket option. For details see man 2 setsockopt. Note that the argument optval carries its length around, such that no 5th argument is necessary.
an integer or fail
Creates a socket, an endpoint for communication. For details see man 2 socket. There is one little special: On systems that have getprotobyname you can pass a string as third argument protocol which is automatically looked up by getprotobyname.
a record or fail
Returns the file metadata for the file pathname. For details see man 2 stat. A &GAP; record is returned with the same entries than a struct stat.
true
or fail
Creates a symbolic link. For details see man 2 symlink.
an integer or fail
Return current location in directory. For details see man 3 telldir. Note that no second argument is required as we have only one DIR struct on the C level.
true
or fail
Delete a name and possibly the file it refers to. For details see man 2 unlink.
a record or fail
Waits for the termination of a child process. For details see man 2 waitpid. Returns a &GAP; record describing PID and exit status. The second argument wait must be either true or false. In the first case, the call blocks until new information about a terminated child process is available. In the second case no such waiting is performed, the call returns immediately. See .
an integer or fail
Writes to a file descriptor. For details see man 2 write. Note that the additional argument offset specifies the position of the data to send within the string st. It is zero based, meaning that zero indicates the start of the string, which is position 1 in &GAP;.
Further C level functions

The following functions do not correspond to functions in the C library, but are there to provide convenience to use other functions:

a string or fail
Makes a struct sockaddr&uscore;in from IP address and port. The IP address must be given as a string of length four, containing the four bytes of an IPv4 address in natural order. The port must be a port number. Returns a string containing the struct, which can be given to all functions above having an address argument.
a list of strings
For details see man environ. Returns the current environment as a list of strings of the form key=value.
true
or false
Installs our SIGCHLD handler. This functions works as an idempotent. That is, calling it twice does exactly the same as calling it once. It returns true when it is called for the first time since then a pointer to the old signal handler is stored in a global variable. See .
Restores the original SIGCHLD handler. This function works as an idempotent. That is, calling it twice does exactly the same as calling it once. It returns true when it is called for the first time after calling . See .
High level functions for buffered I/O

The functions in the previous sections are intended to be a possibility for direct access to the low level I/O functions in the C library. Thus, the calling conventions are strictly as in the original.

The functionality described in this section is implemented completely in the &GAP; language and is intended to provide a good interface for programming in &GAP;. The fundamental object for I/O on the C library level is the file descriptor, which is just a non-negative integer representing an open file of the process. The basic idea is to wrap up file descriptors in &GAP; objects that do the buffering.

Note that considerable care has been taken to ensure that one can do I/O multiplexing with buffered I/O. That is, one always has the possibility to make sure before a read or write operation, that this read or write operation will not block. This is crucial when one wants to serve more than one I/O channel from the same (single-threaded) &GAP; process. This design principle sometimes made it necessary to have more than one function for a certain operation. Those functions usually differ in a subtle way with respect to their blocking behaviour.

One remark applies again to nearly all functions presented here: If an error is indicated by the returned value fail one can use the library function to find out more about the cause of the error. This fact is not mentioned with every single function.

Types and the creation of File objects

The wrapped file objects are in the following category:

true
or false
The category of File objects.

To create objects in this category, one uses the following function:

a File object
The argument fd must be a file descriptor (i.e. an integer) or -1 (see below).

rbufsize can either be false for unbuffered reading or an integer buffer size or a string. If it is an integer, a read buffer of that size is used. If it is a string, then fd must be -1 and a File object that reads from that string is created.

wbufsize can either be false for unbuffered writing or an integer buffer size or a string. If it is an integer, a write buffer of that size is used. If it is a string, then fd must be -1 and a File object that appends to that string is created.

The result of this function is a new File object.

A convenient way to do this for reading or writing of files on disk is the following function: a File object or fail The argument filename must be a string specifying the path name of the file to work on. mode must also be a string with possible values r, w, or a, meaning read access, write access (with creating and truncating), and append access respectively. If mode is omitted, it defaults to r. bufsize, if given, must be a positive integer or false, otherwise it defaults to IO.DefaultBufSize. Internally, the function is used and the result file descriptor is wrapped using with bufsize as the buffer size.

The result is either fail in case of an error or a File object in case of success.

Note that there is a similar function which also creates a File object but with additional functionality with respect to a pipeline for filtering. It is described in its section in Section . There is some more low-level functionality to acquire open file descriptors. These can be wrapped into File objects using .

Reading and writing

Once a File object is created, one can use the following functions on it:

a string or fail
This function reads all data from the file f until the end of file. The data is returned as a &GAP; string. If the file is already at end of file, an empty string is returned. If an error occurs, then fail is returned. Note that you still have to call on the File object to properly close the file later.
a string or fail
This function gets two arguments, the first argument f must be a File object and the second argument len must be a positive integer. The function tries to read len bytes and returns a string of that length. If and only if the end of file is reached earlier, fewer bytes are returned. If an error occurs, fail is returned. Note that this function blocks until either len bytes are read, or the end of file is reached, or an error occurs. For the case of pipes or internet connections it is possible that currently no more data is available, however, by definition the end of file is only reached after the connection has been closed by the other side!
a string or fail
This function gets exactly one argument, which must be a File object f. It reads one line of data, where the definition of line is operating system dependent. The line end character(s) are included in the result. The function returns a string with the line in case of success and fail in case of an error. In the latter case, one can query the error with .

Note that the reading is done via the buffer of f, such that this function will be quite fast also for large amounts of data.

If the end of file is hit without a line end, the rest of the file is returned. If the file is already at end of file before the call, then a string of length 0 is returned. Note that this is not an error but the standard end of file convention!
a list of strings or fail
This function gets one or two arguments, the first of which must always be a File object f. It reads lines of data (where the definition of line is operating system dependent) either until end of file (without a second argument) or up to max lines (with a second argument max. A list of strings with the result is returned, if everything went well and fail oterwise. In the latter case, one can query the error with .

Note that the reading is done via the buffer of f, such that this function will be quite fast also for large amounts of data.

If the file is already at the end of file, the function returns a list of length 0. Note that this is not an error but the standard end of file convention!
true
or false
This function takes one argument f which must be a File object. It returns true or false according to whether there is data to read available in the file f. A return value of true guarantees that the next call to on that file will succeed without blocking and return at least one byte or an empty string to indicate the end of file.
a string or fail
The function gets two arguments, the first of which must be a File object f. The second argument must be a positive integer. The function reads data up to len bytes. A string with the result is returned, if everything went well and fail otherwise. In the latter case, one can query the error with .

Note that the reading is done via the buffer of f, such that this function will be quite fast also for large amounts of data.

If the file is already at the end of the file, the function returns a string of length 0. Note that this is not an error!

If a previous call to or to indicated that there is data available to read, then it is guaranteed that the function does not block and returns at least one byte if the file is not yet at end of file and an empty string otherwise.
an integer or fail
This function can get an arbitrary number of arguments, the first of which must be a File object f. All the other arguments are just written to f if they are strings. Otherwise, the String function is called on them and the result is written out to f.

Note that the writing is done buffered. That is, the data is first written to the buffer and only really written out after the buffer is full or after the user explicitly calls on f.

The result is either the number of bytes written in case of success or fail in case of an error. In the latter case the error can be queried with .

Note that this function blocks until all data is at least written into the buffer and might block until data can be sent again if the buffer is full.
an integer or fail
Behaves like but works on a single string line and sends an (operating system dependent) end of line string afterwards. Also is called automatically after the operation, such that one can be sure, that the data is actually written out after the function has completed.
an integer or fail
Behaves like but works on a list of strings list and sends an (operating system dependent) end of line string after each string in the list. Also is called automatically after the operation, such that one can be sure, that the data is actually written out after the function has completed.
true
or fail
This function gets one argument f, which must be a File object. It writes out all the data that is in the write buffer. This is not necessary before the call to the function , since that function calls automatically. However, it is necessary to call after calls to to be sure that the data is really sent out. The function returns true if everything goes well and fail if an error occurs.

Remember that the functions and implicitly call after they are done.

Note that this function might block until all data is actually written to the file descriptor.
an integer or fail
This function behaves like followed by a call to . It returns either the number of bytes written or fail if an error occurs.
true
or false
This function takes one argument f which must be a File object. It returns true or false according to whether the file f is ready to write. A return value of true guarantees that the next call to on that file will succeed without blocking and accept at least one byte.
an integer or fail
This function takes four arguments. The first one f must be a File object, the second st a string, and the arguments pos and len must be integers, such that positions pos+1 until pos+len are bound in st. The function tries to write up to len bytes from st from position pos+1 to the file f. If a previous call to or to indicates that f is writable, then it is guaranteed that the following call to will not block and accept at least one byte of data. Note that it is not guaranteed that all len bytes are written. The function returns the number of bytes written or fail if an error occurs.
true
or false
This function takes one argument f which must be a File object. It returns true or false according to whether the file f is ready to flush. A return value of true guarantees that the next call to on that file will succeed without blocking and flush out at least one byte. Note that this does not guarantee, that this call succeeds to flush out the whole content of the buffer!
true
, false, or fail
This function takes one argument f which must be a File object. It tries to write all data in the writing buffer to the file descriptor. If this succeeds, the function returns true and false otherwise. If an error occurs, fail is returned. If a previous call to or indicated that f is flushable, then it is guaranteed that the following call to does not block. However, it is not guaranteed that true is returned from that call.
true
or fail
This function closes the File object f after writing all data in the write buffer out and closing the file descriptor. All buffers are freed. In case of an error, the function returns fail and otherwise true. Note that for pipes to other processes this function collects data about the terminated processes using .
Other functions
an integer
This function returns the real file descriptor that is behind the File object f.
a string or false
This function gets one argument f which must be a File object and returns the writing buffer of that File object. This is necessary for File objects, that are not associated to a real file descriptor but just collect everything that was written in their writing buffer. Remember to use this function before closing the File object.
an integer or fail
This function is the corresponding function to for buffered file access. It behaves similarly to that function. The differences are the following: There are four lists of files r, w, f, and e. They all can contain either integers (standing for file descriptors) or File objects. The list r is for checking, whether files or file descriptors are ready to read, the list w is for checking whether they are ready to write, the list f is for checking whether they are ready to flush, and the list e is for checking whether they have exceptions.

For File objects it is always first checked, whether there is either data available in a reading buffer or space in a writing buffer. If so, they are immediately reported to be ready (this feature makes the list of File objects to test for flushability necessary). For the remaining files and for all specified file descriptors, the function is called to get an overview about readiness. The timeout values t1 and t2 are set to zero for immediate returning if one of the requested buffers were ready.

returns the number of files or file descriptors that are ready to serve or fail if an error occurs.

The following function is a convenience function for directory access:

a list of strings or fail
This function gets a string containing a path name as single argument and returns a list of strings that are the names of the files in that directory, or fail, if an error occurred.

The following function is used to create strings describing a pair of an IP address and a port number in a binary way. These strings can be used in connection with the C library functions connect, bind, recvfrom, and sendto for the arguments needing such address pairs.

a string
This function gets a string ipstring containing an IP address in dot notation, i.e. four numbers in the range from 0 to 255 separated by dots ., and an integer portnr, which is a port number. The result is a string of the correct length to be used for the low level C library functions, wherever IP address port number pairs are needed.
a record or fail
Takes no arguments, uses to get the environment and returns a record in which the component names are the names of the environment variables and the values are the values. This can then be changed and the changed record can be given to to produce again a list which can be used for as third argument.
a list of strings
Takes a record as returned by and turns it into a list of strings as needed by as third argument.
Inter process communication
fail
or the path to an executable
If the path name path contains a slash, this function simply checks whether the string path refers to an executable file. If so, path is returned as is. Otherwise, fail is returned. If the path name path does not contain a slash, all directories in the environment variable PATH are searched for an executable with name path. If so, the full path to that executable is returned, otherwise fail.

This function is used whenever one of the following functions gets an argument that should refer to an executable.
nothing
Closes all file descriptors except those listed in exceptions, which must be a list of integers.
a File object or fail
The argument path must refer to an executable file in the sense of .

Starts a child process using the executable in path with either stdout or stdin being a pipe. The argument mode must be either the string r or the string w.

In the first case, the standard output of the child process will be the writing end of a pipe. A File object for reading connected to the reading end of the pipe is returned. The standard input and standard error of the child process will be the same than the calling &GAP; process.

In the second case, the standard input of the child process will be the reading end of a pipe. A File object for writing connected to the writing end of the pipe is returned. The standard output and standard error of the child process will be the same than the calling &GAP; process.

In case of an error, fail is returned.

The process will usually die, when the pipe is closed, but can also do so without that. The File object remembers the process ID of the started process and the function then calls for it to acquire information about the terminated process.

Note that activates our SIGCHLD handler (see ).

In either case the File object will have the attribute ProcessID set to the process ID of the child process.
a record or fail
The argument path must refer to an executable file in the sense of .

A new child process is started using the executable in path The standard input and standard output of it are pipes. The writing end of the input pipe and the reading end of the output pipe are returned as File objects bound to two components stdin and stdout (resp.) of the returned record. This means, you have to write to stdin and read from stdout in the calling &GAP; process. The standard error of the child process will be the same as the one of the calling &GAP; process.

Returns fail if an error occurred.

The process will usually die, when one of the pipes is closed. The File objects remember the process ID of the called process and the function call to for the stdout object will call for it to acquire information about the terminated process.

Note that activates our SIGCHLD handler (see ).

Both File objects will have the attribute ProcessID set to the process ID of the child process, which will also be bound to the pid component of the returned record.
a record or fail
The argument path must refer to an executable file in the sense of .

A new child process is started using the executable in path The standard input, standard output, and standard error of it are pipes. The writing end of the input pipe, the reading end of the output pipe and the reading end of the error pipe are returned as File objects bound to two components stdin, stdout, and stderr (resp.) of the returned record. This means, you have to write to stdin and read from stdout and stderr in the calling &GAP; process.

Returns fail if an error occurred.

The process will usually die, when one of the pipes is closed. All three File objects will remember the process ID of the newly created process and the call to the function for the stdout object will call for it to acquire information about the terminated child process.

Note that activates our SIGCHLD handler (see ).

All three File objects will have the attribute ProcessID set to the process ID of the child process, which will also be bound to the pid component of the returned record.
a record or fail
The argument progs is a list of pairs, the first entry being a path to an executable (in the sense of ), the second an argument list, the argument infd is an open file descriptor for reading, outfd is an open file descriptor for writing, both can be replaced by the string open in which case a new pipe will be opened. The argument switcherror is a boolean indicating whether standard error channels are also switched to the corresponding output channels.

This function starts up all processes and connects them with pipes. The input of the first is switched to infd and the output of the last to outfd.

Returns a record with the following components: pids is a list of process ids if everything worked. For each process for which some error occurred the corresponding pid is replaced by fail. The stdin component is equal to false, or to the file descriptor of the writing end of the newly created pipe which is connected to the standard input of the first of the new processes if infd was open. The stdout component is equal to false or to the file descriptor of the reading end of the newly created pipe which is connected to the standard output of the last of the new processes if outfd was open.

Note that the SIGCHLD handler of the `IO` package is installed by this function (see ) and that it lies in the responsibility of the caller to use to ask for the status information of all child processes after their termination.
a string or fail
Reads the file with the name filename, however, a pipeline is created by the processes described by progs (see ) to filter the content of the file through the pipeline. The result is put into a &GAP; string and returned. If something goes wrong, fail is returned.
a string or fail
Writes the content of the string st to the file with the name filename, however, a pipeline is created by the processes described by progs (see ) to filter the content of the string through the pipeline. The result is put into the file. If the boolean value append is given and equal to true, then the data will be appended to the already existing file. If something goes wrong, fail is returned.
a File object or fail
This function is similar to and behaves nearly identically. The only difference is that a filtering pipeline is switched between the file and the File object such that all things read or written respectively are filtered through this pipeline of processes.

The File object remembers the started processes and upon the final call to automatically uses the function to acquire information from the terminated processes in the pipeline after their termination. This means that you do not have to call any more after the call to .

Note that activates our SIGCHLD handler (see ).

The File object will have the attribute ProcessID set to the list of process IDs of the child processes.
This functions uses to write the whole string st to the File object f. However, this is done by forking off a child process identical to the calling &GAP; process that does the sending. The calling &GAP; process returns immediately, even before anything has been sent away with the result true. The forked off sender process terminates itself immediately after it has sent all data away.

The reason for having this function available is the following: If one uses or to start up a child process with standard input and standard output being a pipe, then one usually has the problem, that the child process starts reading some data, but then wants to write data, before it received all data coming. If the calling &GAP; process would first try to write all data and only start to read the output of the child process after sending away all data, a deadlock situation would occur. This is avoided with the forking and backgrounding approach.

Remember to close the writing end of the standard input pipe in the calling &GAP; process directly after has returned, because otherwise the child process might not notice that all data has arrived, because the pipe persists! See the file popen2.g in the example directory for an example.

Note that with most modern operating systems the forking off of an identical child process does in fact not mean a duplication of the total main memory used by both processes, because the operating system kernel will use copy on write. However, if a garbage collection happens to become necessary during the sending of the data in the forked off sending process, this might trigger doubled memory usage.
a string or fail
Starts the process with the executable given by the file name cmd (in the sense of ) with arguments in the argument list args (a list of strings). The standard input and output of the started process are connected via pipes to the calling process. The content of the string input is written to the standard input of the called process and its standard output is read and returned as a string.

All the necessary I/O multiplexing and non-blocking I/O to avoid deadlocks is done in this function.

This function properly does to wait for the termination of the child process but does not restore the original &GAP; SIGCHLD signal handler (see ).
a record or fail
Starts the process with the executable given by the file name cmd (in the sense of ) with arguments in the argument list args (a list of strings). The standard input, output and error of the started process are connected via pipes to the calling process. The content of the string input is written to the standard input of the called process and its standard output and error are read and returned as a record with components out and err, which are strings.

All the necessary I/O multiplexing and non-blocking I/O to avoid deadlocks is done in this function.

This function properly does to wait for the termination of the child process but does not restore the original &GAP; SIGCHLD signal handler (see ).

The functions returns either fail if an error occurred, or otherwise a record with components out and err which are bound to strings containing the full standard output and standard error of the called process.
Object serialisation (Pickling)

The idea of object serialisation is that one wants to store nearly arbitrary &GAP; objects to disk or transfer them over the network. To this end, one wants to convert them to a byte stream that is platform independent and can later be converted back to a copy of the same object in memory, be it in the same &GAP; process or another one maybe even on another machine. The main problem here are the vast amount of different types occurring in &GAP; and the possibly highly self-referential structure of &GAP; objects.

The `IO` package contains a framework to implement object serialisation and implementations for most of the basic data types in &GAP;. The framework is easily extendible to other types and takes complete care of self-references and corresponding problems. It builds upon the buffered I/O functions described in Section . We start by describing the user interface.

Result objects

The following static objects are used to report about success or failure of the (un-)pickling operations:

This object is returned if an error occurs.
This object is returned when there is nothing to return, for example if an unpickler (see ) encounters the end of a file.
This object is returned if everything went well and there is no other canonical value to return to indicate this.

The only thing you can do with these special values is to compare them to each other and to other objects.

Pickling and unpickling
IO&uscore;OK
or IO&uscore;Error
The argument f must be an open, writable File object. The object ob can be an arbitrary &GAP; object. The operation pickles or serialises the object ob and writes the result into the File object f. If everything is OK, the unique value IO&uscore;OK is returned and otherwise the unique value IO&uscore;Error. The resulting byte stream can be read again using the operation and is platform- and architecture independent. Especially the question whether a system has 32 bit or 64 bit wide words and the question of endianess does not matter.

Note that not all of &GAP;'s object types are supported but it is relatively easy to extend the system. This package supports in particular boolean values, integers, permutations, rational numbers, finite field elements, cyclotomics, strings, polynomials, rational functions, lists, records, compressed vectors and matrices over finite fields (objects are uncompressed in the byte stream but recompressed during unpickling), and straight line programs.

Self-referential objects built from records and lists are handled correctly and are restored completely with the same self-references during unpickling.
IO&uscore;Error
or a &GAP; object
The argument f must be an open, readable File object. The operation reads from f and unpickles the next object. If an error occurs, the unique value IO&uscore;Error is returned. If the File object is at end of file, the value IO&uscore;Nothing is returned. Note that these two values are not picklable, because of their special meaning as return values of this operation here.
Nothing
This function clears the pickle cache. This cache stores all object pickled in the current recursive call to and is necessary to handle self-references. Usually it is not necessary to call this function explicitly. Only in the rare case (that should not happen) that a pickling or unpickling operation enters a break loop which is left by the user, the pickle cache has to be cleared explicitly using this function for later calls to and to work!
Extending the pickling framework

The framework can be extended for other &GAP; object types as follows:

For pickling, a method for the operation has to be installed which does the work. If the object to be pickled has subobjects, then the first action of the method is to call the function IO&uscore;AddToPickled with the object as argument. This will put it into the pickle cache and take care of self-references. Arbitrary subobjects can then be pickled using recursive calls to the operation handing down the same File object into the recursion. The method must either return IO&uscore;Error in case of an error or IO&uscore;OK if everything goes well. Before returning, a method that has called IO&uscore;AddToPickled must call the function IO&uscore;FinalizePickled without arguments under all circumstances. If this call is missing, global data for the pickling procedure becomes corrupt!

Every pickling method must first write a 4 byte magic value such that later during unpickling of the byte stream the right unpickling method can be called (see below). Then it can write arbitrary data, however, this data should be platform- and architecture independent, and it must be possible to unpickle it later without lookahead.

Pickling methods should usually not go into a break loop, because after leaving the user has to call explicitly!

Unpickling is implemented as follows: For every 4 byte magic value there must be a function bound to that value in the record IO&uscore;Unpicklers. If the unpickling operation encounters that magic value, it calls the corresponding unpickling function. This function just gets one File object as argument. Since the magic value is already read, it can immediately start with reading and rebuilding the serialised object in memory. The method has to take care to restore the object including its type completely.

If an object type has subobjects, the unpickling function has to first create a skeleton of the object without its subobjects, then call IO&uscore;AddToUnpickled on this skeleton, before unpickling subobjects. If things are not done in this order, the handling of self-references down in the recursion will not work! An unpickling function that has called IO&uscore;AddToUnpickled at the beginning has to call IO&uscore;FinalizeUnpickled without arguments before returning under all circumstances! If this call is missing, global data for the unpickling procedure becomes corrupt!

Of course, unpickling functions can recursively call to unpickle subobjects. Apart from this, unpickling functions can use arbitrary reading functions on the File object. However, they should only read sequentially and never move the current file position pointer otherwise. An unpickling function should return the newly created object or the value IO&uscore;Error if an error occurred. They should never go into a break loop, because after leaving the user has to call explicitly!

Perhaps the best way to learn how to extend the framework is to study the code for the basic &GAP; objects in the file pkg/io/gap/pickle.gi.

Really random sources

This section describes so called real random sources. It is an extension to the library mechanism of random source objects that uses the devices /dev/random and /dev/urandom available on Linux systems (and maybe on other operating systems) providing random numbers that are impossible to predict. The idea is that such sources of random numbers are useful to produce unpredictable secret keys for cryptographic applications.

The functions
a real random source object or fail
The first argument r must be the &GAP; filter IsRealRandomSource and the second either the string random or the string urandom. A real random source object is created that draws its random numbers from the kernel devices /dev/random and /dev/urandom respectively. Whereas /dev/random always provides random numbers of not guaranteed quality, the device /dev/urandom measures its entropy and produces guaranteed unpredictable numbers. However, it might block until enough random events (like mouse movements) have been accumulated.
A client side implementation of the HTTP protocol

The `IO` package contains an implementation of the client side of the HTTP protocol. The basic purpose of this is of course to be able to download data from web servers from the &GAP; language. However, the HTTP protocol can perform a much bigger variety of tasks.

Functions for client side HTTP
a record
The first argument hostname must be a string containing the hostname of the server to connect. The second argument port must be an integer in the range from 1 to 65535 and describes the port to connect to on the server.

The function opens a TCP/IP connection to the server and returns a record conn with the following components: conn.sock is fail if an error occurs and otherwise a File object linked to the file descriptor of the socket. In case of an error, the component conn.errormsg contains an error message, it is otherwise empty. If everything went well then the component conn.host is the result from the host name lookup (see ) and the component conn.closed is set to false.

No data is sent or received on the socket in this function.
a record
This function performs a complete HTTP request. The first argument must be a connection record as returned by a successful call to . The argument method must be a valid HTTP request method in form of a string. The most common will be GET, POST, or HEAD. The argument uri is a string containing the URI of the request, which is given in the first line of the request. This will usually be a relative or absolute path name given to the server. The argument header must be a &GAP; record. Each bound field of this record will we transformed into one header line with the name of the component being the key and the value the value. All bound values must be strings. The argument body must either be a string or false. If it is a string, this string is sent away as the body of the request. If no string or an empty string is given, no body will be sent. The header field Content-Length is automatically created from the length of the string body. Finally, the argument target can either be false or a string. In the latter case, the body of the request answer is written to the file with the name given in target. The body component of the result will be the file name in this case. If target is false, the full body of the answer is stored into the body component of the result.

The function sends away the request and awaits the answer. If anything goes wrong during the transfer (for example if the connection is broken prematurely), then the component statuscode of the resulting record is 0 and the component status is a corresponding error message. In that case, all other fields may or may not be bound to sensible values, according to when the error occurred. If everything goes well, then statuscode and status are bound to the corresponding values coming from the request answer. statuscode is transformed into a &GAP; integer. The header of the answer is parsed, transformed into a &GAP; record, and stored into the component header of the result. The body component of the result record is set as described above. Finally, the protoversion component contains the HTTP protocol version number used by the server as a string and the boolean value closed indicates, whether or not the function has detected, that the connection has been closed by the server. Note that by default, the connection will stay open, at least for a certain time after the end of the request.

See the description of the global variable for rules how timeouts are done in this function.

Note that if the method is HEAD, then no body is expected (none will be sent anyway) and the function returns immediately with empty body. Of course, the Content-Length value in the header is as if it the request would be done with the GET method.
This global variable holds a list of length two. By default, both entries are fail indicating that should never timeout and wait forever for an answer. Actually, the two values in this variable are given to the function call during I/O multiplexing. That is, the first number is in seconds and the second in milliseconds. Together they lead to a timeout for the HTTP request. If a timeout occurs, an error condition is triggered which returns a record with status code 0 and status being the timeout error message.

You can change the timeout by accessing the two entries of this write protected list variable directly.
nothing
Closes the connection described by the connection record conn. No error can possibly occur.
a record
The arguments are as the corresponding ones in the functions and respectively. This function opens an HTTP connection, tries a single HTTP request and immediately closes the connection again. The result is as for the function. If an error occurs during the opening of the connection, the statuscode value of the result is 0 and the error message is stored in the status component of the result.

The previous function allows for a very simple implementation of a function that checks, whether your current &GAP; installation is up to date:

nothing
This function accesses a web page in St. Andrews and runs some &GAP; code from there. This code knows all the currently released versions of &GAP; and its packages. It prints out a summary and possibly suggests upgrades. If you do not want to executed code downloaded from the internet, then do not call this function.

More concretely, the page accessed is http://www.gap-system.org/Download/upgrade.html and the code executed is a single call to the function SuggestUpgrades function in the &GAP; library.
Examples of usage

For larger examples see the example directory of the package. You find there a small server using the TCP/IP protocol and a corresponding client and another small server using the UDP protocol and a corresponding client.

Further, there is an example for the usage of File objects, that read from or write to strings.

Another example there shows starting up a child process and piping a few megabytes through it using .

In the following, we present a few explicit, interactive short examples for the usage of the functions in this package. Note that you have to load the `IO` package with the command LoadPackage("IO"); before trying these examples.

Writing and reading a file

The following sequence of commands opens a file with name guck and writes some things to it:

gap> f := IO_File("guck","w"); <file fd=3 wbufsize=65536 wdata=0> gap> IO_Write(f,"Hello world\n"); 12 gap> IO_WriteLine(f,"Hello world2!"); 14 gap> IO_Write(f,12345); 5 gap> IO_Flush(f); true gap> IO_Close(f); true

There is nothing special about this, the numbers are numbers of bytes written. Note that only after the command the data is actually written to disk. Before that, it resides in the write buffer of the file. Note further, that the call here would not have been necessary, since the call flushes the buffer anyway.

The file can again be read with the following sequence of commands:

gap> f := IO_File("guck","r"); <file fd=3 rbufsize=65536 rpos=1 rdata=0> gap> IO_Read(f,10); "Hello worl" gap> IO_ReadLine(f); "d\n" gap> IO_ReadLine(f); "Hello world2!\n" gap> IO_ReadLine(f); "12345" gap> IO_ReadLine(f); "" gap> IO_Close(f); true

Note here that reading line-wise can only be done efficiently by using buffered I/O. You can mix calls to and to . The end of file is indicated by an empty string returned by one of the read functions.
Using filtering programs to read and write files

If you want to write a big amount of data to file you might want to compress it on the fly without using much disk space. This can be achieved with the following command:

gap> s := "";; for i in [1..10000] do Append(s,String(i)); od;; gap> Length(s); 38894 gap> IO_FileFilterString("guck.gz",[["gzip",["-9c"]]],s); true gap> sgz := StringFile("guck.gz");; gap> Length(sgz); 18541 gap> ss := IO_StringFilterFile([["gzip",["-dc"]]],"guck.gz");; gap> s=ss; true

This sequence of commands needs that the program gzip is installed on your system.
Using filters when reading or writing files sequentially

If you want to process bigger amounts of data you might not want to store all of it in a single &GAP; string. In that case you might want to access a file on disk sequentially through a filter:

gap> f := IO_FilteredFile([["gzip",["-9c"]]],"guck.gz","w"); <file fd=5 wbufsize=65536 wdata=0> gap> IO_Write(f,"Hello world!\n"); 13 gap> IO_Write(f,Elements(SymmetricGroup(5)),"\n"); 1359 gap> IO_Close(f); true gap> f := IO_FilteredFile([["gzip",["-dc"]]],"guck.gz","r"); <file fd=4 rbufsize=65536 rpos=1 rdata=0> gap> IO_ReadLine(f); "Hello world!\n" gap> s := IO_ReadLine(f);; Length(s); 1359 gap> IO_Read(f,10); "" gap> IO_Close(f); true
Accessing a web page

The `IO` package has an HTTP client implementation. Using this you can access web pages and other web downloads from within &GAP;. Here is an example:

gap> r := SingleHTTPRequest("www.math.rwth-aachen.de",80,"GET", > "/~Max.Neunhoeffer/index.html",rec(),false,false);; gap> RecFields(r); [ "protoversion", "statuscode", "status", "header", "body", "closed" ] gap> r.status; "OK" gap> r.statuscode; 200 gap> r.header; rec( date := "Thu, 07 Dec 2006 22:08:22 GMT", server := "Apache/2.0.55 (Ubuntu)", last-modified := "Thu, 16 Nov 2006 00:21:44 GMT", etag := "\"2179cf-11a5-3c77f600\"", accept-ranges := "bytes", content-length := "4517", content-type := "text/html; charset=ISO-8859-1" ) gap> Length(r.body); 4517

Of course, the time stamps and exact sizes of the answer may differ when you do this.
(Un-)Pickling

Assume you have some &GAP; objects you want to archive to disk grouped together. Then you might do the following:

gap> r := rec( a := 1, b := "Max", c := [1,2,3] ); rec( a := 1, b := "Max", c := [ 1, 2, 3 ] ) gap> r.c[4] := r; rec( a := 1, b := "Max", c := [ 1, 2, 3, ~ ] ) gap> f := IO_File("guck","w"); <file fd=3 wbufsize=65536 wdata=0> gap> IO_Pickle(f,r); IO_OK gap> IO_Pickle(f,[(1,2,3,4),(3,4)]); IO_OK gap> IO_Close(f); true

Then, to read it in again, just do:

gap> f := IO_File("guck"); <file fd=3 rbufsize=65536 rpos=1 rdata=0> gap> IO_Unpickle(f); rec( a := 1, b := "Max", c := [ 1, 2, 3, ~ ] ) gap> IO_Unpickle(f); [ (1,2,3,4), (3,4) ] gap> IO_Unpickle(f); IO_Nothing gap> IO_Close(f); true

Note that this works for a certain amount of builtin objects. If you want to archive your own objects or more sophisticated objects you have to use extend the functionality as explained in Section . However, it works for lists and records and they may be arbitrarily self-referential.

License

This package is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; version 2 or greater of the License.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA