Home

GEONius.com
5-Jul-2016
E-mail

nft_util - FTP Server Framework


Background

The Software

 

Background

The NFT_UTIL package provides a flexible framework for implementing the server side of an FTP client-server session. The session begins when a client establishes a TCP/IP control connection to the FTP server. At that point, the server creates an NftSession object, which essentially layers the FTP protocol on top of the raw TCP/IP network connection. When the session object is created, the server can register functions to process the different FTP commands the client may send. The session ends when the server receives the "QUIT" command from the client or if the control connection is broken.

You may be thinking, "What's the big deal? My computer already comes with a standard FTP server." Back in 1995 or 1996, I worked at NASA's Goddard Space Flight Center in Code 521, the Microelectronics Systems Branch, and we needed a non-standard FTP server. Our system was responsible for receiving and processing 50-Mbps data downlinks from satellites in real-time. To do so, the hardware/design engineers in Code 521 designed application-specific integrated circuits (ASICs) to receive, decode, and process this high-throughput stream of CCSDS packets. The ASICs were hosted on VME boards and controlled by software running under VxWorks. This, the first step in the processing of satellite data, is known as "Level 0 Processing" (LZP).

On one particular part of the project, the data was stored on a hard-disk farm which had a high-speed HIPPI interface. One 680x0 CPU board controlled the disk farm. Dave Kim, a good friend, wrote the software, KDOS, that ran on the CPU board and provided a file-system view of the disk farm. An FTP server was needed to allow clients on Unix workstations to retrieve data from the disk farm. Dave's CPU board apparently didn't have a network interface, so the FTP server ran on an adjacent CPU board that did have a network interface. The only way to communicate between the two boards was through shared-RAM-based mailboxes and data buffers.

Inspired by one of Dave's programs, I wrote a KDOS daemon, kdosd, which ran on the disk farm's CPU board. It would read ASCII text mailbox messages from the FTP board and perform the requested file operations on the disk farm using Dave's libkdos library. An example request might be to read so many bytes from a file and store the data in a buffer accessible to both CPU boards.

On the FTP board, we ran nafta, an FTP server I wrote using the NFT_UTIL package. (Following in the footsteps of YACC, NAFTA stands for "Not Another File Transfer Application"!) When a client workstation connected to nafta, the server would create a new NftSession object and register KDOS-specific functions to be evaluated for the different FTP commands. For example, to get a file from the disk farm (FTP's "RETR" or retrieve command), nafta's "RETR"-processing function would send an "OPEN" message to kdosd on the disk farm board to open the requested file, a series of "READ" messages to read the file's contents and store them in a data buffer accessible to both boards, and a final "CLOSE" message to close the file. Once the file's contents were available on the board, nafta could then forward them over the network to the FTP client to complete the "RETR" request. (To indirectly perform the KDOS file operations using mailbox messages, nafta, for which I no longer have the source code, probably used the KDOS Asynchronous I/O Utilities.)

Back in the late 1990s, I also used the Unix FTP version of nafta, but later replaced it with anise, my pocket WWW/FTP server, which uses the NFT_UTIL package for FTP. The nftPASS() function handled user authentication under SunOS 4.1.3 and VxWorks, but I've not updated it (or researched what needs to be updated) since the 1990s. So, the NFT_UTIL package basically allows free access to its FTP server.

In 2016, I tried out anise's FTP server, which hadn't been exercised in years. I noticed and fixed a bug in the "LIST" command (incorrect file sizes and timestamps). I also changed NFT_UTIL to use a simple array to store command-function mappings instead of a hash table; modern compilers now complain (correctly) that a function (code) pointer cannot be converted to and from a (void *) data pointer, as was being done when I used hash tables. In the process, I reacquainted myself with how the NftSession object works and I got to thinking about one shortcoming that became apparent: when a session object is created, the server has to register any application-specific command-processing functions and set other server-specific parameters. To do so, the server has to maintain some state (whether in data or code form) outside the NftSession object that tells it, the server, how to initialize the session object when a new client establishes a connection. I suppose it would be a good idea to design an NftServer object that holds the command-function mappings and other parameters for its impermanent, child NftSession objects. Oh, well ...

(Dave Kim was a great person to work with. He was one of the few people I've known in a long software career—I think I can count them on the fingers of one hand—who build up their own personal software libraries; i.e., functions, tools, etc. that help them be more productive than others who depend solely on a project's official libraries or tools. In the non-technical realm, I've loved sailboats ever since I was a young kid; I've only ever been sailing once, on the Chesapeake Bay, and, like the landlubber I am at heart and stomach, I got royally seasick. Despite that, I continued to love sailboats and, in Dave Kim, I met a real, live, racing sailor—no cushy cruising for him! I learned a lot about sailing, particularly when racing, from him and I enjoyed his boating stories. He also lent me his copy of The Voyage of American Promise by Dodge Morgan, the first American to sail solo around the globe with no stops. "It takes three things to sail around the world alone. A good boat, an iron will and luck... And, my friends, here is a great boat." A fascinating book, not least because Morgan showed that you don't get sick when you're not around other people, even if you're continually drenched in ice-cold water!)

 

The Software

The NFT_UTIL package provides the basis for implementing a File Transfer Protocol (FTP) server. The implementation of the NFT_UTIL package itself and its companion package of command processing functions, nft_proc.c, was based on the following Request for Comments (available at a number of Internet sites):

RFC 765 - File Transfer Protocol (obsolete)
Although superseded by RFC 959, this RFC described the (defunct?) mail-related FTP commands: MAIL, MLFL, MSAM, MSOM, MSRQ, and MRCP.

RFC 959 - File Transfer Protocol (FTP)
The official FTP RFC, well-written, not dry.

RFC 1123 - Requirements for Internet Hosts -- Application and Support
This all-encompassing RFC clarified some remaining issues in the FTP standard and took into account existing practice.

as well as some empirical testing with the SunOS 4.1.3 and HP/UX 9.05 FTP servers.

FTP Sessions

An FTP server listens for and accepts network connection requests from clients who wish to transfer files. A session for a particular client begins when that client first connects to the FTP server and ends when the client is disconnected from the server; an FTP server with multiple clients would have multiple sessions active simultaneously. Associated with each session are two network connections:

Control - is the connection over which commands are sent to the FTP server and replies returned to the client. The control connection stays open for the life of the session.
Data - is a connection over which files and other data are sent and received. A new data connection is established for each data transfer (e.g., the sending of a single file) and closed when the transfer completes.

FTP commands are CR/LF-terminated, ASCII strings consisting of an upper case, 3- or 4-character keyword followed by zero or more, space-separated arguments; for example, the following command requests the retrieval of a file:

    RETR thisFile

Some of the more common FTP commands are:

USER userName (Automatically issued by the FTP client)
PASS password
log a user onto the FTP server's host.

CWD newDirectory (FTP Client Command: cd newDirectory)
CDUP (FTP Client Command: cd ..)
change the current working directory. CDUP moves up to the parent of the current working directory.

PWD (FTP Client Command: pwd)
returns the current working directory.

NLST [directory] (FTP Client Command: ls)
returns a list of the files in the specified directory (which defaults to the current working directory).

TYPE A|I (FTP Client Command: ascii or binary)
specifies the type of data being transferred, "A" for ASCII and "I" (Image) for binary.

PORT hostAndPort (Automatically issued by the FTP client)
specifies a port on the client's host to which the FTP server should connect for data transfers. This command is usually issued anew prior to each file transfer or directory listing.

RETR fileName (FTP Client Command: get fileName)
retrieves the specified file from the FTP server's host.

STOR fileName (FTP Client Command: put fileName)
store the to-be-transferred data on the FTP server's host under the specified file name.

QUIT (FTP Client Command: bye or quit)
terminates the session.

FTP replies consist of a 3-digit, numeric status code followed by descriptive text. The status codes are enumerated in RFC 959 (but see RFC 1123 for some updates). For example, the RETR command typically results in two replies being returned to the client over the control connection:

    "150 Data connection opened: thisFile (98765 bytes)"

    ... the contents of "thisFile" are sent over the data connection ...

    "226 Transfer complete: thisFile (98765 bytes)

RFCs 959 and 1123 specify what status codes should be used in reply to which commands. Although there are a few exceptions, the implementor is free to choose the format and contents of the reply text.

The NFT Package

The NFT package provides a server implementation with a high-level means of conducting an FTP session. The server is responsible for listening for and answering a network connection request from a client:

    TcpEndpoint  client, server ;
    ...
    tcpListen ("ftp", 99, &server) ;
    tcpAnswer (server, -1.0, &client) ;

Once a client connection has been established, an FTP session can be created:

    NftSession  session ;
    ...
    nftCreate (client, NULL, NULL, NULL, NULL, &session) ;

The server is now ready to read and process FTP commands from the client:

    char  *command ;
    ...
    nftGetLine (session, &command) ;
    nftEvaluate (session, command) ;

When the client connection is broken or an FTP QUIT command is received, the server should terminate the FTP session:

    nftDestroy (session) ;

Putting all of the above together yields a simple - but working - FTP server:

    #include  <stdio.h>			-- Standard I/O definitions.
    #include  "tcp_util.h"		-- TCP/IP networking utilities.
    #include  "nft_util.h"		-- FTP utilities.

    int  main (int argc, char *argv[])
    {
        char  *command ;
        TcpEndpoint  client, server ;
        NftSession  session ;
					- Listen at port 21.
        tcpListen ((argc > 1) ? argv[1] : "21", 99, &server) ;

        for ( ; ; ) {			-- Answer next client.
            tcpAnswer (server, -1.0, &client) ;
            nftCreate (client, NULL, NULL, NULL, NULL, &session) ;
            nftPutLine (session, "220 Service is ready.\n") ;
            for ( ; ; ) {		-- Service connected client.
                if (nftGetLine (session, &command))  break ;
                nftEvaluate (session, command) ;
                if ((nftInfo (session))->logout)  break ;
            }
            nftDestroy (session) ;	-- Lost client.
        }

    }

The server's name is specified as the first argument on the command line (i.e., argv[1]) and defaults to port 21. If a client connection is broken, the server loops back to wait for the next client.

In event-driven applications (e.g., those based on the X Toolkit or the IOX dispatcher), the socket connection underlying an FTP session, returned by nftFd(), can be monitored for input by your event dispatcher. Because input is buffered, the input callback must repeatedly call nftGetLine() or nftRead() while nftIsReadable() is true.

Extending an NFT Server

The NFT_UTIL package provides a means for modifying or extending the functionality of an NFT-based FTP server; it does this by maintaining a table that maps FTP command keywords to the C functions that process those commands. When called to evaluate a command string, nftEvaluate() parses the command line into an argc/argv array of arguments. It then looks up the command keyword (argv[0]) in the table and calls the command processing function bound to that keyword, passing the argc/argv array of arguments to the function.

nftCreate() initializes a session's keyword-function map with default entries for the commands called for in the RFCs. These default entries are defined in the defaultCommands[] and defaultCallbacks[] arrays below. The default command processing functions—except for those for PASV, PORT, and QUIT—are found in the companion package, nft_proc.c. An application can modify the processing of an existing command by registering a new command processing function for the command. The following example replaces the default nftRETR() retrieval function with myRetrieve():

    extern  errno_t  myRetrieve (NftSession session,
                                 int argc, const char *argv[],
                                 void *userData) ;
    ...
    nftRegister (session, "RETR", myRetrieve) ;

nftRegister() can also be used to add entirely new FTP commands to the keyword-function map. When replacing an existing command's callback, study the RFCs and the default implementation of the command so as to ensure that your implementation covers all the bases.

Command Processing Functions

Application-specific command processing functions registered with nftRegister() and invoked by nftEvaluate() should be declared as follows:

    errno_t  myFunction (NftSession  session,
                         int  argc,
                         const  char  *argv[],
                         void  *userData)
    {
        ... body of function ...
    }

argc is the number of arguments (plus the command keyword) in the command string being evaluated; argv[] is an array of character strings, each one being one of the arguments. For example, an FTP ALLO command,

    "ALLO 123 456"

would be parsed into an argc/argv array as shown here:

    argc = 3    argv[0] = "ALLO"
                argv[1] = "123"		-- File size
                argv[2] = "456"		-- Maximum record/page size

The command processing function is responsible for verifying the number and validity of a command's arguments. The userData argument is an arbitrary (void *) pointer specified when the session was created.

A number of NFT functions are available for use in a command processing function. A pointer to the public information in a session structure can be obtained with a call to nftInfo():

    NftSessionInfo  *info = nftInfo (session) ;

nftPutLine() should be used to format and send a reply message to the client over the session's control connection; the following example returns a response for the PWD command:

    nftPutLine (session, "257 \"%s\"\n", info->currentDirectory) ;

Commands such as RETR, STOR, and LIST must establish a separate network connection for transferring data back and forth. A basic implementation of the RETR command illustrates the use of the nftOpen(), nftWrite(), and nftClose() functions to transfer data:

    #include  <stdio.h>			-- Standard I/O definitions.
    #include  <string.h>		-- C Library string functions.
    #include  "nft_util.h"		-- FTP utilities.

    errno_t  myRetrieve (NftSession  session,
                         int  argc,
                         const  char  *argv[],
                         void  *userData)
    {
        char  buffer[1024], fileName[1024] ;
        FILE  *file ;
        size_t  length ;
        NftSessionInfo  *info = nftInfo (session) ;

		        -- Append the file name to the current working
			-- directory and open the file for reading.
        strcpy (fileName, info->currentDirectory) ;
        strcat (fileName, argv[1]) ;
        file = fopen (fileName, "rb") ;

			-- Establish a network connection with the client
			-- for the purpose of transferring data.
        nftOpen (session) ;
        nftPutLine (session, "150 Transferring: %s\n", fileName) ;

			-- Send the contents of the file to the client.
        while ((length = fread (buffer, 1, sizeof buffer, file)) > 0)
            nftWrite (session, length, buffer, NULL) ;

			-- End the data transfer.
        nftPutLine (session, "226 Transfer complete.\n") ;
        nftClose (session) ;
        fclose (file) ;

        return (0) ;

    }

The STOR command is implemented in a similar fashion, with the calls to fread(3) and nftWrite() replaced by calls to nftRead() and fwrite(3), respectively. The NFT_UTIL package handles the PORT and PASV commands that precede a data transfer command, so nftOpen() will establish the appropriate type of data connection with the application (or its programmer) being none the wiser.

The myRetrieve() function above does not strictly conform to the FTP standard in that it only supports binary transfers ("TYPE I") of arbitrary files. Text files should be transferred according to the Telnet protocol. In particular, each end-of-line marker (e.g., "\n") in a text file must be transmitted as a Telnet end-of-line sequence: a carriage return followed by a line feed ("\r\n"). The FTP client and the server are responsible for making the appropriate conversions to and from their hosts' end-of-line conventions when sending or receiving ASCII data.

nftRead() and nftWrite() do not perform these conversions for you. However, the CR/LF utilities (see crlf_util.c) simplify the handling of ASCII text. Using these conversion utilities, the myRetrieve() function needs to be modified in only two places in order to support ASCII text transfers:

    #include  "crlf_util.h"			-- CR/LF utilities.

    errno_t  myRetrieve (...)
    {
        ...
			-- Open the file for reading.
        if (info->representation[0] == 'A')	-- ASCII transfer?
            file = fopen (fileName, "r") ;	-- Open ASCII file.
        else
            file = fopen (fileName, "rb") ;	-- Open binary file.
        ...
			-- Send the contents of the file to the client.
        while ((length = fread (buffer, 1, (sizeof buffer)/2, file)) > 0) {
            if (info->representation[0] == 'A') {
                nl2crlf (buffer, length, sizeof buffer) ;
                length = strlen (buffer) ;
            }
            if (nftWrite (session, length, buffer, NULL))  break ;
        }

        ...

    }

Within the send loop, nl2crlf() is called to convert newline characters in the file to the carriage return/line feed sequence. Note that only half of the buffer is used for reading, thus allowing for the worst case scenario of a string consisting entirely of newlines being expanded to twice its size.


Public Procedures

nftClose() - closes a session's data connection.
nftCreate() - creates an FTP session.
nftDestroy() - deletes an FTP session.
nftEvaluate() - evaluates an FTP command.
nftFd() - returns a session's control or data socket number.
nftGetLine() - reads the next line from a session's control connection.
nftIgnoreCmd() - ignores an FTP command.
nftInfo() - returns a pointer to the session's public information.
nftIsReadable() - checks if input is waiting to be read from a session's control or data connection.
nftIsUp() - checks if a session's control or data connection is up.
nftIsWriteable() - checks if data can be written to a session's control or data connection.
nftLookup() - looks up the function to process an FTP command.
nftName() - returns the name of a session's control or data connection.
nftNextCommand() - reads and evaluates the next FTP command.
nftOpen() - opens a session's data connection.
nftPASV() - processes the FTP PASV command.
nftPeer() - returns the name of the session's peer.
nftPORT() - processes the FTP PORT command.
nftPutLine() - writes a line of output to a session's control connection.
nftQUIT() - processes the FTP QUIT command.
nftRead() - reads input from a session's data connection.
nftRegister() - maps an FTP command to a user-supplied function.
nftSyntax() - returns the syntax of an FTP command.
nftWrite() - writes output to a session's data connection.

Source Files

nft_util.c
nft_util.h

Command Processing Functions

nftAccessCmds() - processes FTP access control commands.
nftCWD() - processes the FTP CWD command.
nftFileCmds() - processes the FTP file management commands.
nftHELP() - processes the FTP HELP command.
nftListCmds() - processes the FTP LIST and NLST commands.
nftMODE() - processes the FTP MODE command.
nftPASS() - processes the FTP PASS command.
nftRETR() - processes the FTP RETR command.
nftServiceCmds() - processes FTP service commands.
nftSTAT() - processes the FTP STAT command.
nftStoreCmds() - processes the FTP APPE and STOR commands.
nftSTRU() - processes the FTP STRU command.
nftTYPE() - processes the FTP TYPE command.
nftUSER() - processes the FTP USER command.

Source Files

nft_proc.c
nft_proc.h

Alex Measday  /  E-mail