Multithread Lab

1. Introduction

The aim of this week's work is to add a bit parallelism in the client-server file exchange application developed in Labs 2 and 3.

Up to now, the server responds serially to all the possible clients that might contact it. This is not much of an issue in our current application since server replies are very quick; but it won't be a good behavior in general. The main problem with the current server design is that we open only one single socket for the communication and that this socket is blocking: only one single communication can occur at a time.

Let's first change a bit our application code to see the problem and then parallelize the server's code using threads. More precisely, you'll have to:

change the protocol to see the problem;
prepare for multithreading: modularize the task to be mutilthreaded;
multithread it;
prevent concurrent access to files: add lock with mutex.

2. Change the protocol a bit

To see the problem, let's add an extra acknowledgment step in the protocol, and most importantly, arbitrarily slow down the server.

2.1 Add an extra acknowledgment

To see the impact of the server on the client side, let's first add an acknowledgment from the server to the client once the file is written: add an extra message, typically send the size written to disk, from the server to the client.

On the client side, before ending, wait for the server acknowledgment and check its value is equal to the actual file size sent.

Test your newly extended protocol.

2.2 Slow the server down

Then, add an artificial slow down of the server: between the closing of the written file and the message printed on the screen by the server that the file is indeed written, add a sleep() command:

fclose(file);
sleep(5); // artificial delay;
printf("%zu bytes wrote to \"%s\".\n", total_written_size, filename);

(and this is, of course, before sending the acknowledgment (size) to the client!)

Choose a delay value (in seconds) that fits your testing needs.

Then test with multiples tabs/terminals: one server (pay attention where to launch it since the server overwrites files!) and N clients (in N+1 tabs/terminals).
Observe that the clients have to wait for the server to be done with the others before being considered.

2.3 Other optional improvement: select port

Since you changed your basic code, it may be worth also adding this extra feature: choose the port from the command line.

When doing several test-and-try in a raw, you might face the issue that the port is still in use by a recently closed former server process:

Error binding to 127.0.0.1:1234: Address already in use
Could not start server: Address already in use

If you'd like to, you can extend your client and server code with providing an optional port number on the command line:

./get_file 3241
./send_file 3241

If you want to do so, it may be worth looking at the optional complementary video about argc & argv and its follow-up example.

You could also benefit from the tool atouint16() function provided in util.c.

3. Modularize the client handling part

3.1 Big picture

The most advanced way to solve single blocking socket problem is to use polling non blocking connections (using poll() or even epoll() for larger servers). But as a first start in this Lab, we choose to implement a simplest way, also illustrating the lectures you soon had: multithreaded blocking connections. Each reply to a client will be opened in a new thread, thus allowing several parallel communications to the server.

Then, all access to the common resources, here the file to write, shall be locked: any interaction with it must be locked for the other threads and unlocked as soon as the interaction is over. We will address this issue at the end of this Lab.

3.2 Modularizing the reply to clients

It's the reply to the client which will be threaded, i.e. the part of your code that handles a client's request and replies to it. So first, if you haven't already, you have to make this part a function.

Put all the code from just after the tcp_accept() (excluded) up to the closing of the active socket (included) inside a handle_connection() function. So the server main loop should now look like:

while (1) {
    puts("Waiting for a filename...");

    struct sockaddr_in cli_addr = {0};
    const int active_socket = tcp_accept(passive_socket, &cli_addr);
    if (active_socket < 0) die("tcp_accept()", passive_socket);

    handle_connection(active_socket, &cli_addr, passive_socket);
}

(this code should, of course, be adapted to your own version.)

We recommend you proceed incrementally and check your updated version at each step.

3.3 Arguments passing

Regarding the passing of the arguments, there are two things to be changed:

allocating (on the heap) an new instance of them, rather sharing the same one on the stack, to avoid race conditions between threads;
passing them in a generic way since threaded code requires generic arguments (void*).

Let's address these points in reverse order.

Since the generic prototype of a threaded function is (man pthread_create):

void* to_do(void* arguments);

we now have to create a data structure (e.g. thread_args_t) to store all the required data for that thread's arguments. This of course needs to be adapted to your specific case, but we typically foresee:

the (active) socket used (type int);
the client address (type struct sockaddr_in).

Regarding the passive socket, it may be easier in our case to make it a global variable. Otherwise, add it to thread_args_t, but pay attention when calling die() from threads, you shall release arguments first, thus first copy the passive socket as a local variable to then pass it to die() after the free() of arguments....

Now, as stated above, this thread_args_t to be passed to handle_connection() cannot be on the stack, but have to be allocated on the heap.
Change your code to do so. Don't forget to release them, in handle_connection(), whenever needed (before each ending).

Also, in handle_connection() you of course have to cast the received void* arguments to a thread_args_t* to properly get all the required information from it.

And, as usual, before making that function a new thread, give it a first try to check that you didn't break anything so far.
Once this checked, you can proceed to making it multithreaded.

4. Go for multiple threads

4.1. Thread creation

So you now have to create a thread in the the infinite loop in your server's main() (where you currently have a single call to handle_connection()):

(should already be done) have the content of your thread_args_t ready and stored on the heap;
create and initialize to PTHREAD_CREATE_DETACHED some pthread attributes; see pthread_attr_init() and pthread_attr_setdetachstate() man-pages; notice that "detached" threads automatically release their resources on exit (but then there is no way to get their return value; we'll ignore them; you can thus simply return NULL);
remove the call to handle_connection() and replace it by the creation of a thread (see pthread_create()) that will run handle_connection() with the proper thread_args_t as parameter;
don't forget to properly handle errors from the calls of these functions and to release the pthread_attr_t with pthread_attr_destroy() once done.

Note: this is a practice exercise for programming threads in C. There is thus a part of understanding on your side here: reviewing your lecture slides, reading man-pages(, asking questions).

We also recommend, for the sake of tracing/debugging to have some appropriate messages at the beginning and a the end of handle_connection(), typically indicating the thread id with pthread_self().

4.2 Signal handling

Now that handle_connection() is multi-threaded, we don't want the SIGTERM and SIGINT signals to be intercepted by it (but leave them to the main thread). For this, simply add this code at the beginning of handle_connection():

    sigset_t mask;
    sigemptyset(&mask);
    sigaddset(&mask, SIGINT );
    sigaddset(&mask, SIGTERM);
    pthread_sigmask(SIG_BLOCK, &mask, NULL);

4.3 Testing with many client requests at the same time

Test the multithreaded approach by launching several client requests at the same time (from different tabs/terminals). You can do this for instance by:

Launching the server, still with some artificial reply delay as mentioned above;
pay attention where you launch it since the server overwrites files!
then have a few (3-5) terminals ready; in N-1 tabs/terminals, launch a bunch of send file:
```
 for i in $(seq 10); do echo "SOMEFILENAME" | ./send_file; done
```
where "SOMEFILENAME" is a valid file of your own, different in each tab/terminal;
in the last terminal, manually launch send file when you want and as many time as you want (make use of the shell history: up-arrow to go back to previously launched command):
```
 echo "SOMEFILENAME" | ./send_file
```
where "SOMEFILENAME" is yet another valid file of your own.

See how this behaves differently with or without multithreading (without multithreading some of these requests are slowed by the others; with multithreading, you should see that the server can handle many files at the same time; the more tabs/terminals in parallel the better you'll see that).

5. Lock file access

In the above testing procedure, we emphasized to use different filenames. What if two clients sent the same file(name)? Well, actually the sent+write is so quick that almost surely nothing harmful will happen. But what if two clients sent the same big filename, which take a while to write?
There might be concurrent write access to the file, with risk of data corruption (rather than having the proper later file).

To avoid this, we must lock the access to a file. Of course, we could benefit from OS file locking facilities like e.g. flock() to have the OS do this for us. But the purpose of this Lab is to train you on threads and mutex, so let's do it ourselves in our own code.

For this, we want to add a lock (a "mutex") to each filename in use. The proper way to do that would be to have a hash-table mapping filenames to mutex. Let's do a simplified version of this here and have a simple static array of structs having a filename and a mutex:

#define NB_FILES_MAX 128

typedef struct {
    char filename[NAME_MAX+1];
    pthread_mutex_t mutex;
} filename_lock_t;

filename_lock_t mutexes[NB_FILES_MAX];

Yes, this is ugly indeed, but enough for this Lab. Actually, to improve a bit, we will rotatively use the NB_FILES_MAX slots to store filenames: once full we go back in 0 (explained below).

Now, since all threads have access to this "hashtable" mutexes, we must also lock its access: we thus have to lock each individual filename, as well as the global data structure which is used to keep track of filenames (namely, the "hashtable"). For the latest (i.e., to lock this "hashtable" mutexes), declare a single global variable of type pthread_mutex_t and initialize that mutex to PTHREAD_MUTEX_INITIALIZER.

Then, write a function get_filename_mutex() which takes a filename and returns a pointer to a pthread_mutex_t. This function proceed as follows:

lock the "hashtable" with pthread_mutex_lock();
do a linear search of the filename in the "hashtable": stop searching if either:
- i is too big;
- filename at i is empty;
- filename at i matches argument filename.
in the first case, set i to 0;
if not in the third case:
- copy the argument filename to position i in the "hashtable";
- initialize its mutex with pthread_mutex_init();
- prepare room for the next filename: set to empty string the next entry in the "hashtable"; if the current position in the "hashtable" is the last one, the next entry is the first one (0).
once all this is done:
- unlock the "hashtable" mutex with pthread_mutex_unlock();
- return the filename mutex (&mutexes[i].mutex).

Then simply lock the filename in handle_connection():

call get_filename_mutex(filename) and lock the corresponding mutex just before opening the file;
unlock the mutex just after closing the file (don't forget to do it for each close()).

And that's it!

Test and debug your code with multiple client (and maybe a small NB_FILES_MAX).