CS-202

ImgFS: Image-oriented File System --- ImgFS format

The aim of this week is to:

  1. become fully acquainted with the framework and concepts of the project;
  2. implement the opening and closing of ImgFS files;
  3. then implement the simplest functionality, which lists the contents of the metadata.

So start by reading the main project description file to understand the general framework of the project. Once you've done that, you can continue below.

Provided material

In your group's GitHub repository, you will find the following files in provided/src/:

  • imgfs.h: function prototypes for the operations described here;
  • imgfscmd.c: the core of your "Filesystem Manager", the command line interface (CLI) to handle imgFS; it reads a command and calls the corresponding functions to manipulate the database;
  • imgfs_tools.c: the tool functions for imgFS; for example to display the data structure;
  • imgfscmd_functions.h and imgfs_cmd.c: prototypes and definitions of the functions used by the CLI;
  • util.h and util.c: macros and miscellaneous functions; you do not need to use them (have a look to see if some may be useful);
  • error.h and error.c: error code and messages;
  • a Makefile containing useful rules and targets;
  • some unit and end-to-end tests in provided/tests/{unit,end-to-end}/;
  • data for your tests in provided/tests/data.

To avoid any trouble, the contents of the provided/ directory must never be modified!

Start by copying the files you need from provided/src/ into the done/ directory at the root of the project and registering it in git (git add); for instance:

cp provided/src/*.h provided/src/Makefile provided/src/imgfs*.c provided/src/util.c provided/src/error.c done
git add done

You'll proceed similarly in the next weeks, whenever you'll need new files from provided/src.

Tasks

The provided code does not compile; some work is still required, in the following steps (which are further detailed below):

  1. define the data structures required for imgFS;
  2. parsing the command line arguments;
  3. define the functions do_open() and do_close();
  4. define the function do_list();
  5. define the function do_list_cmd().

After reaching that point, the code should compile without errors. You will then have to test it.

An example usage of the CLI (the name of which is imgfscmd) is:

./imgfscmd list empty.imgfs

where list is a command provided to the CLI and empty.imgfs is an argument for that command, here simply an ImgFS file (thus a file containing a whole filesystem).

0. STYLE!

Important Note: writing clean code, readable by everyone is very important. From experience, it seems that not everyone does this spontaneously at first ;-). There are tools that can help. For example, astyle is a program designed to reformat source code to follow a standard (man astyle for more details).

We provide you with a shortcuts (which uses astyle): see the target style in the provided Makefile (make style to use it). We recommend you do a make style before any (Git) commit.

1. Define data structures

The exact format of the header and metadata is given in the global project description. The types

  • struct imgfs_header;
  • struct img_metadata;
  • and struct imgfs_file;

are to be defined in replacement of the "TODO WEEK 07: DEFINE YOUR STRUCTS HERE."" in imgfs.h.

2. Parsing of the command line arguments

The second objective of this week is to process the arguments received from the command line. For modularization purposes, we will use function pointers.

To achieve this, the signatures of the functions do_COMMAND_cmd() (and help()) are uniform:

int do_COMMAND_cmd(int argc, char* argv[])

Those functions will handle the parsing of their respective additional arguments, while the main() dispatches through them using the first CLI argument.

Using arrays for simplicity

To process all the different commands, we would like to avoid an "if-then-else" approach. Indeed, this would make adding new commands (which will arrive in the following weeks) more difficult, since it would require to add new cases for each of them. It would also make the code much less readable.

To avoid that, we put the various do_COMMAND_cmd() (and help()) functions in an array. We will take advantage of this to associate the names of the commands with their respective functions (e.g. the string "list" with the do_list_cmd() function), and then simply add a loop to the main() function, to search for the received command among the list of possible commands -- for the moment, "list", "create", "help" and "delete" -- and call the corresponding function.

In imgfscmd.c:

  1. define a command type, a pointer to functions such as those unified above;
  2. define a struct command_mapping type containing a string (constant) and a command.

Then use these definitions to create an array named commands associating the commands "list", "create", "help", and "delete" to the corresponding functions.
Note: The "create", "help" and "delete" commands are not yet implemented, but you can already add them to the array.

Finally, complete the main() using this array inside a loop. When the right command is found, simply call the function pointed to in the corresponding array entry, passing all the command line arguments.

For example, if you call the program

./imgfscmd list imgfs_file

then your code must call do_list_cmd() with the following parameters: argc = 1 and argv = { "imgfs_file", NULL }.

Your code must correctly handle the case where the command is not defined: in this case, simply call help() and return ERR_INVALID_COMMAND.

Your code can perfectly well assume that all commands in the commands array are distinct.

3. do_open() and do_close()

Now, we will implement the functions to open and close existing imgfs files.

You need to write the definitions of do_open() and do_close() in the file imgfs_tools.c.

The do_open() function takes as arguments:

  • the image base file name (const char *);
  • the file opening mode (const char *, e.g. "rb", "rb+");
  • the imgfs_file structure in which to store read data.

The function must

  1. open the file;
  2. read the contents of the header;
  3. allocate the metadata array;
  4. read the contents of the metadata.

The function should return the value ERR_NONE if all went well, and otherwise an appropriate error code in case of problems. You need to handle all possible error cases in this function, using the definitions in error.h (see unit tests below).
Note: to check the validity of a pointer given as parameter, you can use the macro M_REQUIRE_NON_NULL(ptr), which will make the function return ERR_INVALID_ARGUMENT if ptr == NULL (see util.h).

The do_close() function takes a single argument of structure type imgfs_file and must close the file and free the metadata array. It returns no value. Here too, remember to handle the possible error case: if the file (FILE*) is NULL. This should be a reflex when you're writing code, especially when you're using a pointer. We won't mention it again.

4. Define do_list()

Then create a new file imgfs_list.c to implement the do_list() function. If output_mode is STDOUT, the purpose of do_list() is first to print the contents of the "header" using the supplied print_header() tool function, and then to print (examples below)

  • either

    << empty imgFS >>

    if the database does not contain any images;

  • or the metadata of all valid images (see print_metadata(), provided in imgfs.h).

The case output_mode == JSON will be implemented later in the project; you may just call TO_BE_IMPLEMENTED() in this case (see util.h).

Warning: there may be "holes" in the metadata array: one or more invalid images may exists between two valid ones.

5. Complete do_list_cmd()

In order to be able to use the do_list() function from the command line, implement the do_list_cmd() function in imgfscmd_functions.c, which receives the command line arguments as parameters (as explained before).

The first element of the array is the name of the file containing the database. After checking that the parameters are correct, open the database and display its contents, using the above functions.

Examples and tests

To make it easier to understand the various functions described above, a few examples are given here. These examples are in the provided tests (see below).

Black-box (end to end) testing

It's best to start testing your code on simple cases that you're familiar with.

You can test your code with the supplied .imgfs files: the command

./imgfscmd list ../provided/tests/data/empty.imgfs

should display (exact file here):

*****************************************
********** IMGFS HEADER START ***********
TYPE: EPFL ImgFS 2024
VERSION: 0
IMAGE COUNT: 0          MAX IMAGES: 10
THUMBNAIL: 64 x 64      SMALL: 256 x 256
*********** IMGFS HEADER END ************
*****************************************
<< empty imgFS >>

while

./imgfscmd list ../provided/tests/data/test02.imgfs

should display (exact file here) :

*****************************************
********** IMGFS HEADER START ***********
TYPE: EPFL ImgFS 2024
VERSION: 2
IMAGE COUNT: 2          MAX IMAGES: 100
THUMBNAIL: 64 x 64      SMALL: 256 x 256
*********** IMGFS HEADER END ************
*****************************************
IMAGE ID: pic1
SHA: 66ac648b32a8268ed0b350b184cfa04c00c6236af3a2aa4411c01518f6061af8
VALID: 1
UNUSED: 0
OFFSET ORIG.: 21664            SIZE ORIG.: 72876
OFFSET THUMB.: 0                SIZE THUMB.: 0
OFFSET SMALL: 0                SIZE SMALL: 0
ORIGINAL: 1200 x 800
*****************************************
IMAGE ID: pic2
SHA: 95962b09e0fc9716ee4c2a1cf173f9147758235360d7ac0a73dfa378858b8a10
VALID: 1
UNUSED: 0
OFFSET ORIG.: 94540            SIZE ORIG.: 98119
OFFSET THUMB.: 0                SIZE THUMB.: 0
OFFSET SMALL: 0                SIZE SMALL: 0
ORIGINAL: 1200 x 800
*****************************************

Note: you may compare your results by using:

./imgfscmd list ../provided/tests/data/test02.imgfs > mon_res_02.txt
diff -w ../provided/tests/data/list_out/test02.txt mon_res_02.txt

More details: man diff.

Provided tests

setup

The provided test suites require several dependencies: Check and Robot Framework (and its own dependency, parse). On (your own) Ubuntu, you can install them with:

sudo apt install check pip pkg-config

then, depending on how you're used to work in Python, either as root or in your Python virtual environment (maybe to be created):

pip install parse robotframework

(Of course you'll have to run the tests in that Python venv, if that's your usual way to work with Python.)

ON EPFL VMs, you have to setup a personnal Python virtual environment.

If you already have one, activate it and install the two above mentioned packages (parse and robotframework).

It you don't, we recommand you create your personnal Python virtual environment in myfiles:

cd ~/Desktop/myfile
python -m venv mypyvenv
cd mypyvenv
cp -r lib lib64 ## this fixes the first warning
cd ..
python -m venv mypyvenv

Ignore the (second) warnings.

Then activate it:

source mypyvenv/bin/activate

and then install the required packages:

pip install parse robotframework

And you're done.

The only thing you'll have to do next time you login and you want to run the "end to end" tests, is to activate your Python virtual environment:

source ~/Desktop/myfiles/mypyvenv/bin/activate

Of course, you can also add that to your ~/.bashrc!

provided tests

We provide you with a few tests to run against your code by using make check, both unit tests (testing functions one by one) and end-to-end tests (testing the whole executable at once).

We strongly advise you to complete them by adding you own tests for edge cases; the imgFS files are in provided/test/data. You can check the unit tests in provided/test/unit and the end-to-end ones in provided/test/end-to-end to understand how to write your own.
Note: Don't forget to never push modifications in the provided/ directory; instead move the test/ directory to done/ and update the TEST_DIR variable in the Makefile accordingly.

We also provide a make feedback (make feedback-VM-CO if you're working on EPFL VMs) which gives partial feedback on your work. This is normally used for a minimal final check of your work, before handing it in. It's better to run local tests directly on your machine beforehand (including more tests you've added yourself, if necessary).

The Docker image used by make feedback will be tagged latest every week, but if you want to run feedback for a specific week, change (in the Makefile at the line that defines IMAGE) this latest tag to weekNN where NN is the desired week number, e.g.:

IMAGE=chappeli/cs202-feedback:week07

Work organization

It's up to you to organize the group work as best you can, according to your objectives and constraints; but remember to divide the task properly between the two members of the group. If you haven't already read it in full, we recommend you read the end of the foreword page.

Submission

You don't have to formally deliver your work for this first week of the project, as the first deliverable will only be due at the end of the week 10 (deadline: Sunday May 5th, 23:59), together with weeks 8 and 9 work.
Having said that, we strongly advise you to mark with a commit when you think you've completed some part of the work and especially once you reached the end of this week (you can do other commits beforehand, of course!):

  1. add the new imgfs_list.c file to the done/ directory (of your group GitHub repository; i.e. corresponding to the project), along with your own tests if required:

    git add imgfs_list.c

  2. also add the modified files (but NOT the .o, nor the executables!): imgfs_tools.c, imgfs.h and maybe Makefile:

    git add -u

  3. check that everything is ok:

    git status

    or

    git status -uno

    to hide unwanted files, but be careful to not hide any required file!

  4. create the commit:

    git commit -m "final version week07"

In fact, we strongly advise you to systematically make these regular commits, at least weekly, when your work is up and running. This will help you save your work and measure your progress.