Eurecom-Drive: a file repository

General overview

The objective of this project is to design a system to remotely store files. This system is built upon two components:

A server, handling file requests sent by clients. This servers stores files in a configurable directory (given as argument) of the disk.

a client, reading from a file, or from the keyboard, orders from users of the systems, and accordingly sending requests to the server

Typical requests from a client to the server are: adding a file to the remote repository, removing a remote file, getting a remote file, listing available remote files, printing the first n lines of a remote file, and duplicating a remote file. The network connection between the server and the client is provided. We expect you to program different commands to implement the above mentioned functionalities. The specification of each command is given hereafter.

System specification

Packet specification

Your objective is to implement the following commands sent by a client to a server. All commands are to be sent by a client respecting a packet format, defined as follows. A packet (header + data) cannot be longer than 4096 bytes, so several packets have to be used to send a file longer than 4096 minus the header size. The full header is always sent, and the data part of a packet is sent only if necessary.

E [1 byte]

D [1 byte]

command or error [1 byte]

data size [2 bytes]

option 1 [32 bytes]

option 2 [32 bytes]

data...[variable size maximum:4027 bytes]

The first two bytes are fixed: they contain the 'E' and 'D' characters (for Eurecom Drive). Right after, a command or error code is stored in one byte. The size of data follows, stored in two bytes as an unsigned int 16. Then, the next 64 bytes represent 2 options: option 1 then option 2, each stored in 32 bytes. They contain string, unless when specified below with [int]. Depending on the command, the data field may also be used. It contains at most 4027 bytes that represent a string.

The server answers to client commands using the same packet format.

A few more information:

When a string is sent in an option field of the header, it must be terminated by '\0'. The whole string, including its final character, must fit in the allocated part of the header.

An "empty" field, as used several times below, means that the sender has put undetermined characters in that field, meaning that the receiver shall not analyze these characters after they have been read from te network stream.

When an int must be sent as a String in the data field, the int must not start with any "0". For instance, value 23 must be sent as "23" and not "023" nor "0023" etc.

A file name must start with a character between "a" / "z" or "A" and Z". Then, it can contain only alphanumerical characters, and "_" and ".". Files not respecting this specification must be ignored by both client and server, including in the "ls" command (see below).

When the server returns a string in the data field, it is not terminated by "\0".

If a file data transfer is interrupted (connection interrupted, etc.), then the file must not be created by the destination of the data transfer.

Overwriting a file is not allowed at both sides. Therefore, any command that would lead to overwrite a file must fail.

Commands

Adding a remote file

Client request

This command is expected to copy a local file to the Eurecom Drive. The packet format is as follows:

size of data in this packet

Local file name

Size of file [unsigned int]

data...

Server answer

After this packet has been received, the server informs the client with the following packet whether it accepts this new file (0 code) or whether it refuses it (error code).

0 or error code

empty

Client request: cont. (optional)

If the client couldn't fit all data of the local file in the first packet, then the client must continue to send this data to the server. Yet, the client continues to send these data only if it has received a "0" code from the server. The client uses the following packet to continue sending data. the client can use as many packets as necessary to send all file data. There, the client goes until the end, without waiting for a server validation.

size of data in this packet

Local file name

Size of file [unsigned int]

next file data...

Whenever the client fails to complete the file transfer (for instance if a read file error ocurred), the client immediately issues an ABORT error packet (see error codes below) and stops the transfer.

ABORT code

empty

Server answer

After it has received the last file data packet from the client, the server finally answers with a 0-code or with an error code.

0 or error code

empty

This last answer is only sent if there are additional packets and no ABORT has been received. In other words, if the total number of packets is 1 or client does not complete file transfer, only one answer is sent.

Removing a remote file

This command shall remove a file from the Eurecom Drive.

Client request

The packet format is as follows:

Remote file name

Empty

Server answer

The server answers either with a 0 code (success) or with an error code.

0 or error code

empty

Getting a remote file

This command is used to copy a remote file to the local file system.

Client request

The packet format is as follows:

Remote file name

Empty

Server answer

The server shall answer with the following packet:

size of data in this packet

Remote file name

Size of file [unsigned int]

file data...

If the file data cannot fit in one packet, extra packets are sent by the server. Packers are formatted similarly:

size of data in this packet

Remote file name

Size of file [unsigned int]

next file data...

In case of error at server's side, the server issues an error information and the exchange with the client for the on-going request ends immediately.

error code

empty

Listing remote files

This command lists all the files remotely stored in the Eurecom Drive.

Client request

The packet format is as follows:

Remote file name

Empty

Server answer

The server shall answer with the following packet:

size of data in this packet

Total number of packets [unsigned int]

Empty

file1 name,size1,file2 name, size2, ...

File names and sizes are separated with a comma, without any space between. The list must be sorted by file names: we provide the code for this.

If the list of files fits in 1 packet, the "option 1" packet field is set to 1. Otherwise, it is set to the necessary number of packets.

size of data in this packet

Total number of packets [unsigned int]

Empty

filen name,sizen,filen+1 name, sizen+1, ...

Also, if the data field is empty, is means that there are no files in the remote drive.

Finally, as for previous requests, in case of error, the server only sends an error code as follows:

error code

empty

Printing n lines of a file

This command returns the n first lines of a file.

Client request

The packet format is as follows:

Remote file name

Number of lines [unsigned int]

Server answer

The server shall answer with the following packet:

Data size of this packet [unsigned int]

Total number of packets [unsigned int]

Empty

Content of lines ...

More than one packet could be necessary to send the n lines. If the file has less than n lines, then all the content of the file is sent.

Data size of this packet [unsigned int]

Total number of packets [unsigned int]

Empty

remaining lines ...

As for previous requests, in case of error, the server only sends an error code as follows:

error code

empty

Duplicating a remote file

This command duplicates a remote file, that is, the remove drive contains a new file with the exact content of the original file.

Client request

The packet format is as follows:

Remote file name

Name of the duplicated remote file

Server answer

The server shall answer either with a success, or with an error:

0 or error code

empty

Error handling

Server error responses may contain the codes associated to the following errors:

Shortcut	Code	Is critical?	Short explanation
FORMAT	-1	yes	bad packet format
MISS	-2	no	file not found
EXISTS	-3	no	file already exists
ABORT	-4	no	command fails (for other server-side failures)
QUOTA	-5	no	quota exceeded

Some of these error can also happen at client side, also leading to the error message "KO x".

FORMAT is returned when a packet with correct 'ED' header and size of data has been received, but either the command code is not the expected one or the other parameters are not consistant with the command code. After this error, the server closes connection with client.

MISS is returned when a required file has not been found at server side. This error can also happen at client side. After returning this error, client and server wait for the next command.

EXISTS is returned when a file to be written already exists (overwriting file is forbidden). It can also happen at client side (e.g., with the get command). After this error, client and server wait for the next command.

ABORT is returned for other problems at server side. It always means that the command could not be executed by the server.

QUANTUM: use only if bonus question has been implemented. After this error, client and server wait for the next command.

The following errors also lead to "KO x" error messages. They are all detected at client side and do not correspond to server response error codes.

Shortcut	Code	Is critical?	Short explanation
SYNTAX	-6	no	syntax error in command line
RESP	-7	yes	bad response form server
CLOSED	-8	yes	connection closed

SYNTAX denotes an error linked to command line parsing.
RESP occurs when a packet with correct 'ED' header and data size of data has been received, but either its response code is not the expected one or the other parameters are not consistant with the response code.
CLOSED means that the connection has been closed.

For all the errors above, you may provide more information in stderr about the precise cause of the error. Finally, for any other errors, an explanation message must be send to stderr and the client must exit immediately. These other errors may be, for example, related to a failure in socket communication, or to a bad access to client-side files, or to the receiving of packets without the correct header ('ED' and a correct data size).

Implementation

Your implementation must be compilable and runnable on the Linux PCs in the laboratory rooms. We will evaluate the compilation of your code, and then the execution using the provided tests.

We have prepared a C environment (Makefile, libraries, include) for the project in the following git: https://gitlab.eurecom.fr/ludovic.apvrille/basicos_project_fall2023_forstudents. Clone it as follows:

$ git clone git@gitlab.eurecom.fr:ludovic.apvrille/basicos_project_fall2023_forstudents.git

Once cloned, have a look at the .md file at the root of this git repository.

Server

The server must accept the following arguments:

-quotasize [int]: specifies the maximum number of bytes that the remote directory can store.

-quotanumber [int]: specifies the maximum number of files that the remove directory can store.

-directory [string]: specifies the directory to be used to store files. If this directory is non empty when the server starts, then existing files are assumed to be part of the remote drive.

Client

The client must accept the following options. Each option can be provided only once on the command line, otherwise the client must exit immediately.

-analyze [string]: the client executes the command of the file given as input.

-interactive: the client waits for commands on the keyboard. If the "-analyze" option is also provided, then the client first executes of the commands of the referenced files before waiting for keyboard-entered commands.

-directory [string]: specifies the directory to be used for client side files. If this directory is non empty when the client starts, then existing files are assumed to be part of the handled files, as long as they respect the requirements on file names.

Mandatory: All these options must be recognized in the command line even if they are not yet fully implemented. Options "-analyze" and "-directory" must be implemented.

The following commands are related to client requests. The following table lists these commands.

Command	Comment
put filename	adds the referenced local file to the remote drive
rm filename	removes the referenced file from the remote drive
get filename localfilename	locally stores a file available remotely
ls	list remote files
cat filename n	prints the first n lines of a remote file
cp originfilename destinationfilename	duplicates a remote file
quit or exit	exits the client
restart	client reinitialization (reset connection)
help	describes the available commands

When the client operates in "analyze mode", it displays the currently executing command (as specified in the input file) on one line, followed by the result on the subsequent line. This result will be either "OK" or "KO x", where 'x' represents the error code. If the command executes successfully or the error isn't critical, the client proceeds to the next command. However, encountering a critical "KO" result prompts an immediate termination or return of the client. All standard outputs should strictly be fed into stdout. For any additional debug or error messages, use stderr."

There is no defined interaction format in the interactive mode: it is up to you to decide which textual interface you want to offer.

Regarding the "ls" command, the client should display its output using the following format: each filename on a separate line, followed by two tab characters ("\t" represents a tab in C), and then the file size in bytes. For example:

file1       size1
file2       size2
...

Bonus

Handling "?" and "*" in filenames

Storing remote files in a ciphered way

Handling login/password for users

Deadlines, deliverables and organization

Program the specified system, and provide the sources files in C, Makefile and one report by 2023, Dec. 4th, midnight CET. Advices on how to write a report are given here. Everything should be available in the git of your project. No commit after this date will be considered.

All members of the group must work on the general definition of the remote file repository, and the implementation of the core functions. Then, the implementation of commands must be split between the members of the group, as follows:

Members #1 and #2: work on client commands

Members #3 and #4: work on server commands

Grading

10 points are given to the whole group, 10 points are given to each student.

5 points for the report. The report covers the work of the whole group, but must emphasize on the work done by each group member.

5 points for the global approach: global algorithm, common code, regular progression on the project.

10 individual points for personal implementation. This grade is individual. If your implementation in C is not ready by the deadline, provide the corresponding algorithm in pseudo code: you can get up five points with a pseudo-code.

How to proceed?

Form 16 groups of 4 students. As soon as your group is setup, send me an email with the name of the group leader, as well as the other names, and the email of each group member. I will validate the group by answering to your email and giving you a group number. Please organize yourselves into 16 groups, each consisting of 4 students. Once your group has been established, I would like the designated group leader to send me an email providing the full names and email addresses of all the group members. In the subject line of your email, please include the phrase "[BasicOS] New group". Upon receiving your email, I will confirm your group's formation, assign you a group number.

Setup a gitlab projet (use gitlab.eurecom.fr for this) called basicos2023-teamXX (With XX your group number) and send me the link to your gitlab: I will follow your progression that way. Do allow me to clone your gitlab project.

Clearly define the general structure of your project. Two documents we have prepared should help you for this: the first one on how to structure a C program and the second one on complementary information on the project specification

Program together the main framework: handling of arguments, handling of commands, handling of errors.

Individually program the commands you are meant to implement, at client or server side, and provide unit tests for each command.

Integrate together all commands, and check that all the tests you provide run: a part of your grade is based on this.

Provide at least 5 more tests with more that 10 commands each.

Your git must be ready (=DEADLINE) the 7th of December, 18h00, Paris time.

And do not forget to regularly commit&push on the git. Working in a regular way is part of the final grade.