Using the Grid/Grid storage

From BiGGrid Wiki
Jump to: navigation, search


Grid Storage Illustration

Illustration of grid storage


Each cluster on the Grid is equipped with a Storage Element or SE where data is stored. On which SE's you can store data depends on your VO, and can be inspected with the following command:

$> lcg-infosites --vo lsgrid se
Avail Space(Kb) Used Space(Kb)  Type    SEs
981270000       118902          n.a
8250147054295   282442433       n.a
1320000000      4429468         n.a
376360000       44957579        n.a
536752010       118902          n.a
1190000000      18978219        n.a
1290000000      19570444        n.a
1320000000      4372571         n.a
375067564       44362835        n.a

Sometimes it is useful to store your data on more than one SE. This is called replication. This allows jobs to run close to where the data is located, reducing long range network traffic.

SE's identify files by Storage URL or SURL, but these SURL's are not very descriptive; e.g. a SURL might look like this:


The logical file catalog (LFC) allows you to give more descriptive names to your files, and to order your files in a directory structure. Bear in mind that the LFC is not itself a storage system. It's just a database that keeps track of your files.

About the command line tools

You can manipulate your data with a set of command line tools. Some of these commands start with lfc-..., while others start with lcg-..., which can be confusing at first.

All these commands operate on the data itself. Additionally, some of these commands have "side effects" in the LFC: e.g. the lcg-cr command uploads a file to an SE and registers this file in the LFC under a certain name.
These commands only operate on the LFC, and don't manipulate data. E.g. the command lfc-mkdir creates a new directory in the LFC.

Important: before you can store, retrieve and replicate data, you need:

  • a valid Grid proxy
  • the environment variable LFC_HOST set to you LFC server. For all SARA's user interface machines, this should be You could put the following in your $HOME/.bash_profile:
    export LFC_HOST=''
  • LCG command line utilities reference guide is here
  • Or an overview with explantation lcg: here
  • lfc commands here

Listing files and directories

For each of the supported VO's, a separate "top level" directory exists under the /grid/ directory. E.g. to see all the files that are stored for the lsgrid VO, make sure you have a running lsgrid VOMS proxy and then type:

$> lfc-ls -l /grid/lsgrid/
drwxrwxr-x   2 30125    3010                      0 Feb 05 12:56 arni
drwxrwxr-x   3 30146    3010                      0 Mar 06 15:21 dutilh
drwxrwxr-x   3 30147    3010                      0 Feb 22 16:12 emc-gwatest

Rather than having to type an absolute path for every file and directory you use, it is instead possible to define a home directory from which you may use relative file/directory paths. You can do this by setting the environment variable LFC_HOME:

$> export LFC_HOME='/grid/lsgrid' 

Creating a new directory

In the next examples, you should of course replace your_username and your_vo, so don't just cut and paste !

Before you can register any file of your own, you must create a new directory in the file catalog:

$> lfc-mkdir /grid/your_vo/your_username

To check that you have created your directory type:

$> export LFC_HOME=/grid/your_vo
$> lfc-ls -l 

and you should see your directory (plus possibly those of others).

Storing a file on a Storage Element (SE)

Use the lcg-cr command (cr stands for copy-register) to upload a file to an SE. This command not only uploads the file, but also registers it into the LFC, so you can easily find it later. For example, to store a local file text_file.txt on an SE (in this example the SRM at SARA), and register it in the LFC with a logical name, do the following:

$> echo 'Put something here' > text_file.txt
$> lcg-cr --vo your_vo -d srm:// \
   -l lfn:/grid/your_vo/your_username/text_file.txt "file://$PWD/text_file.txt"

The guid is a unique identifier for the file you just uploaded. As you will see below, replica's of the same file share this guid.

You should now be able to see this file in the LFC:

$> lfc-ls -l /grid/your_vo/your_username/text_file.txt
-rw-rw-r--   1 30085    3010     19 Mar 12 15:16 /grid/your_vo/your_username/text_file.txt

Before continuing it is worth noting the difference between the command used to store the file and the creation of the directory in previous. The directory created is just a virtual directory and only exists within the catalog of LFN's. The file, on the other hand, physically exists on an SE but has an additional "virtual" filename in the catalog.

File replication

A single file can be stored on multiple SE's. Running jobs can retrieve the files they need from an SE nearby, thus giving faster access times to the data. This also helps protect against failures/access difficulties with a particular SE.

To find out which replica's exist for a certain file, use the lcg-lr command. (lr stands for List Replicas.)

$> lcg-lr lfn:/grid/your_vo/your_username/text_file.txt

In this example, there are two replica's of the file.

To replicate this file to yet another SE, do this:

$> lcg-rep --vo your_vo -d yet_another_se lfn:/grid/your_vo/your_username/text_file.txt

If you now run the lcg-lr command again, you should see that there's a third, new, replica of the same file.

Note how the path for each replica is different. This demonstrates how the use of a "lfn" avoids the need to understand the local filesystem where the replica is actually stored.

Retrieving files from a Storage Element

To download a file you already uploaded, use the lcg-cp command. (cp stands for copy.)

$> lcg-cp --vo your_vo lfn:/grid/your_vo/your_username/text_file.txt file://$PWD/text_file.txt

Unfortunately, this command doesn't necessarily download the file from a nearby SE. If the file is replicated by various SE's, then the file is downloaded from a random SE on which a replica of the file resides.

To make sure you download the replica from a nearby SE, do this:

$> SURL=`lcg-lr "lfn:/grid/your_vo/your_username/text_file.txt" |
         grep "//$VO_YOURVO_DEFAULT_SE/"`
$> lcg-cp --vo your_vo "$SURL" file://$PWD/text_file.txt

The first line lists all replica SURL's of the file, and greps for the name of the nearby SE. Not the file LFN but the SURL is then used to retrieve the file. The $VO_YOURVO_DEFAULT_SE environment variable always contains the name of the SE nearest to your current User interface or Worker node. To find out which variable to use, try this:

$> set | grep _DEFAULT_SE

Your VO should be listed there.

Deleting files from a Storage Element

To delete a file from all storage elements, do

$> lcg-del -a lfn:/grid/your_vo/your_username/text_file.txt

The -a option makes sure that all replicas of a file are removed. See the lcg-del man-page for more options.


The logical file catalog is a place where you register files, so you can find their replicas that a physically stored on a storage element.

If the physical storage is removed or lost, and you don't have any other replica's, you end up with only a registration in the lfc.

Working from a non Glite Environment


E-bioinfra wiki, with some useful scripts

Personal tools