gfal2-util

Description

GFAL 2 utils are a group of command line tools for file manipulations with any protocol managed by gfal2

Installation

yum install gfal2-util gfal2-all

List of command line tools

gfal-cat
Concatenate a list of files into the standard output.
gfal-chmod
Change permissions for a file or directory.
gfal-copy
Copy files from source to destination(s).
gfal-bringonline
Perform a staging operation on the given SURL.
gfal-archivepoll
Perform an archive polling operation on a given SURL.
gfal-legacy-register
Register a new replica into a catalog.
gfal-legacy-replicas
List the replicas of an entry in a catalog.
gfal-legacy-unregister
Remove a replica from a catalog.
gfal-ls
Equivalent to the system `ls` command.
gfal-mkdir
Create a directory.
gfal-rename
Rename a file or directory.
gfal-rm
Remove a file or directory (with -r).
gfal-save
Write standard input to a file.
gfal-stat
Show extended information about a file or directory.
gfal-sum
Retrieve the file checksum.
gfal-xattr
List or display a file's extended attributes.

Third party copy

To understand some parts of this documentation, it is useful to know the difference between a third party copy and a regular one.

Third party copy: When doing a copy between two remote endpoints, the data is sent directly between the two participating storages.
Non-third party copy: The data goes through the client doing the copy.

Gfal2 is capable of doing both, depending on the protocol combination. Some protocols do support third party copy (see below), some others do not. In any case, when copying between two different protocols, the copy will not be third party copy (e.g from GridFTP to DAV).
SRM is an exception to this.

Protocol support

Gfal2 relies on plugins to add support for different protocols. Depending on your installation, some may not be available.
To use a given protocol, the general syntax is protocol://host[:port]/path. For instance:

SRM is an exception to this.

You can check what is installed using rpm -qa | grep gfal2-plugin. The normally available protocols are:

file://
Local file access. It can be omitted (i.e. gfal-ls /).
gsiftp://
GridFTP protocol. It has third-party copy support.
srm://
SRM protocol. Its support for third-party copies depends on the transfer URL negotiated with the endpoint, and the protocol used for the other peer.
gfal2 is capable of resolving the full SURL using the information system (e.g srm://host/path), but otherwise, you can also use the full SURL (e.g srm://host:8446/srm/managerv2?SFN=/path)
root://
XRootD protocol. Some storages support third party copies, others do not.
dav(s)://
WebDAV protocol. Some storages support third party copies, but to enable it, you need to use davs+3rd://
https://
Plain HTTPS protocol. Some storages support third party copies, but to enable it, you need to use https+3rd://
s3://
S3 protocol. Does not support third party copy, but it could still be achieved if the peer is a davs+3rd endpoint
lfc://
LFC endpoints. Requires the endpoint to be part of the url (lfc://lfc.cern.ch/path)
guid:
LFC endpoints, by GUID. It uses the LFC_HOST configured under /etc/gfal2.d/lfc_plugin.conf
rfio://
RFIO protocol.
dcap://
DCAP protocol, only supported by dCache endpoints.

These plugins may not be normally installed.

sftp://
Experimental. SFTP support. Works with an SSH private key, instead of a X509 proxy.
mock://
Useful for testing. Allows to force a given reply depending on the URL used. See its documentation.

Configuration

All plugin configuration files are stored under /etc/gfal2.d/. You can navigate them and have a look at the different configuration options on the gfal2 repository.

In any case, for all commands, any value can be overridden by using the flag -D"GROUP:PARAM=VALUE"

For example, let's say we want to do a listing of an SRM endpoint, but using a different infosys endpoint. 

gfal-ls -D"BDII:LCG_GFAL_INFOSYS=myinfosys.cern.ch:2170" srm://endpoint.cern.ch/dpm/cern.ch/home/dteam/

Examples

Uploading a file to the Grid

gfal-copy -fpK /home/doe/file1 srm://dpmhead-rc.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/dir/nested/file1

In this example, the flag -f tells gfal-copy to overwrite the destination if it already exists, -p enables automatic directory creation, and -K enables checksum validation.

Copying files out of the Grid

gfal-copy can be used for uploading, downloading, and also third party copies. Same syntax applies. 

gfal-copy -fpK srm://dpmhead-rc.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/dir/nested/file1 /home/doe/file1

Listing files and directories

gfal-ls -l srm://dpmhead-rc.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/
-rw-rw-r--   1 46    45          192 Jun 21  2016 853.test
-rw-rw-r--   1 46    45       641064 Jun 16  2016 aaaabbaaa
-rw-rw-r--   1 46    48            0 Jun  6  2016 build.out
-rw-rw-r--   1 46    49        49185 Nov 18 16:04 dmc-901.test
-rw-rw-r--   1 46    49        49185 Nov 18 16:05 dmc-902.test
-rw-rw-r--   1 46    50      5279801 Apr 28 16:20 test

In this example, -l enables long listing, which displays then mode, owner, size and mtime next to each entry. Without -l, the short version is displayed instead: only the names.

gfal-ls uses the environment variable LS_COLORS when -l is passed, if the terminal supports it.

Get information about a file or directory

gfal-stat srm://dpmhead-rc.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/
  File: 'srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch'
  Size: 0	directory
Access: (0755/drwxr-xr-x)	Uid: 45	Gid: 45
Access: 1970-01-01 01:00:00.000000
Modify: 2015-06-10 18:26:00.000000
Change: 2017-05-09 11:42:52.000000

Get the checksum of a file

gfal-sum srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/aalvarez/aaaabbaaa adler32
srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/aalvarez/aaaabbaaa c88e70d6
gfal-sum srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/aalvarez/aaaabbaaa md5
srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/aalvarez/aaaabbaaa 64fd36ae477e970b06949fe90d0c8c20

Get information about space tokens

gfal-xattr srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/ spacetoken
["65e3ad04-ada1-11e6-b531-c860001bd938","bd1dae50-108c-11e7-82fa-c860001bd938","77056126-194f-11e7-82fa-c860001bd938","4ad03b22-2a92-11e7-82fa-c860001bd938","45e35c78-2b52-11e7-82fa-c860001bd938"]
gfal-xattr srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/ spacetoken.token?65e3ad04-ada1-11e6-b531-c860001bd938
{ "spacetoken": "65e3ad04-ada1-11e6-b531-c860001bd938", "owner": "root", "totalsize": 14937625, "unusedsize": 112193261767, "usedsize": 0, "guaranteedsize": 107374182400, "lifetimeleft": -1, "retention": "CUSTODIAL", "accesslatency": "NEARLINE" }

Apendix: lcg-util equivalence table

Description lcg-util gfal-util
Add an alias for a given GUID lcg-aa Deprecated
Bring SURLs online lcg-bringonline gfal-bringonline
Poll archiving status of SURLs gfal-archivepoll
Copy files with no catalog involved lcg-cp gfal-copy
Copy and register a file lcg-cr gfal-copy
Delete a file / directory lcg-del gfal-rm
Get the file checksum lcg-get-checksum gfal-sum
Get the TURLs for given SURLs lcg-getturls gfal-xattr "user.replicas"
Get the TURL for a given SURL lcg-gt gfal-xattr "user.replicas"
List aliases for a given LFN/GUID lcg-la Deprecated
Get the GUID for a given LFN lcg-lg gfal-xattr "user.guid"
Lists the replicas for a given LFN lcg-lr gfal-xattr "user.replicas"
List information / directory lcg-ls gfal-ls
Remove an alias lcg-ra Deprecated
Copy between Ses using the catalog lcg-rep gfal-copy
Register a file in a catalog lcg-rf gfal-copy
Set a file status to done lcg-sd Deprecated*
Get space tokens associated to a description lcg-stmd gfal-xattr "spacetoken"/"spacetoken.token?$tok"/"spacetoken.description?$desc"
Unregister a file lcg-uf gfal-legacy-unregister
Create a directory gfal-mkdir
Dump the standard input into a file gfal-save
Dump a file into the standard output gfal-cat

(*) These actions are actually performed internally by the SRM plugin when needed.

License

This software is licensed under the Apache 2 License. See LICENSE for details.

Copyright (c) 2013-2022 CERN Copyright (c) 2012-2013 Members of the EMI Collaboration See http://www.eu-emi.eu/partners for details on the copyright holders.

Release Notes

See RELEASE-NOTES for a detailed changelog.

Contact

You can notify bugs or ask for feature requests via

Tracker
https://its.cern.ch/jira/browse/DMC/component/12918
Mailing list
dmc-support@cern.ch