FTS Usage
|
EGEE Home | JRA1 Home | JRA1 DM Home | EDMS Documents | People | Calendar | Agenda maker | Glossary |
|
|
|
FTS Usage |
|
|
|
|
|
These three interfaces provide the following functionality:
Command-Line ToolsThe gLite FTS provides a set of command-line client tools that can be called as executables to interact with an FTS service. Please note that some calls require special authorization. See the authorization matrix. There are several sets of command line tools:
All FTS Interface command-line tools (the first three in the list above) and the SRM support the options in the table below. These interfaces also have manual pages (online versions thereof are linked in the tables) that you can always view from the command line if your MANPATH variable contains $GLITE_LOCATION/share/man.
CLI for File Transfer InterfaceThe following command-line tools are available for the File Transfer interface funcitons. All commands accept the common options described above.
CLI for the Channel Management InterfaceThe following command-line tools are available for the File Transfer interface funcitons. All commands accept the options described above.
|
| glite-transfer-stats-agents | List all agents running on the service |
||||||
glite-transfer-stats-channel CHANNEL |
Lists the summary and activity of channel usage for the given channel.
|
||||||
| glite-transfer-stats-vo VONAME | Lists the summary and activity of VO usage.
|
The following command-line tools should be used carefully, they allow direct interaction with the SRM interface so you should be familiar with the SRM to use them correctly. Currently only version 1 of the SRM interface is supported, but version 2 will be supported too as well. These command-line tools also support the usual common options as the FTS tools do. It is not really part of the FTS interface but rather a separate set of tools, usually used to check whether an SRM responds properly to the proper SOAP calls by a given client.
| glite-srm-delete SURL ... | Do an srm delete on the given SURLs. |
||||
glite-srm-get SURL .. |
Perform an SRM get call on the given SURLs. The purpose of that call is to retrieve a valid Transport URL (TURL) with an actual access protocol (by default gsiftp, or specified with the -p option) for the given SURLs. The call returns the request ID which can be used on subsequent status commands to see the status of the current request and to actually retrieve the TURLs of the SURLs.
|
||||
| glite-srm-get-protocols | List the protocols supported by the SRM. |
||||
| glite-srm-get-status REQUEST_ID | Retrieve the complete information
on a request previously made on the SRM. This information includes the id, type, state and other items belonging to the request itself, as well as additional information on each file inside it. |
||||
| glite-srm-mk-permanent SURL .. | Indicate to the SRM that the given SURLs are to be made persistent. | ||||
| glite-srm-pin SURL .. | Advise the SRM that the client will be using the file in the near future. This pin has as expiration time, set by the SRM. A second pin operation will renew the timeout, subject to local policy. | ||||
| glite-srm-ping | Check that the serivce is alive. | ||||
| glite-srm-set-status REQUEST_ID FILE_ID STATE | Update the current status of a file in a specific request. This will normaly be changing the state from 'Pending' to 'Running' once a transfer is started, and then to 'Done' once it is finished, or from any state to 'Failed'. | ||||
| glite-srm-put | Perform an SRM put call on the given SURLs. The purpose of that call is to retrieve a valid Transport URL (TURL) with an actual access protocol (by default gsiftp, or specified with the -p option) for the given SURLs. The call returns the request ID which can be used on subsequent status commands to see the status of the current request and to actually retrieve the TURLs of the SURLs.
|
||||
| glite-srm-unpin TURL .. REQUEST_ID | Release the pin previously acquired with srm-pin on the TURLs. |
The following command-line tools allow the user to send direct commands to a Gridftp Server. They are the gLite port and extension of the EDG Gridftp command line tools. The common options they all share are:
|
Option
|
Description
|
| -h | Print a short help message on parameters and usage, and exit. |
| -n | No data channel authentication (gridftp NODCAU option). (Only relevant for exists, ls) |
All commands take a fully qualified gridftp URL as their argument. These URLs have 'gsiftp' for their schema and contain the full hostname of the remote server. Optionally also the port can be specified (it defaults to 2811). Example: gsiftp://gserver.cern.ch/data/user/smith/file1.dat
| glite-gridftp-exists GRIDFTP_URL | Check the existence of a file. The -n option will switch off data channel authentication (nodcau). |
glite-gridftp-ls [-l] GRIDFTP_URL |
List a directory on a remote gridftp server. The given GRIDFTP_URL must refer to a directory. If the -l option is given, the listing will be more verbose, printing the size and date of the files. |
| glite-gridftp-mkdir [-p] GRIDFTP_URL | Make a new directory, given in the argument. If the -p option is given, nonexisting parent directories will also be created (just like for the unix mkdir command). |
| glite-gridftp-rename SOURCE_URL DEST_URL | Rename a file on a remote
gridftp server.
The two URLs must have the same host.
|
| glite-gridftp-rm GRIDFTP_URL | Remove a file from the gridftp server. |
| glite-gridftp-size GRIDFTP_URL | Get the size of a file. |
The following command-line tools can be used to test SRM - SRM transfer directly. These commands are used by the FTS server when the actual transfers are performed so it is also a great tool to test the transfer part of the FTS. For the regular operations please use the FTS instead. The command is derived from the well-known globus-url-copy command that has been part of the globus toolkt since GT2. The glite version of this command has the same functionality as the globus version, with two important additions:
The table below gives the common options for the glite-url-copy command line tools.
glite-url-copy SOURCE TARGET |
Copy a file. This command is asynchronous and will return immediately with a url-copy ID that can be used with subsequent calls of glite-url-copy-status. The source and target are to be given as fully qualified URLs. Accepted URL schemas are:
The 'srm' schema is used for SURLs. In order to copy files to or from gridftp servers (gsiftp schema) or SRMs, you have to have a valid proxy certificate at the time of the call. The command accepts the following options:
|
||||||||||||||||||||||||||||
| glite-url-copy-srm SOURCE TARGET ... | -f FILE | The srm-copy version of the glite-url-copy command. In the glite-url-copy case, an SURL is translated into a valid gsiftp TURL using the SRM interface and gridftp is used to transfer the file. If this command is used, the native SRM copy mechanism is used to transfer the file instead. The drawback in using this method is the very limited control the client has on the command once it's been issued. There can be a number of sources and targets given on the command line (or a file containing one pair per line can be given) as the SRM copy method accepts multiple files to be copied with a single call.
|
||||||||||||||||||||||||||||
| glite-url-copy-status ID | Print the status of the ongoing transfer with the given ID.
|
||||||||||||||||||||||||||||
| glite-url-copy-stop ID | Stop the ongoing transfer with the given ID.
|
This is the info provider for the BDII information system, to be used to publish instances of FTS in to BDII. The full description and options can be found here. The info provider is only used by the administrators of the FTS service.
See the Javadoc pages for details as well.
The gLite FTS is a secure Web Service that comes with a WSDL and communicates over https using SOAP. Therefore clients can be generated in every language easly from the WSDL. The Java client is automatically generated using the Apache Axis wsdl2java tool. There are four packages in the Java interface. These are:
The following listing gives an example how to connect to the FTS using the Java API. For brevity, some familiarity with Java is assumed, in particular how to catch and handle properly standard Java language exceptions (the listing code is not a good example in this regard); the class definition is also omitted. The FileTransferServiceLocator class is used to retrieve an instance of the FileTransfer interface, reachable at the service endpoint specified by the URL that is passed to the getFileTransfer method (line 16).
| 1 import org.glite.data.transfer.fts.FileTransfer; 2 import org.glite.data.transfer.fts.FileTransferServiceLocator; 3 import java.net.URL; 4 import javax.xml.rpc.ServiceException; 5 6 public static final String SERVICE_ENDPOINT = 7 "https://host.domain:8443/SERVICEID/" + 8 "glite-data-transfer-fts/services/FileTransfer"; 9 10 private FileTransfer getService() throws Exception { 11 URL url = new URL(SERVICE_ENDPOINT); 12 FileTransfer fts = null; 13 FileTransferServiceLocator ftsLoc = new FileTransferServiceLocator(); 14 15 try { 16 fts = ftsLoc.getFileTransfer(url); 17 } catch (ServiceException se) { 18 // Cannot connect. Handle the exception. 19 } 20 21 return fts; 22 } |
Listing 1: Acquiring the FTS client object.
The same mechanism can be used to acquire a ChannelManagement or a FileTransferStats interface implementation. The Locator object has to be created and its getInterface method invoked where Interface is actually the name of the interface desired.
In order to submit a job through Java, you have to call the 'submit' or 'placementSubmit' methods on the interface, as explained in the Javadoc.
In order to submit a job, the job description needs to be built up.
This is done using the TransferJob object.
The listing below shows an example of creating and submitting
a
basic job. For brevity, the surrounding class and the imports are omitted.
The class TransferJob has three members with normal Java bean setter
methods:
| 1 private void runExample() { 2 FileTransfer fts = getService(); 3 TransferJob tjob = new TransferJob(); 4 5 TransferJobElement[] te_array = new TransferJobElement[10]; 6 for (int i = 0; i < 10; i++) { 7 String src = 8 "srm://host.test1:8443/srm/managerv1?SFN=/data/test1/testfile" + i; 9 String dest = 10 "srm://dest.test2:8443/srm/managerv1?SFN=/filestore/destfile" + i; 11 TransferJobElement te = new TransferJobElement(); 12 te.setSource(src); 13 te.setDest(dest); 14 te_array[i] = te; 15 } 16 17 tjob.setTransferJobElements(te_array); 18 19 tjob.setJobParams(null); // the current service does not support this 20 21 tjob.setCredential("myproxypassword"); // your Myproxy password 22 23 try { 24 String jobId = fts.submit(tjob); 25 } catch (InvalidArgumentException e) { 26 // Malformed job description. Handle the exception.. 27 } catch (InternalException e) { 28 // Internal service error. Handle the exception. 29 } catch (AuthorizationException e) { 30 // Not allowed to submit or bad credentials were supplied. Handle it. 31 } catch (ServiceBusyException e) { 32 // The service cannot process your request just now. Handle it. 33 } catch (RemoteException e) { 34 // Something else went wrong. Handle the exception. 35 // This exception is also the superclass of all the others, 36 // so can be used alone to catch all of them. 37 } 38 } // end runExample |
Listing 2: Submitting a simple job with 10 files.
| Line 2 | The connection stub is initialised. |
| Line 5 | This job consists of 10 files |
| Line 6-15 | 10 TransferJobElements are created, and the source and destination files set using their Java bean setter methods. |
| Line 17 | The array of TransferJobElements is added to the job using the appropriate setter method. |
| Line 19-21 | The other setters are called. |
| Line 24 | The call is made to the web-service. Upon success the job identifier returned from the service will be put into the String variable jobId. |
| Line 25 | InvalidArgumetException. This exception occurs if the job object is malformed or incomplete. |
| Line 27 | InternalException. This exception occurs if there is an internal problem on the service. |
| Line 29 | AuthorizationException. This exception occurs if the credential sent to the service is not authorized for the submit action or if the client's credential is invalid. |
| Line 31 | ServiceBusyException. This exception occurs if the service is too busy to accept the job. |
| Line 33 | RemoteException.
This exception occurs if something
goes wrong in the web-service layer or with the connection. It must always
be caught. This class is the superclass of all the other exceptions, so catching it is a lazy way of catching all the others (though it is recommended to catch and handle the others first). |
The status ID as returned by the submit call can be used on subsequent 'status' calls to poll the status of the given transfer. See the Javadoc for more details.
The FTS comes with a detailed C API and also a library of autogenerated C++ classes. The C API provides a simple version of the API where dedicated constructor and destructor methods are provided to create and destroy the complex types that are used in the API.
This 'simple' C API comes with a complete doxygen documentation.
The CLI tools themselves are the best examples on how to use this API. See the source of these tools in CVS.
The FTS also has API bindings generated for Perl:
Actually any language may have bindings that has some mechanism to generate client-side code from the WSDL. The WSDL for the FTS can be found at the autogenerated documentation pages of JRA1 DM. The versions for gLite 1.5 are:
|
| Disclaimer | Mail to FTS Support | Last Modified: 08/12/2005December 8, 2005 |
