• Getting a user certificate
• Get your credentials
  • Generate an RSA private key and public key pair
  • Get the X509 DN
  • Email the admins
  • scp the X509 key and certificate to the Client VM
  • Connect to the Client VM with ssh
  • Confirm that you can connect to GridFTP
• Rucio Basics
• Uploading a dataset
  • List Datasets:
  • Create a Dataset:
  • Create a File:
  • Upload a File:
  • Attach a File to a Dataset:
  • Replicate the dataset:
  • Download the dataset from a replicate:
  • Delete the rule:

Getting a user certificate

Generate your userkey and certificate, through your institutional login. Select your institution from the https://cilogon.org and click the 'LOG IN' button:

Authenticate by entering your username and password and clicking `LOG IN` at the authentication portal:

The authentication portal may be different for different institutions. As of this writing, UChicago users will need to go through the Duo authentication:

Enter a username and password at the portal and click get new certificate:

Download the p12 file. Take note of the path where you downloaded it to. You will need to know this path in step 3.

Get your credentials

Generate an RSA private key and public key pair

Generate an RSA private key and public key pair with the ssh-keygen tool:

ssh-keygen -t rsa -C "{your email address here}"

A prompt will appear. Press enter until you reach the randomart image:

Start a new email, utilizing this form letter:

Note the path.

Get the X509 DN

To get the x509 DN, run:

cd {location where you stored the P12 in step 2}
openssl pkcs12 -in {filename of your p12 file}.p12 -nocerts -out userkey.pem
openssl pkcs12 -in {filename of your p12 file}.p12 -clcerts -nokeys -out usercert.pem

Get the X509 DN with:

openssl x509 -in usercert.pem -noout -subject

Email the admins

Hello site adminstrators, my name is {name}. I am writing to be added to the Rucio instance. My RSA public key is:

{ssh public key}

My X509 DN is:

{x509 DN}

Send the email to manglano@uchicago.edu, jlstephen@uchicago.edu, lincolnb@uchicago.edu. Continue to the next section after they have replied.

scp the X509 key and certificate to the Client VM

Copy the user key and user certificate to `/home/{your username}/.globus/` with scp.

scp -i /path/to/key userkey.pem {your username}@128.135.158.203:~/userkey.pem
scp -i /path/to/key usercert.pem {your username}@128.135.158.203:~/usercert.pem

Connect to the Client VM with ssh

ssh to the Rucio client at 128.135.158.203 with the username the administrators provide, using the key from the step "Generate an RSA private key and public key pair":

ssh -i /path/to/key {your username}@128.135.158.203

Then `cp` the certificate & key to ~/.globus/

mkdir -p ~/.globus
cp userkey.pem ~/.globus
cp usercert.pem ~/.globus
chmod 644 /home/{your username}/.globus/usercert.pem
chmod 400 /home/{your username}/.globus/userkey.pem

Set the RUCIO_ACCOUNT e.v. to your username, initialize your proxy, and set the X509_CLIENT_PROXY e.v. to your hostname:

export rucio_account={your username}
grid-proxy-init
id -u {your username}
export X509_CLIENT_PROXY=/tmp/x509up_u{result of previous command}

Confirm that you can connect to GridFTP

Run the following to verify your credentials with the GridFTP instances: Initialize the proxy cert to authenticate to GridFTP:

grid-proxy-init

Create a file in your home directory.

cd ~
echo "This is a file for transfer" > uploadable.txt

Copy the file to the GridFTP instances with a new file name and then download those.

gfal-copy file:///home/{your username}/uploadable.txt gsiftp://ruciogridftp1.grid.uchicago.edu/scratch/{your username}/uploadable1.txt
gfal-copy file:///home/{your username}/uploadable.txt gsiftp://ruciogridftp2.grid.uchicago.edu/scratch/{your username}/uploadable2.txt
gfal-copy gsiftp://ruciogridftp1.grid.uchicago.edu/scratch/{your username}/uploadable1.txt file:///home/{your username}/uploadable1.txt
gfal-copy gsiftp://ruciogridftp2.grid.uchicago.edu/scratch/{your username}/uploadable2.txt file:///home/{your username}/uploadable2.txt

Compare the files to the original file.

cmp --print-bytes uploadable1.txt uploadable.txt
cmp --print-bytes uploadable2.txt uploadable.txt

Without seeing output, continue to test the client. The client connects only to the dedicated server instance at this time. Start by pinging the server by `rucio ping`:

rucio ping

Rucio Server will reply with the version. Query your user account information with `rucio whoami`, which should return something like:

status: ACTIVE
account: user
account_type : USER
created_at : 2019-03-26T19:42:33
suspended_at : None
updated_at : 2019-03-26T19:42:33
deleted_at : None
email: None

Rucio Basics

We will introduce the basic motifs of the Rucio CLI here: Upload, Download, List, Rules, RSEs, Scopes, Datasets, and Containers. RSEs, or Rucio Storage Elements, are managed by the instance adminstrators, who can tell you more about available storage options and storage types.

A scope will have been created for your user account. DIDs, or Data Identifiers, have a Scope, which can belong to one or more users; their format is SCOPE:NAME, and each has one of three types: container, dataset, or file. Datasets are groups of files, and containers are groups of datasets, as in the following diagram:

More information on this concept is available here: https://rucio.readthedocs.io/en/latest/overview_File_Dataset_Container.html

When `rucio upload` is called, a file will automatically be registered and uploaded. A scope should have been created for you, called `yourusername`; create a Dataset DID for your account, then view your available datasets with `rucio list-dids {your Rucio account name}:*`.

Uploading a dataset

Run the first two words of any command followed by `-h` to see more information about the command and its functions.

List Datasets:

List the datasets available:

rucio list-dids {your Rucio account name}:*

The table will be empty if you have not done this before.

Create a Dataset:

rucio add-dataset {your Rucio account name}:one_file

Create a File:

Now, create a file, upload a file, and attach it to the dataset you created. If you have an authentication failure, run `grid-proxy-init` again, and if there is another failure, contact your administrator:

echo "This file is to be uploaded" > this.txt

Upload a File:

rucio upload --rse GRIDFTP1SCRATCH_MOCK --scope {your Rucio account name} this.txt

Attach a File to a Dataset:

rucio attach {your Rucio account name}:one_file {your Rucio account name}:this.txt

Replicate the dataset:

rucio add-rule {your Rucio account name}:one_file 1 GRIDFTP2SCR_MOCK

Download the dataset from a replicate:

You will need to wait about ten minutes for replication.

 rucio download --rse GRIDFTP2SCR_MOCK {your Rucio account name}:one_file

Delete the rule:

rucio delete-rule {rule ID}
rucio update-rule --lifetime 1

-- DavidManglano - 03 Apr 2019