TroubleShooting080212dq2

New features added to dq2_put

Working with the registration of Monica's and Fred's datasets I found the need for some new features in dq2_put. I implemented these features only in LRC (no LFC support) since this was my (and Fred's) immediate need. If none of the new switches is used the behavior is the same.

New root directory

Option: -n | --newbase

Overrides the base URL in the PFN name. Pay attention that there is no check that files actually exist in the SE and are correct. All thew files have to be in the same directory in the SE (the only element changing in the PFN is the file name). Local copy is used for MD5 sum evaluation

Fast registration

dq2_put normally works by evaluating first file size and MD5 checksum of all the files to register. That's an expensive task. Furthermore sometime it has to copy the files to a local working directory before being able to evaluate file size and MD5 checksum, one file at the time. These options speed up dataset registration if the files are already registered in the LRC.

Option: -e | --exist

Register only files that already exist in the Local Catalog (e.g. LRC, LFC, ...). This allows to avoid to copy the file locally and having to evaluate file size and MD5 checksum. dq2_put fails if one or more files are not in the catalog.

NOTE: Currently works only with LRC

Option: -f | --fast-registration

Check first if LFN already exist in the Local Catalog (e.g. LRC, LFC, ...). If the LFN exists, GUID, file size and md5 checksum are extracted from the catalog. This avoids to copy that file locally and having to evaluate file size and MD5 checksum for files in the catalog. It speeds up considerably the registration process.

NOTE: Currently works only with LRC

Dataset registration problems were caused by a bug

I found that there is a bug in dq2_put that causes errors in dataset registrations when:

• the dataset is bigger than 40 files
• files are registered in LRC and some files are not new files (GUID is in LRC)

Description:

in function _getPFNsLRC the double use of variable name lfn was causing to miss the last lfns if these were more than 40. There was no query to LRC and file data was not added to the list and ignored.

The same error caused Bug #31586 (CERN savannah portal). It was not BNL LRC problem.

Below is a fixed version of dq2_put (sent also to Tadashi)

I asked for help to fix a dataset registration done when dq2_put had still the bug. The files in the central catalog have different metadata (wrong one) and I cannot add them to the dataset. I cannot change filename and GUID and register them as new files (these are pool files) The dataset in question is Monica's one.

dq2_put.marco -v -e -p /ecache/marco/for_monica/PoolFileCatalog.xml -n gsiftp://uct2-dc1.uchicago.edu/pnfs/uc... user.MarcoMambelli.forMonicaDunford.mSugraGrid_25x25_test080211_2

...

ERROR : addFilesToDataset dq2.content.DQContentException.DQInvalidFileMetadataException [USER][OTHER] The file LFN or GUID is already registered! [4C54BD8B-6505-DC11-ABA9-00E081456413 :: user.MonicaDunford.mSugraGrid_25x25_m0_1020_m12_1050_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root]

Solution

The bug has been submitted to DDM-operations: https://savannah.cern.ch/bugs/index.php?33441 The submission included a fixed version of dq2_put

A follow up of the bug is: https://savannah.cern.ch/bugs/?33445

Some of the tests:

Note that when a dataset name is used it cannot be reused, even if dq2-erase is used. This is to avoid problems with remaining files.

Timing of the copy

Directories in dCache are all of size 512 (no mateer the content). This may cause different numbers if you execute du -sb of the directory. Check the size of the directory itself.

globus-url-copy: 3.23
[uct3-edge5] /ecache/marco/for_monica > time globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/tier3/user/user.MonicaDunford/user.MonicaDunford.mSugraGrid_25x25/ file:///ecache/marco/for_monica/

real    1174m52.369s
user    65m27.974s
sys     64m8.040s
[uct3-edge5] /ecache/marco/for_monica > 

[uct3-edge5] /ecache/marco/for_monica > du -s .
841719892       .
[uct3-edge5] /ecache/marco/for_monica > ls |wc
    579     579   63645

[uct3-edge5] /ecache/marco/for_monica > du -sb .
861919978959    .

Previous registrations of Monica's dataset

user.MonicaDunford.mSugraGrid_25x25_m0_1020_m12_450_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
-rw-r--r--  1 marco mwt2 1900168480 Jan 30 15:13 /ecache/marco/for_monica//user.MonicaDunford.mSugraGrid_25x25_m0_1020_m12_450_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
ls -l /ecache/marco/for_monica//user.MonicaDunford.mSugraGrid_25x25_m0_1020_m12_510_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
-rw-r--r--  1 marco mwt2 1900204921 Jan 31 05:13 /ecache/marco/for_monica//user.MonicaDunford.mSugraGrid_25x25_m0_1020_m12_510_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
...

rid_25x25%2Fuser.MonicaDunford.mSugraGrid_25x25_m0_900_m12_150_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root%2Bgsiftp%3A%2F%2Fuct2-dc1.uchicago.edu%2Fpnfs%2Fuchicago.edu%2Fdata%2Ftier3%2Fuser%2Fuser.MonicaDunford%2Fuser.MonicaDunford.mSugraGrid_25x25%2Fuser.MonicaDunford.mSugraGrid_25x25_m0_900_m12_210_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root%2Bgsiftp%3A%2F%2Fuct2-dc1.uchicago.edu%2Fpnfs%2Fuchicago.edu%2Fdata%2Ftier3%2Fuser%2Fuser.MonicaDunford%2Fuser.MonicaDunford.mSugraGrid_25x25%2Fuser.MonicaDunford.mSugraGrid_25x25_m0_900_m12_270_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root%2Bgsiftp%3A%2F%2Fuct2-dc1.uchicago.edu%2Fpnfs%2Fuchicago.edu%2Fdata%2Ftier3%2Fuser%2Fuser.MonicaDunford%2Fuser.MonicaDunford.mSugraGrid_25x25%2Fuser.MonicaDunford.mSugraGrid_25x25_m0_900_m12_30_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root%2Bgsiftp%3A%2F%2Fuct2-dc1.uchicago.edu%2Fpnfs%2Fuchicago.edu%2Fdata%2Ftier3%2Fuser%2Fuser.MonicaDunford%2Fuser.MonicaDunford.mSugraGrid_25x25%2Fuser.MonicaDunford.mSugraGrid_25x25_m0_900_m12_330_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root%2Bgsiftp%3A%2F%2Fuct2-dc1.uchicago.edu%2Fpnfs%2Fuchicago.edu%2Fdata%2Ftier3%2Fuser%2Fuser.MonicaDunford%2Fuser.MonicaDunford.mSugraGrid_25x25%2Fuser.MonicaDunford.mSugraGrid_25x25_m0_900_m12_390_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root%2Bgsiftp%3A%2F%2Fuct2-dc1.uchicago.edu%2Fpnfs%2Fuchicago.edu%2Fdata%2Ftier3%2Fuser%2Fuser.MonicaDunford%2Fuser.MonicaDunford.mSugraGrid_25x25%2Fuser.MonicaDunford.mSugraGrid_25x25_m0_900_m12_450_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root%2Bgsiftp%3A%2F%2Fuct2-dc1.uchicago.edu%2Fpnfs%2Fuchicago.edu%2Fdata%2Ftier3%2Fuser%2Fuser.MonicaDunford%2Fuser.MonicaDunford.mSugraGrid_25x25%2Fuser.MonicaDunford.mSugraGrid_25x25_m0_900_m12_510_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root%2Bgsiftp%3A%2F%2Fuct2-dc1.uchicago.edu%2Fpnfs%2Fuchicago.edu%2Fdata%2Ftier3%2Fuser%2Fuser.MonicaDunford%2Fuser.MonicaDunford.mSugraGrid_25x25%2Fuser.MonicaDunford.mSugraGrid_25x25_m0_900_m12_570_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root%2Bgsiftp%3A%2F%2Fuct2-dc1.uchicago.edu%2Fpnfs%2Fuchicago.edu%2Fdata%2Ftier3%2Fuser%2Fuser.MonicaDunford%2Fuser.MonicaDunford.mSugraGrid_25x25%2Fuser.MonicaDunford.mSugraGrid_25x25_m0_900_m12_630_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root%2Bgsiftp%3A%2F%2Fuct2-dc1.uchicago.edu%2Fpnfs%2Fuchicago.edu%2Fdata%2Ftier3%2Fuser%2Fuser.MonicaDunford%2Fuser.MonicaDunford.mSugraGrid_25x25%2Fuser.MonicaDunford.mSugraGrid_25x25_m0_900_m12_690_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root%2Bgsiftp%3A%2F%2Fuct2-dc1.uchicago.edu%2Fpnfs%2Fuchicago.edu%2Fdata%2Ftier3%2Fuser%2Fuser.MonicaDunford%2Fuser.MonicaDunford.mSugraGrid_25x25%2Fuser.MonicaDunford.mSugraGrid_25x25_m0_900_m12_810_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root%2Bgsiftp%3A%2F%2Fuct2-dc1.uchicago.edu%2Fpnfs%2Fuchicago.edu%2Fdata%2Ftier3%2Fuser%2Fuser.MonicaDunford%2Fuser.MonicaDunford.mSugraGrid_25x25%2Fuser.MonicaDunford.mSugraGrid_25x25_m0_900_m12_870_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root%2Bgsiftp%3A%2F%2Fuct2-dc1.uchicago.edu%2Fpnfs%2Fuchicago.edu%2Fdata%2Ftier3%2Fuser%2Fuser.MonicaDunford%2Fuser.MonicaDunford.mSugraGrid_25x25%2Fuser.MonicaDunford.mSugraGrid_25x25_m0_900_m12_90_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root%2Bgsiftp%3A%2F%2Fuct2-dc1.uchicago.edu%2Fpnfs%2Fuchicago.edu%2Fdata%2Ftier3%2Fuser%2Fuser.MonicaDunford%2Fuser.MonicaDunford.mSugraGrid_25x25%2Fuser.MonicaDunford.mSugraGrid_25x25_m0_900_m12_930_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root%2Bgsiftp%3A%2F%2Fuct2-dc1.uchicago.edu%2Fpnfs%2Fuchicago.edu%2Fdata%2Ftier3%2Fuser%2Fuser.MonicaDunford%2Fuser.MonicaDunford.mSugraGrid_25x25%2Fuser.MonicaDunford.mSugraGrid_25x25_m0_900_m12_990_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root"
data="archivals=P%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP%2BP"
(0, 1)

Missing files in Monica's dataset

These were due to the bug. 579 divided by 40 has 19 remaining at the end.

Registered files: 560, total: 579, Missing files are: +user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_1350_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root+user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_1410_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root+user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_1470_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root+user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_150_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root+user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_210_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root+user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_270_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root+user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_30_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root+user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_330_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root+user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_390_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root+user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_450_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root+user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_510_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root+user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_570_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root+user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_630_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root+user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_690_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root+user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_810_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root+user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_870_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root+user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_90_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root+user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_930_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root+user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_990_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root

LRC SQL investigation

To find if the problem was in the LRC registration of the files and to remove registrations (to start from fresh)

At the beginning I was worried of inconsistencies in LRC tables. This is difficult to verify since DB is not frozen. Content keep changing, so numbers do not match because the content is different. Even if not ruled out completely, probably there are no inconsistencies.

Only investigation possible is to focus on a set of entries

select count(*) from t_pfn join t_lfn where t_pfn.guid=t_lfn.guid;
176199
select count(*) from t_lfn;
176120 
select count(*) from t_pfn;
176143

select count(*) from t_lfn left join t_pfn on t_lfn.guid=t_pfn.guid;
176685 


mysql> select count(*) from t_lfn group by lfname having (count(*)!=1);
Empty set (0.15 sec)

mysql> select count(*) from t_lfn group by guid having (count(*)!=1);
Empty set (0.43 sec)

mysql> select count(*) from t_pfn group by guid having (count(*)!=1);
Empty set (0.38 sec)

mysql> select count(*) from t_pfn group by pfname having (count(*)!=1);
Empty set (0.28 sec)

mysql> select count(*) from t_pfn group by pfname having (count(*)>1);
Empty set (0.27 sec)

Registration of test dataset

Trying different options. Files are recorded in PoolFileCatalog.xml to avoid to have different guids each time dq2_put is executed

[uct3-edge5] /ecache/marco/for_dq2put_test > mv dq2puttest_myfile1.txt dq2puttest_myfile1a.txt 
[uct3-edge5] /ecache/marco/for_dq2put_test > mv dq2puttest_myfile2.txt dq2puttest_myfile2a.txt
[uct3-edge5] /ecache/marco/for_dq2put_test > mv dq2puttest_myfile3.txt dq2puttest_myfile3a.txt
[uct3-edge5] /ecache/marco/for_dq2put_test > mv dq2puttest_myfile4.txt dq2puttest_myfile4a.txt
[uct3-edge5] /ecache/marco/for_dq2put_test > mv PoolFileCatalog.xml* old_pfc1/
[uct3-edge5] /ecache/marco/for_dq2put_test > FCregisterPFN -p dq2puttest_myfile1a.txt 
E8E94DFA-F6D8-DC11-98E5-00E08143888C
[uct3-edge5] /ecache/marco/for_dq2put_test > ls
dq2puttest_myfile1a.txt  dq2puttest_myfile2a.txt  dq2puttest_myfile3a.txt  dq2puttest_myfile4a.txt  old_pfc1  PoolFileCatalog.xml
[uct3-edge5] /ecache/marco/for_dq2put_test > FCregisterPFN -p dq2puttest_myfile2a.txt 
64072C00-F7D8-DC11-8E61-00E08143888C
[uct3-edge5] /ecache/marco/for_dq2put_test > FCregisterPFN -p dq2puttest_myfile3a.txt 
E4046102-F7D8-DC11-A4DF-00E08143888C
[uct3-edge5] /ecache/marco/for_dq2put_test > 
[uct3-edge5] /ecache/marco/for_dq2put_test > dq2_put-marco4.py -v -p /ecache/marco/for_dq2put_test/PoolFileCatalog.xml -n gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/tier3/user/user.MarcoMambelli/user.MarcoMambelli.dq2puttest.t5/ user.MarcoMambelli.dq2puttest.t5
ls -l dq2puttest_myfile1a.txt
-rw-r--r--  1 marco mwt2 15 Feb  6 11:34 dq2puttest_myfile1a.txt
ls -l dq2puttest_myfile2a.txt
-rw-r--r--  1 marco mwt2 15 Feb  6 11:34 dq2puttest_myfile2a.txt
ls -l dq2puttest_myfile3a.txt
-rw-r--r--  1 marco mwt2 200 Feb  6 11:37 dq2puttest_myfile3a.txt
curl --user-agent "dqcurl" --silent --insecure --config def8444e-282f-4694-9117-04d350abbbff http://uct2-grid1.uchicago.edu:8000/dq2/lrc/PoolFileCatalog
data="lfns=dq2puttest_myfile1a.txt+dq2puttest_myfile2a.txt+dq2puttest_myfile3a.txt"
(0, 'Error. Must GET with lfns or guids!')
curl --user-agent "dqcurl" --silent --get --insecure --data "lfns=dq2puttest_myfile1a.txt+dq2puttest_myfile2a.txt+dq2puttest_myfile3a.txt" http://uct2-grid1.uchicago.edu:8000/dq2/lrc/PoolFileCatalog
(0, 'Error. LFNs not found')
curl --user-agent "dqcurl" --silent --insecure --config 3fc66a99-8b2a-4147-87cf-76a52e0e60e5 http://uct2-grid1.uchicago.edu:8000/dq2/lrc/files
data="guids=E8E94DFA-F6D8-DC11-98E5-00E08143888C%2B64072C00-F7D8-DC11-8E61-00E08143888C%2BE4046102-F7D8-DC11-A4DF-00E08143888C"
data="fsizes=15%2B15%2B200"
data="lfns=dq2puttest_myfile1a.txt%2Bdq2puttest_myfile2a.txt%2Bdq2puttest_myfile3a.txt"
data="md5sums=6dac5700f0463020b95791f3715b11a2%2Bb999777fb947d4764cfbb31eb089dce0%2B2c5bb0d7d532b21a401f5f95974ae866"
data="pfns=gsiftp%3A%2F%2Fuct2-dc1.uchicago.edu%2Fpnfs%2Fuchicago.edu%2Fdata%2Ftier3%2Fuser%2Fuser.MarcoMambelli%2Fuser.MarcoMambelli.dq2puttest.t5%2Fuct2-dc1.uchicago.edudq2puttest_myfile1a.txt%2Bgsiftp%3A%2F%2Fuct2-dc1.uchicago.edu%2Fpnfs%2Fuchicago.edu%2Fdata%2Ftier3%2Fuser%2Fuser.MarcoMambelli%2Fuser.MarcoMambelli.dq2puttest.t5%2Fuct2-dc1.uchicago.edudq2puttest_myfile2a.txt%2Bgsiftp%3A%2F%2Fuct2-dc1.uchicago.edu%2Fpnfs%2Fuchicago.edu%2Fdata%2Ftier3%2Fuser%2Fuser.MarcoMambelli%2Fuser.MarcoMambelli.dq2puttest.t5%2Fuct2-dc1.uchicago.edudq2puttest_myfile3a.txt"
data="archivals=P%2BP%2BP"
(0, 1)
[uct3-edge5] /ecache/marco/for_dq2put_test > dq2_ls user.MarcoMambelli.dq2puttest.t5
user.MarcoMambelli.dq2puttest.t5
[uct3-edge5] /ecache/marco/for_dq2put_test > dq2_ls -f user.MarcoMambelli.dq2puttest.t5
user.MarcoMambelli.dq2puttest.t5   Total: 3  - Local: 3
    dq2puttest_myfile1a.txt
    dq2puttest_myfile2a.txt
    dq2puttest_myfile3a.txt
[uct3-edge5] /ecache/marco/for_dq2put_test > FCregisterPFN -p dq2puttest_myfile4a.txt 0ED2C44B-F8D8-DC11-9C57-00E08143888C
[uct3-edge5] /ecache/marco/for_dq2put_test > 

Later attempt to register Monica's dataset

The error returned at the end must be fixed directly handling the central DQ2 DB. Submitted a help request: https://savannah.cern.ch/support/index.php?103413

pnfs/uchicago.edu/data/tier3/user/user.MonicaDunford/user.MonicaDunford.mSugraGrid_25x25/uct2-dc1.uchicago.eduuser.MonicaDunford.mSugraGrid_25x25_m0_900_m12_1230_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root"/>\n    \n    \n      \n    \n    \n    \n    \n    \n  \n\n  \n    \n      \n    \n    \n      \n    \n    \n    \n    \n    \n  \n\n  \n    \n      \n    \n    \n      \n    \n    \n    \n    \n    \n  \n\n')
ls -l user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_1350_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
-rw-r--r--  1 marco mwt2 1440864342 Jan 30 15:06 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_1350_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
ls -l user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_1410_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
-rw-r--r--  1 marco mwt2 1417764911 Jan 31 04:57 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_1410_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
ls -l user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_1470_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
-rw-r--r--  1 marco mwt2 1382890364 Jan 30 21:34 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_1470_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
ls -l3232 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_150_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
-rw-r--r--  1 marco mwt2 1548829237 Jan 30 16:31 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_150_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
ls -l user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_210_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
-rw-r--r--  1 marco mwt2 1677094863 Jan 31 08:18 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_210_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
ls -l user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_270_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
-rw-r--r--  1 marco mwt2 1783837222 Jan 30 22:51 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_270_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
ls -l user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_30_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
-rw-r--r--  1 marco mwt2 1045038350 Jan 31 01:42 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_30_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
ls -l2424 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_330_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
-rw-r--r--  1 marco mwt2 1877862796 Jan 30 15:48 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_330_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
ls -l user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_390_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
-rw-r--r--  1 marco mwt2 1900388387 Jan 31 03:05 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_390_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
ls -l user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_450_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
-rw-r--r--  1 marco mwt2 1900043448 Jan 30 21:38 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_450_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
ls -l user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_510_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
-rw-r--r--  1 marco mwt2 1901076356 Jan 30 15:06 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_510_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
ls -l1616 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_570_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
-rw-r--r--  1 marco mwt2 1864204660 Jan 31 02:02 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_570_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
ls -l user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_630_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
-rw-r--r--  1 marco mwt2 1776378026 Jan 30 18:00 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_630_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
ls -l user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_690_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
-rw-r--r--  1 marco mwt2 1707341150 Jan 31 06:18 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_690_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
ls -l user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_810_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
-rw-r--r--  1 marco mwt2 1617650987 Jan 30 17:05 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_810_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
ls -l 88 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_870_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
-rw-r--r--  1 marco mwt2 1600819587 Jan 31 04:59 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_870_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
ls -l user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_90_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
-rw-r--r--  1 marco mwt2 1347618679 Jan 31 08:09 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_90_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
ls -l user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_930_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
-rw-r--r--  1 marco mwt2 1577585375 Jan 30 23:36 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_930_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
ls -l user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_990_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
-rw-r--r--  1 marco mwt2 1548111387 Jan 30 13:49 user.MonicaDunford.mSugraGrid_25x25_m0_900_m12_990_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root
ERROR : some files are not registered in LRC, run withot option -e (-exist)
[uct3-edge5] /ecache/marco/for_monica > grep 22E9AE39-C306-DC11-AB68-00E081456413 PoolFileCatalog.xml
  
[uct3-edge5] /ecache/marco/for_monica > less PoolFileCatalog.xml
[uct3-edge5] /ecache/marco/for_monica > 

ERROR : addFilesToDataset dq2.content.DQContentException.DQInvalidFileMetadataException [USER][OTHER] The file LFN or GUID is already registered! [4C54BD8B-6505-DC11-ABA9-00E081456413 :: user.MonicaDunford.mSugraGrid_25x25_m0_1020_m12_1050_A0_0_tanBeta_10_sgnMu_p_mTop_175_12.0.6.3_0_aod.pool.root]

-- MarcoMambelli - 12 Feb 2008

• dq2_put: Fixed dq2_put

• dq2_put.marco: Fixed + modified dq2_put

• files_in_user.MarcoMambelli.forMonicaDunford.mSugraGrid_25x25_test080212_1.txt: Files in Monica's dataset