Jobsub ID 44845.76@dunegpschedd01.fnal.gov

Jobsub ID	44845.76@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-11 10:04:34
Site	US_UCSD
Entry	CMSHTPC_T2_US_UCSD_gw6
Last heartbeat	2025-09-11 12:16:05
From worker node	Hostname	mh-7763-20.t2.ucsd.edu
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-11 10:25:50
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_1005_20231203T084741Z_gen_g4_detsim_hitreco__20240509T201917Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_664_20231207T150512Z_gen_g4_detsim_hitreco__20240510T061732Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_339_20231207T024352Z_gen_g4_detsim_hitreco__20240510T043722Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_63_20231207T022347Z_gen_g4_detsim_hitreco__20240510T042531Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74548636_235_20231207T070730Z_gen_g4_detsim_hitreco__20240510T055958Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_302_20231208T054227Z_gen_g4_detsim_hitreco__20240510T045633Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_639_20231204T044938Z_gen_g4_detsim_hitreco__20240509T210108Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_511_20231204T003748Z_gen_g4_detsim_hitreco__20240509T205543Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_59_20231208T073751Z_gen_g4_detsim_hitreco__20240510T055517Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_418_20231208T000832Z_gen_g4_detsim_hitreco__20240510T030657Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-11 12:16:05
Saved logs	justin-logs:44845.76-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.000380825    0.0492409      4.63883     0.00393554     0.276283        500    
reco:opdec:Deconvolution                               0.455312      0.884193       2.98503      0.809939      0.306307        500    
reco:ophitspe:OpHitFinderDeco                           1.3409        1.40572       2.52918       1.36696      0.0998403       500    
reco:opflash:OpFlashFinder                            0.000226976   0.00165835    0.00643485    0.00160572    0.000733929      500    
reco:opslicer:OpSlicer                                7.8959e-05     0.310528       1.37999      0.239236       0.25638        500    
reco:rns:RandomNumberSaver                             2.583e-05    5.68851e-05   0.000516782   5.20095e-05   3.09618e-05      500    
reco:anglereconue:NuAngularReco                       0.000209227   0.00175379     0.046747     0.000550713   0.00453472       500    
reco:anglereconumu:NuAngularReco                      8.9169e-05     0.0015091     0.0449966    0.000315215   0.00440605       500    
reco:anglereconuepfps:NuAngularReco                   0.000131648    0.001658      0.0481225    0.000461203   0.00449715       500    
reco:anglereconumupfps:NuAngularReco                  0.000112158   0.00159449     0.046841     0.000406239   0.00446233       500    
reco:anglerecohits:NuAngularReco                      0.000135809   0.00240252     0.0529393    0.000921757   0.00521138       500    
[art]:TriggerResults:TriggerResultInserter             9.72e-06     1.63863e-05   8.7459e-05    1.52945e-05   6.56372e-06      500    
end_path:cafmaker:CAFMaker                            0.00142015     0.0234356     0.619381     0.00781511     0.0563325       500    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3649.78 MB
  Peak resident set size usage (VmHWM): 1971.53 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 11-Sep-2025 05:15:38 PDT ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/77/97/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_302_20231208T054227Z_gen_g4_detsim_hitreco__20240510T045633Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 626!
Begin processing the 492nd record. run: 74548636 subRun: 1 event: 23592 at 11-Sep-2025 05:06:35 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 451!
Begin processing the 493rd record. run: 74548636 subRun: 1 event: 23593 at 11-Sep-2025 05:06:43 PDT
Boundary wire vector sizes: 538, 581, 359
minwire 0: 2267
minwire 1: 130
minwire 2: 2578
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000398744, 0.997942, 0.00165904, 
Output 1: 0.211377, 0.699108, 0.0830978, 0.00641786, 
Output 2: 0.0193248, 0.90591, 0.0730766, 0.00168837, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1267!
Begin processing the 494th record. run: 74548636 subRun: 1 event: 23594 at 11-Sep-2025 05:06:53 PDT
Boundary wire vector sizes: 74, 72, 51
minwire 0: 2271
minwire 1: 972
minwire 2: 2213
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000437889, 0.998053, 0.00150886, 
Output 1: 0.194294, 0.800851, 0.004649, 0.000205285, 
Output 2: 0.998906, 0.00106548, 2.75737e-05, 1.05864e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 927!
Begin processing the 495th record. run: 74548636 subRun: 1 event: 23595 at 11-Sep-2025 05:07:05 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 950!
Begin processing the 496th record. run: 74548636 subRun: 1 event: 23596 at 11-Sep-2025 05:07:12 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1197!
Begin processing the 497th record. run: 74548636 subRun: 1 event: 23597 at 11-Sep-2025 05:07:20 PDT
Boundary wire vector sizes: 422, 343, 254
minwire 0: 2199
minwire 1: 429
minwire 2: 2436
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00083551, 0.991461, 0.00770347, 
Output 1: 0.977613, 0.0219034, 0.000335015, 0.000148737, 
Output 2: 0.998184, 0.00174975, 5.17315e-05, 1.45788e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1346!
Begin processing the 498th record. run: 74548636 subRun: 1 event: 23598 at 11-Sep-2025 05:07:32 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1025!
Begin processing the 499th record. run: 74548636 subRun: 1 event: 23599 at 11-Sep-2025 05:07:39 PDT
Boundary wire vector sizes: 72, 79, 84
minwire 0: 1763
minwire 1: 629
minwire 2: 1814
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.242499, 0.00384723, 0.753654, 
Output 1: 0.00977752, 0.346454, 0.627067, 0.0167024, 
Output 2: 0.853266, 0.142058, 0.00462292, 5.31006e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1028!
Begin processing the 500th record. run: 74548636 subRun: 1 event: 23600 at 11-Sep-2025 05:07:47 PDT
Boundary wire vector sizes: 72, 42, 48
minwire 0: 755
minwire 1: 933
minwire 2: 21
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 1144, 1443
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.285425, 0.00500892, 0.709566, 
Output 1: 0.251626, 0.601116, 0.134179, 0.0130784, 
Output 2: 0.374027, 0.60875, 0.0169227, 0.00030068, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1458!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a2/e7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_1005_20231203T084741Z_gen_g4_detsim_hitreco__20240509T201917Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/1c/42/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_664_20231207T150512Z_gen_g4_detsim_hitreco__20240510T061732Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/eb/17/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_339_20231207T024352Z_gen_g4_detsim_hitreco__20240510T043722Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/36/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_63_20231207T022347Z_gen_g4_detsim_hitreco__20240510T042531Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/8a/07/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74548636_235_20231207T070730Z_gen_g4_detsim_hitreco__20240510T055958Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/77/97/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_302_20231208T054227Z_gen_g4_detsim_hitreco__20240510T045633Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b4/d5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_639_20231204T044938Z_gen_g4_detsim_hitreco__20240509T210108Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ac/b2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_511_20231204T003748Z_gen_g4_detsim_hitreco__20240509T205543Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/1c/d9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_59_20231208T073751Z_gen_g4_detsim_hitreco__20240510T055517Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/3f/a8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_418_20231208T000832Z_gen_g4_detsim_hitreco__20240510T030657Z_reco2.root
.:
total 23472
-rw-r--r-- 1 cuser cuser 8864958 Sep 11 05:15 flatcaf.root
-rw-r--r-- 1 cuser cuser 7346388 Sep 11 05:15 caf.root
-rw-r--r-- 1 cuser cuser 7346388 Sep 11 05:15 caf_fd_hd_atmo_2501_20250911T102600Z.root
-rw-r--r-- 1 cuser cuser  305468 Sep 11 05:15 caf_20250911T102600Z.log
-rw-r--r-- 1 cuser cuser  112956 Sep 11 05:15 jobscript.log
-rw-r--r-- 1 cuser cuser    1883 Sep 11 05:15 caf_20250911T102600Z.file
-rw-r--r-- 1 cuser cuser    1883 Sep 11 05:15 caf_20250911T102600Z.pfns
-rw-r--r-- 1 cuser cuser    1883 Sep 11 03:26 file.list
-rw-r--r-- 1 cuser cuser    1883 Sep 11 05:15 justin-processed-pfns.txt
-rw-r--r-- 1 cuser cuser    1371 Sep 11 03:26 all-input-dids.txt
-rw-r--r-- 1 cuser cuser    1371 Sep 11 05:15 caf_20250911T102600Z.did
-rw-r--r-- 1 cuser cuser    1371 Sep 11 03:26 did.list
-rw-r--r-- 1 cuser cuser       0 Sep 11 03:26 debugprod.log