Jobsub ID 44845.72@dunegpschedd01.fnal.gov

Jobsub ID	44845.72@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-11 10:04:34
Site	US_UCSD
Entry	CMSHTPC_T2_US_UCSD_gw6
Last heartbeat	2025-09-11 12:09:15
From worker node	Hostname	mh-7763-18.t2.ucsd.edu
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-11 10:25:21
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496970_242_20231202T044225Z_gen_g4_detsim_hitreco__20240508T053044Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_122_20231207T123643Z_gen_g4_detsim_hitreco__20240510T042351Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_573_20231208T071022Z_gen_g4_detsim_hitreco__20240510T055811Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1793_259_20231027T175134Z_gen_g4_detsim_hitreco__20240503T040306Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_770_20231208T024415Z_gen_g4_detsim_hitreco__20240510T034307Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50568178_873_20231202T172715Z_gen_g4_detsim_hitreco__20240507T210525Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_737_20231202T122042Z_gen_g4_detsim_hitreco__20240507T212837Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_1037_20231205T055615Z_gen_g4_detsim_hitreco__20240509T222244Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74530544_545_20231207T023634Z_gen_g4_detsim_hitreco__20240510T041043Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_232_20231027T230811Z_gen_g4_detsim_hitreco__20240503T044214Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-11 12:09:15
Saved logs	justin-logs:44845.72-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

119706        500    
reco:energyrecnumumcsllhd:EnergyReco                  0.000494684    0.0381971      1.28743     0.00676185     0.119127        500    
reco:opdec:Deconvolution                                0.47368      0.817948       1.5094       0.821827      0.195302        500    
reco:ophitspe:OpHitFinderDeco                           1.36157       1.38109       1.55806       1.37846      0.0147433       500    
reco:opflash:OpFlashFinder                            0.000187568   0.00277871     0.0174713     0.0021693    0.00189097       500    
reco:opslicer:OpSlicer                                8.1959e-05     0.355867       1.24325       0.26177      0.281379        500    
reco:rns:RandomNumberSaver                             2.433e-05    6.21205e-05   0.00313199     4.598e-05    0.000143578      500    
reco:anglereconue:NuAngularReco                       0.000207508   0.00180566     0.0551735    0.000533928   0.00429199       500    
reco:anglereconumu:NuAngularReco                      0.000103279   0.00158072     0.0485274    0.000314101   0.00406443       500    
reco:anglereconuepfps:NuAngularReco                   0.000148318   0.00174159     0.0495959    0.000444294   0.00412438       500    
reco:anglereconumupfps:NuAngularReco                  0.000137398   0.00169817     0.0492571    0.000422985   0.00412077       500    
reco:anglerecohits:NuAngularReco                      0.000128618   0.00260139     0.0552259    0.000991067   0.00488815       500    
[art]:TriggerResults:TriggerResultInserter             1.067e-05    1.88603e-05   9.1969e-05     1.731e-05    8.57373e-06      500    
end_path:cafmaker:CAFMaker                            0.00136847     0.023727      0.566107     0.00706354     0.0524147       500    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3677.95 MB
  Peak resident set size usage (VmHWM): 2101.42 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 11-Sep-2025 05:09:13 PDT ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/50/2d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_770_20231208T024415Z_gen_g4_detsim_hitreco__20240510T034307Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 697, 417, 449
minwire 0: 1085
minwire 1: 2087
minwire 2: 685
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0481453, 0.212442, 0.739413, 
Output 1: 0.265268, 0.587323, 0.132216, 0.0151924, 
Output 2: 0.0523203, 0.837014, 0.106431, 0.0042338, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2610!
Begin processing the 495th record. run: 1793 subRun: 1 event: 51995 at 11-Sep-2025 05:00:42 PDT
Boundary wire vector sizes: 538, 490, 336
minwire 0: 1632
minwire 1: 1130
minwire 2: 1628
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00247093, 0.98217, 0.015359, 
Output 1: 0.0230722, 0.736101, 0.231161, 0.00966507, 
Output 2: 0.979303, 0.0201296, 0.000555102, 1.25467e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2211!
Begin processing the 496th record. run: 1793 subRun: 1 event: 51996 at 11-Sep-2025 05:00:52 PDT
Boundary wire vector sizes: 370, 249, 242
minwire 0: 2405
minwire 1: 405
minwire 2: 2514
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00357902, 0.924258, 0.0721632, 
Output 1: 0.640354, 0.320948, 0.0357728, 0.00292486, 
Output 2: 0.946878, 0.0519142, 0.00115308, 5.46258e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1428!
Begin processing the 497th record. run: 1793 subRun: 1 event: 51997 at 11-Sep-2025 05:01:01 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1475!
Begin processing the 498th record. run: 1793 subRun: 1 event: 51998 at 11-Sep-2025 05:01:08 PDT
Boundary wire vector sizes: 79, 128, 96
minwire 0: 703
minwire 1: 2821
minwire 2: 92
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000562757, 0.998376, 0.00106115, 
Output 1: 0.00344728, 0.991744, 0.0046708, 0.000137837, 
Output 2: 0.997957, 0.0019833, 5.82989e-05, 1.82544e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 916!
Begin processing the 499th record. run: 1793 subRun: 1 event: 51999 at 11-Sep-2025 05:01:16 PDT
Boundary wire vector sizes: 191, 68, 107
minwire 0: 1523
minwire 1: 871
minwire 2: 1775
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.286115, 0.00228356, 0.711602, 
Output 1: 0.0245479, 0.962314, 0.012527, 0.000610989, 
Output 2: 0.255874, 0.723453, 0.0204607, 0.000212103, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1443!
Begin processing the 500th record. run: 1793 subRun: 1 event: 52000 at 11-Sep-2025 05:01:25 PDT
Boundary wire vector sizes: 808, 866, 828
minwire 0: 2406
minwire 1: 689
minwire 2: 2112
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00325494, 0.900524, 0.0962212, 
Output 1: 0.182852, 0.793222, 0.0228391, 0.00108713, 
Output 2: 0.986557, 0.0130579, 0.00036586, 1.92682e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2351!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/7e/bb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496970_242_20231202T044225Z_gen_g4_detsim_hitreco__20240508T053044Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/48/98/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_122_20231207T123643Z_gen_g4_detsim_hitreco__20240510T042351Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b9/05/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_573_20231208T071022Z_gen_g4_detsim_hitreco__20240510T055811Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/34/55/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1793_259_20231027T175134Z_gen_g4_detsim_hitreco__20240503T040306Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/50/2d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_770_20231208T024415Z_gen_g4_detsim_hitreco__20240510T034307Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ac/e8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50568178_873_20231202T172715Z_gen_g4_detsim_hitreco__20240507T210525Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/b1/2b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_737_20231202T122042Z_gen_g4_detsim_hitreco__20240507T212837Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b1/20/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_1037_20231205T055615Z_gen_g4_detsim_hitreco__20240509T222244Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/cc/0a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74530544_545_20231207T023634Z_gen_g4_detsim_hitreco__20240510T041043Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ae/ec/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_232_20231027T230811Z_gen_g4_detsim_hitreco__20240503T044214Z_reco2.root
.:
total 23040
-rw-r--r-- 1 cuser cuser 8689494 Sep 11 05:09 flatcaf.root
-rw-r--r-- 1 cuser cuser 7208357 Sep 11 05:09 caf.root
-rw-r--r-- 1 cuser cuser 7208357 Sep 11 05:09 caf_fd_hd_atmo_2501_20250911T102539Z.root
-rw-r--r-- 1 cuser cuser  304536 Sep 11 05:09 caf_20250911T102539Z.log
-rw-r--r-- 1 cuser cuser  124556 Sep 11 05:09 jobscript.log
-rw-r--r-- 1 cuser cuser    1892 Sep 11 05:09 caf_20250911T102539Z.file
-rw-r--r-- 1 cuser cuser    1892 Sep 11 05:09 caf_20250911T102539Z.pfns
-rw-r--r-- 1 cuser cuser    1892 Sep 11 03:25 file.list
-rw-r--r-- 1 cuser cuser    1892 Sep 11 05:09 justin-processed-pfns.txt
-rw-r--r-- 1 cuser cuser    1370 Sep 11 03:25 all-input-dids.txt
-rw-r--r-- 1 cuser cuser    1370 Sep 11 05:09 caf_20250911T102539Z.did
-rw-r--r-- 1 cuser cuser    1370 Sep 11 03:25 did.list
-rw-r--r-- 1 cuser cuser       0 Sep 11 03:26 debugprod.log