Jobsub ID 41555.6@dunegpschedd01.fnal.gov

Jobsub ID	41555.6@dunegpschedd01.fnal.gov
Workflow ID	1980
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-06 12:42:43
Site	FR_CCIN2P3
Entry	DUNE_FR_CCIN2P3_cccondorce01
Last heartbeat	2025-09-06 12:47:14
From worker node	Hostname	ccwcondor0396
	cpuinfo	AMD EPYC 9334 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	106200 (29 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-06 12:44:08
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_303_20231201T071118Z_gen_g4_detsim_hitreco__20240507T184456Z_reco2.root
Jobscript	Exit code	0
	Real time	2m (165s)
	CPU time	0m (37s = 22%)
	Max RSS bytes	1058181120 (1009 MiB)
Outputting started	2025-09-06 12:46:54
Output files
Finished	2025-09-06 12:47:14
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

ecord. run: 50515154 subRun: 1 event: 30331 at 06-Sep-2025 14:45:13 CEST
Begin processing the 32nd record. run: 50515154 subRun: 1 event: 30332 at 06-Sep-2025 14:45:14 CEST
Begin processing the 33rd record. run: 50515154 subRun: 1 event: 30333 at 06-Sep-2025 14:45:16 CEST
Begin processing the 34th record. run: 50515154 subRun: 1 event: 30334 at 06-Sep-2025 14:45:17 CEST
Begin processing the 35th record. run: 50515154 subRun: 1 event: 30335 at 06-Sep-2025 14:45:19 CEST
Begin processing the 36th record. run: 50515154 subRun: 1 event: 30336 at 06-Sep-2025 14:45:20 CEST
Begin processing the 37th record. run: 50515154 subRun: 1 event: 30337 at 06-Sep-2025 14:45:22 CEST
Begin processing the 38th record. run: 50515154 subRun: 1 event: 30338 at 06-Sep-2025 14:45:23 CEST
Begin processing the 39th record. run: 50515154 subRun: 1 event: 30339 at 06-Sep-2025 14:45:24 CEST
Begin processing the 40th record. run: 50515154 subRun: 1 event: 30340 at 06-Sep-2025 14:45:26 CEST
Begin processing the 41st record. run: 50515154 subRun: 1 event: 30341 at 06-Sep-2025 14:45:27 CEST
Begin processing the 42nd record. run: 50515154 subRun: 1 event: 30342 at 06-Sep-2025 14:45:28 CEST
Begin processing the 43rd record. run: 50515154 subRun: 1 event: 30343 at 06-Sep-2025 14:45:30 CEST
Begin processing the 44th record. run: 50515154 subRun: 1 event: 30344 at 06-Sep-2025 14:45:31 CEST
Begin processing the 45th record. run: 50515154 subRun: 1 event: 30345 at 06-Sep-2025 14:45:33 CEST
Begin processing the 46th record. run: 50515154 subRun: 1 event: 30346 at 06-Sep-2025 14:45:34 CEST
Begin processing the 47th record. run: 50515154 subRun: 1 event: 30347 at 06-Sep-2025 14:45:34 CEST
Begin processing the 48th record. run: 50515154 subRun: 1 event: 30348 at 06-Sep-2025 14:45:36 CEST
Begin processing the 49th record. run: 50515154 subRun: 1 event: 30349 at 06-Sep-2025 14:45:36 CEST
Begin processing the 50th record. run: 50515154 subRun: 1 event: 30350 at 06-Sep-2025 14:45:38 CEST
Begin processing the 51st record. run: 50515154 subRun: 1 event: 30351 at 06-Sep-2025 14:45:39 CEST
Begin processing the 52nd record. run: 50515154 subRun: 1 event: 30352 at 06-Sep-2025 14:45:40 CEST
Begin processing the 53rd record. run: 50515154 subRun: 1 event: 30353 at 06-Sep-2025 14:45:42 CEST
Begin processing the 54th record. run: 50515154 subRun: 1 event: 30354 at 06-Sep-2025 14:45:43 CEST
Begin processing the 55th record. run: 50515154 subRun: 1 event: 30355 at 06-Sep-2025 14:45:44 CEST
Begin processing the 56th record. run: 50515154 subRun: 1 event: 30356 at 06-Sep-2025 14:45:46 CEST
Begin processing the 57th record. run: 50515154 subRun: 1 event: 30357 at 06-Sep-2025 14:45:47 CEST
Begin processing the 58th record. run: 50515154 subRun: 1 event: 30358 at 06-Sep-2025 14:45:49 CEST
Begin processing the 59th record. run: 50515154 subRun: 1 event: 30359 at 06-Sep-2025 14:45:50 CEST
Begin processing the 60th record. run: 50515154 subRun: 1 event: 30360 at 06-Sep-2025 14:45:51 CEST
Begin processing the 61st record. run: 50515154 subRun: 1 event: 30361 at 06-Sep-2025 14:45:53 CEST
Begin processing the 62nd record. run: 50515154 subRun: 1 event: 30362 at 06-Sep-2025 14:45:54 CEST
Begin processing the 63rd record. run: 50515154 subRun: 1 event: 30363 at 06-Sep-2025 14:45:56 CEST
Begin processing the 64th record. run: 50515154 subRun: 1 event: 30364 at 06-Sep-2025 14:45:57 CEST
Begin processing the 65th record. run: 50515154 subRun: 1 event: 30365 at 06-Sep-2025 14:45:58 CEST
Begin processing the 66th record. run: 50515154 subRun: 1 event: 30366 at 06-Sep-2025 14:46:00 CEST
Begin processing the 67th record. run: 50515154 subRun: 1 event: 30367 at 06-Sep-2025 14:46:01 CEST
Begin processing the 68th record. run: 50515154 subRun: 1 event: 30368 at 06-Sep-2025 14:46:02 CEST
Begin processing the 69th record. run: 50515154 subRun: 1 event: 30369 at 06-Sep-2025 14:46:04 CEST
Begin processing the 70th record. run: 50515154 subRun: 1 event: 30370 at 06-Sep-2025 14:46:05 CEST
Begin processing the 71st record. run: 50515154 subRun: 1 event: 30371 at 06-Sep-2025 14:46:07 CEST
Begin processing the 72nd record. run: 50515154 subRun: 1 event: 30372 at 06-Sep-2025 14:46:09 CEST
Begin processing the 73rd record. run: 50515154 subRun: 1 event: 30373 at 06-Sep-2025 14:46:10 CEST
Begin processing the 74th record. run: 50515154 subRun: 1 event: 30374 at 06-Sep-2025 14:46:14 CEST
Begin processing the 75th record. run: 50515154 subRun: 1 event: 30375 at 06-Sep-2025 14:46:16 CEST
Begin processing the 76th record. run: 50515154 subRun: 1 event: 30376 at 06-Sep-2025 14:46:17 CEST
Begin processing the 77th record. run: 50515154 subRun: 1 event: 30377 at 06-Sep-2025 14:46:18 CEST
Begin processing the 78th record. run: 50515154 subRun: 1 event: 30378 at 06-Sep-2025 14:46:20 CEST
Begin processing the 79th record. run: 50515154 subRun: 1 event: 30379 at 06-Sep-2025 14:46:21 CEST
Begin processing the 80th record. run: 50515154 subRun: 1 event: 30380 at 06-Sep-2025 14:46:23 CEST
Begin processing the 81st record. run: 50515154 subRun: 1 event: 30381 at 06-Sep-2025 14:46:25 CEST
Begin processing the 82nd record. run: 50515154 subRun: 1 event: 30382 at 06-Sep-2025 14:46:27 CEST
Begin processing the 83rd record. run: 50515154 subRun: 1 event: 30383 at 06-Sep-2025 14:46:29 CEST
Begin processing the 84th record. run: 50515154 subRun: 1 event: 30384 at 06-Sep-2025 14:46:29 CEST
Begin processing the 85th record. run: 50515154 subRun: 1 event: 30385 at 06-Sep-2025 14:46:31 CEST
Begin processing the 86th record. run: 50515154 subRun: 1 event: 30386 at 06-Sep-2025 14:46:33 CEST
Begin processing the 87th record. run: 50515154 subRun: 1 event: 30387 at 06-Sep-2025 14:46:33 CEST
Begin processing the 88th record. run: 50515154 subRun: 1 event: 30388 at 06-Sep-2025 14:46:35 CEST
Begin processing the 89th record. run: 50515154 subRun: 1 event: 30389 at 06-Sep-2025 14:46:36 CEST
Begin processing the 90th record. run: 50515154 subRun: 1 event: 30390 at 06-Sep-2025 14:46:38 CEST
Begin processing the 91st record. run: 50515154 subRun: 1 event: 30391 at 06-Sep-2025 14:46:39 CEST
Begin processing the 92nd record. run: 50515154 subRun: 1 event: 30392 at 06-Sep-2025 14:46:40 CEST
Begin processing the 93rd record. run: 50515154 subRun: 1 event: 30393 at 06-Sep-2025 14:46:42 CEST
Begin processing the 94th record. run: 50515154 subRun: 1 event: 30394 at 06-Sep-2025 14:46:43 CEST
Begin processing the 95th record. run: 50515154 subRun: 1 event: 30395 at 06-Sep-2025 14:46:45 CEST
Begin processing the 96th record. run: 50515154 subRun: 1 event: 30396 at 06-Sep-2025 14:46:46 CEST
Begin processing the 97th record. run: 50515154 subRun: 1 event: 30397 at 06-Sep-2025 14:46:47 CEST
Begin processing the 98th record. run: 50515154 subRun: 1 event: 30398 at 06-Sep-2025 14:46:49 CEST
Begin processing the 99th record. run: 50515154 subRun: 1 event: 30399 at 06-Sep-2025 14:46:50 CEST
Begin processing the 100th record. run: 50515154 subRun: 1 event: 30400 at 06-Sep-2025 14:46:52 CEST
06-Sep-2025 14:46:53 CEST  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/3a/cc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_303_20231201T071118Z_gen_g4_detsim_hitreco__20240507T184456Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.356228       1.16998       2.59904       1.16962      0.275355        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0189652     0.0288087     0.039598      0.0234148    0.00928256       100    
prod:pandorapid:Chi2ParticleID                 0.110843      0.162571      0.191712      0.167818      0.0181297       100    
prod:energyreconue:EnergyReco                  0.196033      0.416921      0.702805      0.395957      0.0809778       100    
prod:energyreconumu:EnergyReco                0.00154013     0.0786784     0.330464      0.067311      0.0600789       100    
prod:energyreconc:EnergyReco                  0.00310049     0.0280029     0.173147      0.0297163     0.0223257       100    
[art]:TriggerResults:TriggerResultInserter    1.3931e-05    2.17672e-05   0.000104228   1.8388e-05    1.24204e-05      100    
end_path:ana:PIDASelectionAna                  0.0220122     0.454361       1.19674      0.438368      0.148541        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19439.8 MB
  Peak resident set size usage (VmHWM): 1058.18 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 56
-rw-r--r-- 1 dune001 lbno 20280 Sep  6 14:46 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_303_20231201T071118Z_gen_g4_detsim_hitreco__20240507T184456Z_reco2_pida_2025-09-06T_124414Z.root
-rw-r--r-- 1 dune001 lbno 14411 Sep  6 14:46 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_303_20231201T071118Z_gen_g4_detsim_hitreco__20240507T184456Z_reco2_reco_2025-09-06T_124414Z.log
-rw-r--r-- 1 dune001 lbno 12111 Sep  6 14:46 jobscript.log
-rw-r--r-- 1 dune001 lbno   217 Sep  6 14:46 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno   138 Sep  6 14:44 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno     0 Sep  6 14:44 debugprod.log