Jobsub ID 33925.18@dunegpschedd02.fnal.gov

Jobsub ID	33925.18@dunegpschedd02.fnal.gov
Workflow ID	1980
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-05 05:33:07
Site	CH_UNIBE-LHEP
Entry	OSG_CH_UNIBE_LHEP_ce02
Last heartbeat	2025-09-05 05:40:50
From worker node	Hostname	wn-1-10.local
	cpuinfo	Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	86400 (24 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-05 05:34:26
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_67_20231028T031158Z_gen_g4_detsim_hitreco__20240503T042622Z_reco2.root
Jobscript	Exit code	0
	Real time	6m (361s)
	CPU time	0m (57s = 15%)
	Max RSS bytes	1023537152 (976 MiB)
Outputting started	2025-09-05 05:40:28
Output files
Finished	2025-09-05 05:40:50
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

13528 at 05-Sep-2025 07:38:34 CEST
Begin processing the 129th record. run: 1500 subRun: 1 event: 13529 at 05-Sep-2025 07:38:36 CEST
Begin processing the 130th record. run: 1500 subRun: 1 event: 13530 at 05-Sep-2025 07:38:39 CEST
Begin processing the 131st record. run: 1500 subRun: 1 event: 13531 at 05-Sep-2025 07:38:40 CEST
Begin processing the 132nd record. run: 1500 subRun: 1 event: 13532 at 05-Sep-2025 07:38:41 CEST
Begin processing the 133rd record. run: 1500 subRun: 1 event: 13533 at 05-Sep-2025 07:38:43 CEST
Begin processing the 134th record. run: 1500 subRun: 1 event: 13534 at 05-Sep-2025 07:38:45 CEST
Begin processing the 135th record. run: 1500 subRun: 1 event: 13535 at 05-Sep-2025 07:38:46 CEST
Begin processing the 136th record. run: 1500 subRun: 1 event: 13536 at 05-Sep-2025 07:38:48 CEST
Begin processing the 137th record. run: 1500 subRun: 1 event: 13537 at 05-Sep-2025 07:38:49 CEST
Begin processing the 138th record. run: 1500 subRun: 1 event: 13538 at 05-Sep-2025 07:38:50 CEST
Begin processing the 139th record. run: 1500 subRun: 1 event: 13539 at 05-Sep-2025 07:38:53 CEST
Begin processing the 140th record. run: 1500 subRun: 1 event: 13540 at 05-Sep-2025 07:38:55 CEST
Begin processing the 141st record. run: 1500 subRun: 1 event: 13541 at 05-Sep-2025 07:38:57 CEST
Begin processing the 142nd record. run: 1500 subRun: 1 event: 13542 at 05-Sep-2025 07:38:58 CEST
Begin processing the 143rd record. run: 1500 subRun: 1 event: 13543 at 05-Sep-2025 07:39:00 CEST
Begin processing the 144th record. run: 1500 subRun: 1 event: 13544 at 05-Sep-2025 07:39:01 CEST
Begin processing the 145th record. run: 1500 subRun: 1 event: 13545 at 05-Sep-2025 07:39:03 CEST
Begin processing the 146th record. run: 1500 subRun: 1 event: 13546 at 05-Sep-2025 07:39:04 CEST
Begin processing the 147th record. run: 1500 subRun: 1 event: 13547 at 05-Sep-2025 07:39:07 CEST
Begin processing the 148th record. run: 1500 subRun: 1 event: 13548 at 05-Sep-2025 07:39:08 CEST
Begin processing the 149th record. run: 1500 subRun: 1 event: 13549 at 05-Sep-2025 07:39:09 CEST
Begin processing the 150th record. run: 1500 subRun: 1 event: 13550 at 05-Sep-2025 07:39:11 CEST
Begin processing the 151st record. run: 1500 subRun: 1 event: 13551 at 05-Sep-2025 07:39:12 CEST
Begin processing the 152nd record. run: 1500 subRun: 1 event: 13552 at 05-Sep-2025 07:39:14 CEST
Begin processing the 153rd record. run: 1500 subRun: 1 event: 13553 at 05-Sep-2025 07:39:14 CEST
Begin processing the 154th record. run: 1500 subRun: 1 event: 13554 at 05-Sep-2025 07:39:16 CEST
Begin processing the 155th record. run: 1500 subRun: 1 event: 13555 at 05-Sep-2025 07:39:16 CEST
Begin processing the 156th record. run: 1500 subRun: 1 event: 13556 at 05-Sep-2025 07:39:17 CEST
Begin processing the 157th record. run: 1500 subRun: 1 event: 13557 at 05-Sep-2025 07:39:18 CEST
Begin processing the 158th record. run: 1500 subRun: 1 event: 13558 at 05-Sep-2025 07:39:19 CEST
Begin processing the 159th record. run: 1500 subRun: 1 event: 13559 at 05-Sep-2025 07:39:20 CEST
Begin processing the 160th record. run: 1500 subRun: 1 event: 13560 at 05-Sep-2025 07:39:22 CEST
Begin processing the 161st record. run: 1500 subRun: 1 event: 13561 at 05-Sep-2025 07:39:23 CEST
Begin processing the 162nd record. run: 1500 subRun: 1 event: 13562 at 05-Sep-2025 07:39:25 CEST
Begin processing the 163rd record. run: 1500 subRun: 1 event: 13563 at 05-Sep-2025 07:39:29 CEST
Begin processing the 164th record. run: 1500 subRun: 1 event: 13564 at 05-Sep-2025 07:39:29 CEST
Begin processing the 165th record. run: 1500 subRun: 1 event: 13565 at 05-Sep-2025 07:39:31 CEST
Begin processing the 166th record. run: 1500 subRun: 1 event: 13566 at 05-Sep-2025 07:39:32 CEST
Begin processing the 167th record. run: 1500 subRun: 1 event: 13567 at 05-Sep-2025 07:39:34 CEST
Begin processing the 168th record. run: 1500 subRun: 1 event: 13568 at 05-Sep-2025 07:39:35 CEST
Begin processing the 169th record. run: 1500 subRun: 1 event: 13569 at 05-Sep-2025 07:39:37 CEST
Begin processing the 170th record. run: 1500 subRun: 1 event: 13570 at 05-Sep-2025 07:39:39 CEST
Begin processing the 171st record. run: 1500 subRun: 1 event: 13571 at 05-Sep-2025 07:39:40 CEST
Begin processing the 172nd record. run: 1500 subRun: 1 event: 13572 at 05-Sep-2025 07:39:42 CEST
Begin processing the 173rd record. run: 1500 subRun: 1 event: 13573 at 05-Sep-2025 07:39:43 CEST
Begin processing the 174th record. run: 1500 subRun: 1 event: 13574 at 05-Sep-2025 07:39:45 CEST
Begin processing the 175th record. run: 1500 subRun: 1 event: 13575 at 05-Sep-2025 07:39:46 CEST
Begin processing the 176th record. run: 1500 subRun: 1 event: 13576 at 05-Sep-2025 07:39:47 CEST
Begin processing the 177th record. run: 1500 subRun: 1 event: 13577 at 05-Sep-2025 07:39:49 CEST
Begin processing the 178th record. run: 1500 subRun: 1 event: 13578 at 05-Sep-2025 07:39:50 CEST
Begin processing the 179th record. run: 1500 subRun: 1 event: 13579 at 05-Sep-2025 07:39:52 CEST
Begin processing the 180th record. run: 1500 subRun: 1 event: 13580 at 05-Sep-2025 07:39:53 CEST
Begin processing the 181st record. run: 1500 subRun: 1 event: 13581 at 05-Sep-2025 07:39:54 CEST
Begin processing the 182nd record. run: 1500 subRun: 1 event: 13582 at 05-Sep-2025 07:39:56 CEST
Begin processing the 183rd record. run: 1500 subRun: 1 event: 13583 at 05-Sep-2025 07:39:57 CEST
Begin processing the 184th record. run: 1500 subRun: 1 event: 13584 at 05-Sep-2025 07:40:00 CEST
Begin processing the 185th record. run: 1500 subRun: 1 event: 13585 at 05-Sep-2025 07:40:01 CEST
Begin processing the 186th record. run: 1500 subRun: 1 event: 13586 at 05-Sep-2025 07:40:03 CEST
Begin processing the 187th record. run: 1500 subRun: 1 event: 13587 at 05-Sep-2025 07:40:04 CEST
Begin processing the 188th record. run: 1500 subRun: 1 event: 13588 at 05-Sep-2025 07:40:06 CEST
Begin processing the 189th record. run: 1500 subRun: 1 event: 13589 at 05-Sep-2025 07:40:08 CEST
Begin processing the 190th record. run: 1500 subRun: 1 event: 13590 at 05-Sep-2025 07:40:10 CEST
Begin processing the 191st record. run: 1500 subRun: 1 event: 13591 at 05-Sep-2025 07:40:11 CEST
Begin processing the 192nd record. run: 1500 subRun: 1 event: 13592 at 05-Sep-2025 07:40:14 CEST
Begin processing the 193rd record. run: 1500 subRun: 1 event: 13593 at 05-Sep-2025 07:40:15 CEST
Begin processing the 194th record. run: 1500 subRun: 1 event: 13594 at 05-Sep-2025 07:40:17 CEST
Begin processing the 195th record. run: 1500 subRun: 1 event: 13595 at 05-Sep-2025 07:40:18 CEST
Begin processing the 196th record. run: 1500 subRun: 1 event: 13596 at 05-Sep-2025 07:40:20 CEST
Begin processing the 197th record. run: 1500 subRun: 1 event: 13597 at 05-Sep-2025 07:40:21 CEST
Begin processing the 198th record. run: 1500 subRun: 1 event: 13598 at 05-Sep-2025 07:40:23 CEST
Begin processing the 199th record. run: 1500 subRun: 1 event: 13599 at 05-Sep-2025 07:40:24 CEST
Begin processing the 200th record. run: 1500 subRun: 1 event: 13600 at 05-Sep-2025 07:40:26 CEST
05-Sep-2025 07:40:27 CEST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/df/d4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_67_20231028T031158Z_gen_g4_detsim_hitreco__20240503T042622Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.351915       1.28642       7.04614       1.18342      0.695577        200    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0177603     0.0379814     0.0769996     0.0376533     0.0134915       200    
prod:pandorapid:Chi2ParticleID                 0.105087      0.169872      0.381927      0.168931      0.0371684       200    
prod:energyreconue:EnergyReco                  0.185288      0.413121      0.835942      0.395559      0.105791        200    
prod:energyreconumu:EnergyReco                0.00133156     0.112135       3.71953      0.0684406      0.30198        200    
prod:energyreconc:EnergyReco                  0.00286992     0.0400088     0.380901      0.0342077     0.0501266       200    
[art]:TriggerResults:TriggerResultInserter     9.188e-06    1.42426e-05    4.311e-05    1.33995e-05   4.30497e-06      200    
end_path:ana:PIDASelectionAna                  0.0206725     0.512988       3.41896      0.486813       0.32781        200    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7780.88 MB
  Peak resident set size usage (VmHWM): 1023.54 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 65
-rw-r--r-- 1 dune002 dune 28124 Sep  5 07:40 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_67_20231028T031158Z_gen_g4_detsim_hitreco__20240503T042622Z_reco2_pida_2025-09-05T_053430Z.root
-rw-r--r-- 1 dune002 dune 23606 Sep  5 07:40 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_67_20231028T031158Z_gen_g4_detsim_hitreco__20240503T042622Z_reco2_reco_2025-09-05T_053430Z.log
-rw-r--r-- 1 dune002 dune 11809 Sep  5 07:40 jobscript.log
-rw-r--r-- 1 dune002 dune   182 Sep  5 07:40 justin-processed-pfns.txt
-rw-r--r-- 1 dune002 dune   133 Sep  5 07:34 all-input-dids.txt
-rw-r--r-- 1 dune002 dune     0 Sep  5 07:34 debugprod.log