justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258258.1@dunegpschedd02.fnal.gov

Jobsub ID258258.1@dunegpschedd02.fnal.gov
Workflow ID11012
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 22:51:13
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-06 10:49:08
From worker nodeHostnamedunegli-7989276-0-fnpc18006.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-06 01:38:06
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T001408Z_gen_002099_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time9h (32624s)
CPU time8h (31547s = 96%)
Max RSS bytes3065446400 (2923 MiB)
Outputting started2025-12-06 10:41:51
Output files
Finished2025-12-06 10:49:08
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

19, 0.0686932, 0.208452, 0.625832, 
Output 2: 0.0464358, 0.132606, 0.799338, 0.0216204, 
Output 3: 0.972353, 0.0211464, 0.00236994, 0.00413107, 
Output 4: 0.294207, 0.621238, 0.0767203, 0.0078352, 
Output 5: 0.352129, 0.41486, 0.179522, 0.0534888, 
Output 6: 0.0602912, 0.132811, 0.110655, 0.696242, 

Begin processing the 10th record. run: 5027 subRun: 0 event: 20990 at 06-Dec-2025 09:54:52 UTC
Boundary wire vector sizes: 6565, 6479, 6648
minwire 0: 54
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 195, 694
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0708957, 
Output 1: 0.551664, 0.0431845, 0.162351, 0.242801, 
Output 2: 0.0344051, 0.0854411, 0.87239, 0.00776412, 
Output 3: 0.917204, 0.0741768, 0.00413703, 0.00448219, 
Output 4: 0.636261, 0.319228, 0.0355294, 0.00898139, 
Output 5: 0.406677, 0.435277, 0.118667, 0.0393796, 
Output 6: 0.166807, 0.262413, 0.129922, 0.440858, 

06-Dec-2025 10:39:37 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 10:39:37 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/cb/d9/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T001408Z_gen_002099_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2676.91       3225.09       4376.22       3090.76       426.359        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00252589     0.0147942     0.0496253    0.00952562       0.013         10     
reco:pandora:StandardPandora                            2672.74       3218.96       4365.34       3083.94       424.973        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00180304     0.0451743     0.226981      0.0201053     0.0650362       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000873086    0.0118004     0.0829119    0.00344144     0.023769        10     
reco:pandoracalo:Calorimetry                          0.000379305    0.0103497     0.0448761    0.00102281     0.0169926       10     
reco:pandorapid:Chi2ParticleID                        0.000145274   0.000712058   0.00463549    0.000248929   0.00131462       10     
reco:cvnmap:CVNMapper                                  0.0481753     0.0851673     0.281844      0.0655074     0.0660789       10     
reco:cvneva:CVNEvaluator                                3.29129       5.03818       8.86225       4.34672       1.76328        10     
reco:energyrecnumu:EnergyReco                         0.00809655     0.126109      0.166722      0.138402      0.0442655       10     
reco:energyrecnue:EnergyReco                          0.00233405    0.00870855     0.0177386    0.00501824    0.00619243       10     
reco:energyrecnc:EnergyReco                            0.0145031     0.0426253     0.148721      0.0167318     0.0524139       10     
[art]:TriggerResults:TriggerResultInserter            3.3974e-05    6.52419e-05    0.0001964     5.048e-05    4.58053e-05      10     
end_path:out1:RootOutput                              1.4928e-05    2.59738e-05    6.397e-05    1.77935e-05   1.53102e-05      10     
end_path:out1:RootOutput(write)                         0.62776      0.739571       1.13076      0.703444       0.1367         10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21787.4 MB
  Peak resident set size usage (VmHWM): 3065.45 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
Number of reconstructed particles:  2
This pfp has size:  68
Added hit list length: 68 | Max was set to: -3.40282e+38
True Neutron Energy: 939.567
True Electron Energy: 47.3184
Completed MCParticle Loop
True Neutrino Energy: 64.5742
 Size of all hits: 68
hit initial : 191.481 | hit correction : 1.21266
hit initial : 42.9276 | hit correction : 1.21688
hit initial : 17.3998 | hit correction : 1.21686
hit initial : 45.4945 | hit correction : 1.21608
hit initial : 121.922 | hit correction : 1.21607
hit initial : 171.693 | hit correction : 1.21607
hit initial : 51.3075 | hit correction : 1.21608
hit initial : 64.962 | hit correction : 1.21549
hit initial : 80.3109 | hit correction : 1.21536
hit initial : 300.485 | hit correction : 1.21528
hit initial : 687.307 | hit correction : 1.21503
hit initial : 636.302 | hit correction : 1.21463
hit initial : 445.864 | hit correction : 1.21409
hit initial : 447.871 | hit correction : 1.21355
hit initial : 363.327 | hit correction : 1.21297
hit initial : 36.9022 | hit correction : 1.21257
 *** Total Charge : 4500.38
 Looking at view: 0 | Number of hits in view: 16
hit initial : 19.2661 | hit correction : 1.21682
hit initial : 44.3625 | hit correction : 1.21685
hit initial : 41.5129 | hit correction : 1.21698
hit initial : 60.9709 | hit correction : 1.21601
hit initial : 231.306 | hit correction : 1.21605
hit initial : 336.001 | hit correction : 1.21608
hit initial : 275.2 | hit correction : 1.21602
hit initial : 221.486 | hit correction : 1.21572
hit initial : 204.817 | hit correction : 1.21545
hit initial : 185.882 | hit correction : 1.21534
hit initial : 183.152 | hit correction : 1.21535
hit initial : 215.724 | hit correction : 1.21532
hit initial : 275.546 | hit correction : 1.21521
hit initial : 268.362 | hit correction : 1.21511
hit initial : 228.766 | hit correction : 1.215
hit initial : 206.335 | hit correction : 1.21487
hit initial : 200.52 | hit correction : 1.21475
hit initial : 194.062 | hit correction : 1.2146
hit initial : 181.262 | hit correction : 1.21443
hit initial : 200.178 | hit correction : 1.21428
hit initial : 224.312 | hit correction : 1.21405
hit initial : 233.75 | hit correction : 1.21376
hit initial : 233.077 | hit correction : 1.21346
hit initial : 226.411 | hit correction : 1.21312
hit initial : 205.256 | hit correction : 1.2128
hit initial : 161.309 | hit correction : 1.21259
hit initial : 50.3716 | hit correction : 1.21253
 *** Total Charge : 6206.7
 Looking at view: 1 | Number of hits in view: 27
hit initial : 72.3397 | hit correction : 1.21683
hit initial : 247.176 | hit correction : 1.21612
hit initial : 126.544 | hit correction : 1.21614
hit initial : 215.521 | hit correction : 1.21607
hit initial : 153.152 | hit correction : 1.21599
hit initial : 166.033 | hit correction : 1.216
hit initial : 166.927 | hit correction : 1.21577
hit initial : 141.877 | hit correction : 1.21556
hit initial : 138.149 | hit correction : 1.21538
hit initial : 177.596 | hit correction : 1.21536
hit initial : 181.553 | hit correction : 1.21529
hit initial : 196.201 | hit correction : 1.21535
hit initial : 214.503 | hit correction : 1.21525
hit initial : 304.29 | hit correction : 1.21512
hit initial : 178.927 | hit correction : 1.21505
hit initial : 200.076 | hit correction : 1.21494
hit initial : 172.885 | hit correction : 1.21484
hit initial : 172.177 | hit correction : 1.21471
hit initial : 170.589 | hit correction : 1.21459
hit initial : 214.62 | hit correction : 1.21441
hit initial : 283.835 | hit correction : 1.21421
hit initial : 335.736 | hit correction : 1.21384
hit initial : 357.458 | hit correction : 1.21338
hit initial : 302.755 | hit correction : 1.2129
hit initial : 213.789 | hit correction : 1.21257
 *** Total Charge : 6201.23
 Looking at view: 2 | Number of hits in view: 25
06-Dec-2025 10:41:49 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.129262      0.311781       1.01932      0.180397      0.259473        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000554767   0.000813026   0.00185683    0.000716812   0.00035907       10     
end_path:ana:AnalyzeEvents          0.12817      0.310638       1.01648      0.179502      0.258958        10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19476.5 MB
  Peak resident set size usage (VmHWM): 1538.62 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 14:53:11 UTC       justIN version: 01.05.03