justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 265279.58@dunegpschedd01.fnal.gov

Jobsub ID265279.58@dunegpschedd01.fnal.gov
Workflow ID11116
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-09 10:51:45
SiteUS_FNAL-T1
EntryCMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat2025-12-10 09:09:41
From worker nodeHostnamedunegli-48793-0-cmswn4069.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-09 16:42:42
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250116T134923Z_gen_000598_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time16h (58815s)
CPU time16h (57983s = 98%)
Max RSS bytes2597101568 (2476 MiB)
Outputting started2025-12-10 09:02:57
Output files
Finished2025-12-10 09:09:41
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

2: 0.0467693, 0.15937, 0.763514, 0.030347, 
Output 3: 0.97466, 0.0132846, 0.00281301, 0.00924199, 
Output 4: 0.379414, 0.561697, 0.0538359, 0.00505239, 
Output 5: 0.406406, 0.394924, 0.156349, 0.0423205, 
Output 6: 0.0176679, 0.0485066, 0.0619258, 0.8719, 

Begin processing the 10th record. run: 4696 subRun: 0 event: 5980 at 10-Dec-2025 08:26:07 UTC
Boundary wire vector sizes: 6684, 6609, 6687
minwire 0: 34
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1625, 2124
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0841911, 
Output 1: 0.0482987, 0.0698628, 0.0664976, 0.815341, 
Output 2: 0.117054, 0.250211, 0.580934, 0.0518008, 
Output 3: 0.974602, 0.00769099, 0.00125585, 0.0164512, 
Output 4: 0.579472, 0.398752, 0.0196912, 0.00208468, 
Output 5: 0.774227, 0.183085, 0.0286539, 0.0140343, 
Output 6: 0.0122164, 0.0302736, 0.0461994, 0.911311, 

10-Dec-2025 09:02:24 UTC  Closed output file "matt_reco2.root"
10-Dec-2025 09:02:24 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/9e/5e/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250116T134923Z_gen_000598_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2172.85       5831.14       8816.63       6214.47       1750.41        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0121818     0.0233572     0.0409431     0.0228346    0.00805958       10     
reco:pandora:StandardPandora                            2171.68       5825.52        8789         6211.55       1745.54        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00109127     0.104669      0.616476      0.0149011      0.18034        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00225658     0.0155496     0.0786548    0.00576972     0.0231416       10     
reco:pandoracalo:Calorimetry                          0.000420619    0.026326      0.117959     0.00731612     0.0412871       10     
reco:pandorapid:Chi2ParticleID                        0.000106021   0.00123265    0.00516276    0.00023706    0.00191442       10     
reco:cvnmap:CVNMapper                                  0.0604468     0.226645       0.53971      0.179696      0.153064        10     
reco:cvneva:CVNEvaluator                               0.729135       4.15336       24.2951       2.14983       6.74169        10     
reco:energyrecnumu:EnergyReco                         0.00577964     0.131415       0.45566      0.0774273     0.138394        10     
reco:energyrecnue:EnergyReco                          0.000854463    0.0126773     0.0311437    0.00697305     0.0100637       10     
reco:energyrecnc:EnergyReco                           0.00934998     0.0983557     0.227351      0.0586832     0.0854268       10     
[art]:TriggerResults:TriggerResultInserter            2.7482e-05    0.000173837   0.000628845   0.000142446   0.000160791      10     
end_path:out1:RootOutput                              1.1903e-05    6.97174e-05   0.000289099   4.3703e-05    7.67757e-05      10     
end_path:out1:RootOutput(write)                        0.239343      0.812382       1.43607      0.772769      0.357873        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21679.2 MB
  Peak resident set size usage (VmHWM): 2597.1 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 235.743 | hit correction : 1.10224
hit initial : 256.469 | hit correction : 1.1023
hit initial : 282.142 | hit correction : 1.10231
hit initial : 281.654 | hit correction : 1.10229
hit initial : 241.702 | hit correction : 1.10221
hit initial : 341.929 | hit correction : 1.10209
hit initial : 267.161 | hit correction : 1.102
hit initial : 319.108 | hit correction : 1.10173
hit initial : 254.859 | hit correction : 1.10163
hit initial : 469.703 | hit correction : 1.10206
hit initial : 283.404 | hit correction : 1.102
 *** Total Charge : 5142.05
 Looking at view: 1 | Number of hits in view: 22
Begin processing the 10th record. run: 4696 subRun: 0 event: 5980 at 10-Dec-2025 09:02:55 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  52
Added hit list length: 52 | Max was set to: -3.40282e+38
True Neutron Energy: 939.57
True Neutron Energy: 940.354
True Electron Energy: 33.7933
Completed MCParticle Loop
True Neutrino Energy: 46.3976
 Size of all hits: 52
hit initial : 34.0007 | hit correction : 1.44066
hit initial : 21.3637 | hit correction : 1.43949
hit initial : 197.2 | hit correction : 1.44289
hit initial : 43.5745 | hit correction : 1.44036
hit initial : 40.4469 | hit correction : 1.43935
hit initial : 80.0577 | hit correction : 1.43271
hit initial : 209.936 | hit correction : 1.44296
hit initial : 109.458 | hit correction : 1.43272
hit initial : 130.17 | hit correction : 1.44293
hit initial : 38.4084 | hit correction : 1.43271
hit initial : 128.667 | hit correction : 1.44186
hit initial : 575.418 | hit correction : 1.44038
hit initial : 636.328 | hit correction : 1.4382
hit initial : 66.0982 | hit correction : 1.4379
hit initial : 20.712 | hit correction : 1.43707
hit initial : 243.602 | hit correction : 1.43613
hit initial : 117.022 | hit correction : 1.43588
 *** Total Charge : 3874.75
 Looking at view: 0 | Number of hits in view: 17
hit initial : 26.3055 | hit correction : 1.43271
hit initial : 42.482 | hit correction : 1.43578
hit initial : 96.3074 | hit correction : 1.43271
hit initial : 87.8177 | hit correction : 1.43271
hit initial : 97.5568 | hit correction : 1.44292
hit initial : 43.7816 | hit correction : 1.43932
hit initial : 222.42 | hit correction : 1.44294
hit initial : 58.3492 | hit correction : 1.43941
hit initial : 209.173 | hit correction : 1.44292
hit initial : 50.5833 | hit correction : 1.44294
hit initial : 46.8045 | hit correction : 1.44246
hit initial : 8.58109 | hit correction : 1.44191
hit initial : 253.611 | hit correction : 1.44115
hit initial : 190.755 | hit correction : 1.43973
hit initial : 418.945 | hit correction : 1.43841
hit initial : 162.919 | hit correction : 1.4381
hit initial : 264.884 | hit correction : 1.43645
hit initial : 225.116 | hit correction : 1.43614
 *** Total Charge : 3606.71
 Looking at view: 1 | Number of hits in view: 18
hit initial : 21.0243 | hit correction : 1.43282
hit initial : 97.5628 | hit correction : 1.43272
hit initial : 104.335 | hit correction : 1.4327
hit initial : 80.0942 | hit correction : 1.44298
hit initial : 378.356 | hit correction : 1.44293
hit initial : 18.6687 | hit correction : 1.4428
hit initial : 512.852 | hit correction : 1.44249
hit initial : 45.6524 | hit correction : 1.44202
hit initial : 252.454 | hit correction : 1.4405
hit initial : 211.1 | hit correction : 1.43962
hit initial : 239.275 | hit correction : 1.43918
hit initial : 226.177 | hit correction : 1.43835
hit initial : 210.903 | hit correction : 1.4379
hit initial : 175.321 | hit correction : 1.43714
hit initial : 163.302 | hit correction : 1.43645
hit initial : 158.532 | hit correction : 1.43595
hit initial : 93.3261 | hit correction : 1.43555
 *** Total Charge : 4302.11
 Looking at view: 2 | Number of hits in view: 17
10-Dec-2025 09:02:57 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0531585     0.0653055     0.0742272     0.0673894    0.00747486       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000432061   0.000689754   0.00147525    0.000575648   0.000298049      10     
end_path:ana:AnalyzeEvents         0.0524352     0.0644086     0.0735178     0.0666582    0.00726649       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19472.5 MB
  Peak resident set size usage (VmHWM): 1524.47 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 23:18:00 UTC       justIN version: 01.05.03