justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 265279.96@dunegpschedd01.fnal.gov

Jobsub ID265279.96@dunegpschedd01.fnal.gov
Workflow ID11116
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-09 10:51:45
SiteUS_FNAL-T1
EntryCMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat2025-12-10 09:10:18
From worker nodeHostnamedunegli-48793-0-cmswn4069.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-09 16:59:23
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250116T135912Z_gen_000710_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time16h (57852s)
CPU time15h (56362s = 97%)
Max RSS bytes2653720576 (2530 MiB)
Outputting started2025-12-10 09:03:35
Output files
Finished2025-12-10 09:10:18
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

.745323, 0.0365299, 
Output 3: 0.974905, 0.00820928, 0.00275688, 0.0141283, 
Output 4: 0.860575, 0.121068, 0.0150198, 0.00333729, 
Output 5: 0.432544, 0.252192, 0.216514, 0.09875, 
Output 6: 0.024243, 0.0514076, 0.0627179, 0.861631, 

Begin processing the 10th record. run: 4696 subRun: 0 event: 7100 at 10-Dec-2025 08:29:59 UTC
Boundary wire vector sizes: 6875, 6781, 6668
minwire 0: 31
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1812, 2311
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.107668, 
Output 1: 0.177763, 0.104798, 0.226214, 0.491224, 
Output 2: 0.037686, 0.129183, 0.818439, 0.0146924, 
Output 3: 0.928673, 0.0600881, 0.00550054, 0.00573808, 
Output 4: 0.492054, 0.434845, 0.0630194, 0.0100815, 
Output 5: 0.342149, 0.440026, 0.172396, 0.0454288, 
Output 6: 0.0837185, 0.151016, 0.11541, 0.649856, 

10-Dec-2025 09:03:06 UTC  Closed output file "matt_reco2.root"
10-Dec-2025 09:03:06 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/c3/8a/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250116T135912Z_gen_000710_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1982.93       5705.75       8260.38       6077.75       1684.76        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00704294     0.0273252     0.0580948     0.0282795     0.0138248       10     
reco:pandora:StandardPandora                            1981.69       5633.41       8087.11       6009.26       1653.48        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00107725     0.113871      0.578569      0.0461187     0.167866        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000530899    0.0166546     0.0918131    0.00539865     0.0268132       10     
reco:pandoracalo:Calorimetry                          0.000360315    0.0225985     0.130771     0.00220464     0.0384975       10     
reco:pandorapid:Chi2ParticleID                        0.000109949   0.00124859    0.00910909    0.000307409   0.00263586       10     
reco:cvnmap:CVNMapper                                  0.0304642     0.176442       0.63681      0.130402      0.158983        10     
reco:cvneva:CVNEvaluator                               0.753397       11.7083       103.61        1.31015       30.6387        10     
reco:energyrecnumu:EnergyReco                         0.00687203      32.8977       71.3442       30.7252       32.7481        10     
reco:energyrecnue:EnergyReco                          0.00083696     0.0124869     0.031492     0.00775598     0.0110857       10     
reco:energyrecnc:EnergyReco                           0.00741437      26.5881       70.3651      0.077446       32.6165        10     
[art]:TriggerResults:TriggerResultInserter            2.2783e-05    0.000139835   0.000448041   0.000122522   0.000118837      10     
end_path:out1:RootOutput                               9.508e-06    5.33206e-05   0.000132051   5.10975e-05   4.12558e-05      10     
end_path:out1:RootOutput(write)                        0.238261      0.778385       1.54487      0.796607      0.352209        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21687.1 MB
  Peak resident set size usage (VmHWM): 2653.72 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
Begin processing the 9th record. run: 4696 subRun: 0 event: 7099 at 10-Dec-2025 09:03:31 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 3.39348
Completed MCParticle Loop
True Neutrino Energy: 6.67748
 Size of all hits: 0
Begin processing the 10th record. run: 4696 subRun: 0 event: 7100 at 10-Dec-2025 09:03:33 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  59
Added hit list length: 59 | Max was set to: -3.40282e+38
True Neutron Energy: 939.567
True Electron Energy: 52.0239
Completed MCParticle Loop
True Neutrino Energy: 64.7403
 Size of all hits: 59
hit initial : 53.597 | hit correction : 1.08974
hit initial : 18.1431 | hit correction : 1.0888
hit initial : 26.9795 | hit correction : 1.08719
hit initial : 33.8306 | hit correction : 1.08463
hit initial : 36.957 | hit correction : 1.09106
hit initial : 99.0228 | hit correction : 1.08719
hit initial : 253.109 | hit correction : 1.09108
hit initial : 37.8495 | hit correction : 1.08477
hit initial : 163.81 | hit correction : 1.08175
hit initial : 100.512 | hit correction : 1.08083
hit initial : 199.002 | hit correction : 1.0911
hit initial : 50.1136 | hit correction : 1.09108
hit initial : 60.703 | hit correction : 1.09002
hit initial : 61.6952 | hit correction : 1.08081
hit initial : 157.72 | hit correction : 1.08136
hit initial : 164.801 | hit correction : 1.08237
hit initial : 397.256 | hit correction : 1.08318
hit initial : 19.5047 | hit correction : 1.08376
hit initial : 146.102 | hit correction : 1.08475
hit initial : 75.7245 | hit correction : 1.08647
hit initial : 108.676 | hit correction : 1.08718
hit initial : 20.0464 | hit correction : 1.08808
hit initial : 50.8251 | hit correction : 1.08873
hit initial : 40.5851 | hit correction : 1.08955
hit initial : 77.8935 | hit correction : 1.08965
hit initial : 164.941 | hit correction : 1.09032
hit initial : 255.863 | hit correction : 1.09103
 *** Total Charge : 3124.39
 Looking at view: 0 | Number of hits in view: 27
hit initial : 32.4259 | hit correction : 1.09202
hit initial : 98.6491 | hit correction : 1.0911
hit initial : 308.815 | hit correction : 1.09109
hit initial : 90.1755 | hit correction : 1.08716
hit initial : 41.739 | hit correction : 1.08647
hit initial : 112.488 | hit correction : 1.08718
hit initial : 11.8459 | hit correction : 1.08647
hit initial : 40.2937 | hit correction : 1.08718
hit initial : 85.4544 | hit correction : 1.0808
hit initial : 28.641 | hit correction : 1.0808
hit initial : 47.3872 | hit correction : 1.08132
hit initial : 51.502 | hit correction : 1.08133
hit initial : 372.248 | hit correction : 1.08284
hit initial : 31.9128 | hit correction : 1.08383
hit initial : 179.72 | hit correction : 1.08471
hit initial : 124.285 | hit correction : 1.08472
hit initial : 44.1226 | hit correction : 1.08487
hit initial : 539.728 | hit correction : 1.08872
hit initial : 23.1635 | hit correction : 1.08947
hit initial : 322.37 | hit correction : 1.08955
hit initial : 21.4806 | hit correction : 1.08983
hit initial : 44.1129 | hit correction : 1.0903
 *** Total Charge : 2883.49
 Looking at view: 1 | Number of hits in view: 22
hit initial : 75.8789 | hit correction : 1.08072
hit initial : 92.3261 | hit correction : 1.08081
hit initial : 376.326 | hit correction : 1.08133
hit initial : 647.493 | hit correction : 1.08264
hit initial : 184.706 | hit correction : 1.08375
hit initial : 83.3042 | hit correction : 1.08481
hit initial : 2233.28 | hit correction : 1.08644
hit initial : 933.083 | hit correction : 1.08924
hit initial : 177.483 | hit correction : 1.09107
hit initial : 129.48 | hit correction : 1.09107
 *** Total Charge : 5357.87
 Looking at view: 2 | Number of hits in view: 10
10-Dec-2025 09:03:35 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0479426     0.0605623     0.0881427     0.0589694     0.0122653       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000401453   0.000554978   0.000886574   0.000477317   0.00016999       10     
end_path:ana:AnalyzeEvents         0.0474067     0.0598315     0.086763      0.0583945     0.0121376       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19476.4 MB
  Peak resident set size usage (VmHWM): 1535.41 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 23:17:39 UTC       justIN version: 01.05.03