justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258531.13@dunegpschedd01.fnal.gov

Jobsub ID258531.13@dunegpschedd01.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 09:47:19
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce04
Last heartbeat2025-11-26 14:08:37
From worker nodeHostnamewn123
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 09:48:48
Input fileshd-protodune:pdhd_prod_beam__246710_107_1_20251110T235656Z_gen_g4_IonScintPDExt.root_249558_65_1_20251122T075017Z_PDInt.root
JobscriptExit code0
Real time4h (15067s)
CPU time2h (9998s = 66%)
Max RSS bytes4567113728 (4355 MiB)
Outputting started2025-11-26 13:59:56
Output files
Finished2025-11-26 14:08:37
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

gged "wiener"
FrameSaver: q=3.2012e+06 n=1198513 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 15962 XUs and 12812 XVs -> 172 XUVs
C:0 T:1 154920 XUs and 121845 XVs -> 6727 XUVs
C:0 T:2 14326 XUs and 19612 XVs -> 1909 XUVs
C:0 T:3 7 XUs and 2 XVs -> 0 XUVs
C:0 T:4 37 XUs and 25 XVs -> 1 XUVs
C:0 T:5 83411 XUs and 63380 XVs -> 1629 XUVs
C:0 T:6 11809 XUs and 15327 XVs -> 382 XUVs
C:0 T:7 12 XUs and 8 XVs -> 0 XUVs
10820 XUVs total
3166 collection wire objects
10820 potential space points
Neighbour search...
734984 tests to find 285046 neighbours
Iterating with no regularization...
Begin: 4.32842e+07
0 3.7858e+07
1 3.73879e+07
2 3.73346e+07
3 3.73217e+07
Now with regularization...
Begin: 3.54841e+07
0 3.54738e+07
26-Nov-2025 10:59:44 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_13_1_20251126T094854Z_detsim_reco1.root"
26-Nov-2025 10:59:44 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            191.721       261.955       349.078       261.242       52.1494        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                               0.0029968     0.0114123     0.0188442     0.011464     0.00554775       10     
reco:opdec:Deconvolution                              1.19978       1.48419       1.77411       1.53852      0.208282        10     
reco:ophitspe:OpHitFinderDeco                        0.219578      0.366205      0.454949      0.368765      0.0614843       10     
reco:opflash:OpFlashFinder                           0.0936708     0.162312      0.269462      0.164497      0.0525064       10     
reco:opslicer:OpSlicer                                0.25597      0.469893      0.589074      0.492089      0.0883833       10     
reco:wclsdatahd:WireCellToolkit                       76.107        78.7774       82.7512       78.6026       1.93545        10     
reco:gaushit:GausHitFinder                            1.14701       1.64117       1.90528       1.66623      0.224263        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.00061303    0.00124452    0.00257266    0.00124715    0.000550246      10     
reco:reco3d:SpacePointSolver                          1.11043       2.33151       3.69627       2.26272      0.877345        10     
reco:hitpdune:DisambigFromSpacePoints                 0.74211       1.62811       2.38417       1.79073      0.564064        10     
reco:pandora:StandardPandora                          27.5045       53.9586       112.696       43.0325       28.0706        10     
reco:pandoraWriter:StandardPandora                   0.628415      0.867675       1.47259       0.7343       0.276089        10     
reco:pandoraTrack:LArPandoraTrackCreation             2.07516       4.62613       7.79995       4.84259       1.5631         10     
reco:pandoraShower:LArPandoraShowerCreation           1.94176       4.63575       6.90763       4.75411       1.81756        10     
reco:pandoracalo:Calorimetry                          1.03008       2.27762       2.98225       2.50118      0.674734        10     
reco:pandoracalonosce:Calorimetry                     1.11811       2.25958       2.82966       2.52323      0.616665        10     
reco:pandoraShowercalo:ShowerCalorimetry              2.03135       4.29013       7.21815       4.51772       1.56625        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         2.06166       3.80881       6.73028       3.93597       1.37663        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0854418     0.124112       0.16557      0.119447      0.0238306       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0622962      0.12326      0.165388      0.124502      0.0304408       10     
reco:emtrkmichelid:EmTrackMichelId                    43.9813       58.6612       72.1082       61.127        8.70145        10     
reco:crttag:SingleCRTMatchingProducer               0.00754684     0.0145702     0.0200501     0.0148351    0.00321262       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00585839     0.0120271     0.0193031     0.0116212    0.00456008       10     
reco:pandora2Track:LArPandoraTrackCreation            5.2028        12.0202       19.3917       11.5531       4.97087        10     
reco:pandora2calo:ShowerCalorimetry                   1.49546       3.5966        6.94786       3.54989       1.57846        10     
reco:pandora2calonosce:ShowerCalorimetry              1.52803       3.54892       6.83948       3.67759       1.51628        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0606851     0.110074      0.153255      0.114992      0.029507        10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0628412     0.112003      0.151651      0.115015      0.0284757       10     
reco:pandora2Shower:LArPandoraShowerCreation          2.12194       4.86782       7.61027       4.98936       2.08753        10     
reco:pandora2Showercalo:ShowerCalorimetry             2.24428       4.80744       9.04619       4.91342       2.18859        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.88084       4.64386       8.54765       4.93388       2.03904        10     
[art]:TriggerResults:TriggerResultInserter          3.4192e-05    5.61468e-05   0.000157097   4.7012e-05    3.43968e-05      10     
end_path:out1:RootOutput                             6.81e-06     1.08514e-05   3.6482e-05     7.71e-06     8.58828e-06      10     
end_path:out1:RootOutput(write)                       4.21629       5.45669       6.99905       5.56729      0.798826        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6496.14 MB
  Peak resident set size usage (VmHWM): 4278.15 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_13_1_20251126T094854Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_13_1_20251126T094854Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_13_1_20251126T094854Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_13_1_20251126T094854Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_13_1_20251126T094854Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_13_1_20251126T094854Z_detsim_reco1.root.temp.json ====
.:
total 6251420
-rw-r--r-- 1 nobody nobody      42772 Nov 26 10:59 jobscript.log
-rw-r--r-- 1 nobody nobody        204 Nov 26 10:59 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 10:59 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_13_1_20251126T094854Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 10:59 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_13_1_20251126T094854Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 10:59 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_13_1_20251126T094854Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 10:59 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 10:59 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 10:59 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     204800 Nov 26 10:59 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 10:59 time.db
-rw-r--r-- 1 nobody nobody 6346104306 Nov 26 10:59 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_13_1_20251126T094854Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   55033350 Nov 26 10:57 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 10:17 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 10:12 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 10:12 detsim_single_protodunehd_hist.root
justIN time: 2026-02-04 07:21:22 UTC       justIN version: 01.06.00