justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258670.10@dunegpschedd01.fnal.gov

Jobsub ID258670.10@dunegpschedd01.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 11:32:24
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce04
Last heartbeat2025-11-26 17:40:09
From worker nodeHostnamewn117
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 11:33:53
Input fileshd-protodune:pdhd_prod_beam__238205_33_1_20251106T043404Z_gen_g4_IonScintPDExt.root_256311_67_1_20251124T190033Z_PDInt.root
JobscriptExit code0
Real time6h (21664s)
CPU time4h (14963s = 69%)
Max RSS bytes4179812352 (3986 MiB)
Outputting started2025-11-26 17:35:01
Output files
Finished2025-11-26 17:40:09
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 23511 XUs and 18688 XVs -> 315 XUVs
C:0 T:1 219061 XUs and 170392 XVs -> 7101 XUVs
C:0 T:2 33673 XUs and 38919 XVs -> 2189 XUVs
C:0 T:3 14 XUs and 5 XVs -> 0 XUVs
C:0 T:4 41 XUs and 62 XVs -> 0 XUVs
C:0 T:5 89834 XUs and 82393 XVs -> 2748 XUVs
C:0 T:6 16228 XUs and 6938 XVs -> 315 XUVs
C:0 T:7 45 XUs and 19 XVs -> 0 XUVs
12668 XUVs total
5006 collection wire objects
12668 potential space points
Neighbour search...
881234 tests to find 333684 neighbours
Iterating with no regularization...
Begin: 6.79179e+07
0 6.10678e+07
1 6.05994e+07
2 6.0567e+07
Now with regularization...
Begin: 5.83494e+07
0 5.83385e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 14:34:30 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_10_1_20251126T113411Z_detsim_reco1.root"
26-Nov-2025 14:34:30 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            337.274       614.116       1367.81       486.71        303.938        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00294863     0.0526264     0.272174      0.0180059     0.0799875       10     
reco:opdec:Deconvolution                              1.27062       1.9474        2.34899       1.93791      0.322764        10     
reco:ophitspe:OpHitFinderDeco                        0.417733       1.27569       2.12907       1.26454      0.568447        10     
reco:opflash:OpFlashFinder                           0.0957396     0.143805      0.215157      0.140822      0.0346762       10     
reco:opslicer:OpSlicer                               0.285178      0.417096      0.719633      0.374128      0.118107        10     
reco:wclsdatahd:WireCellToolkit                       126.46        156.708       185.939       159.97        17.9885        10     
reco:gaushit:GausHitFinder                           0.945376       1.7105        2.7227        1.66576      0.552612        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.00219313    0.00463985    0.00809832     0.0040564    0.00191624       10     
reco:reco3d:SpacePointSolver                          1.76612       4.17496       12.9204       2.49934       3.69216        10     
reco:hitpdune:DisambigFromSpacePoints                0.794946       2.76366       9.61518       1.33521        2.707         10     
reco:pandora:StandardPandora                          25.6387       128.004       588.642       59.7043       163.432        10     
reco:pandoraWriter:StandardPandora                    0.68164        1.064        2.06033       0.85737      0.422789        10     
reco:pandoraTrack:LArPandoraTrackCreation             4.64115       17.0784       49.1066       13.9707       12.6914        10     
reco:pandoraShower:LArPandoraShowerCreation           2.81108       13.7669       34.8946       11.0329       9.41755        10     
reco:pandoracalo:Calorimetry                          1.98485        6.374        16.1142       5.39544       4.31016        10     
reco:pandoracalonosce:Calorimetry                     1.9171        6.07135       13.9319       4.8136        3.75977        10     
reco:pandoraShowercalo:ShowerCalorimetry              2.82142       9.98008       20.028        6.79044       6.25545        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         2.48194       9.99448       18.3498       8.13635       6.02108        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0786831     0.171955      0.468823      0.148766      0.106452        10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0747145      0.16283       0.41187      0.137625      0.0914376       10     
reco:emtrkmichelid:EmTrackMichelId                    90.5364       140.549       202.374       133.634       32.8749        10     
reco:crttag:SingleCRTMatchingProducer                0.0169957     0.0636668     0.248091      0.0541253     0.063082        10     
reco:crtreco:TwoCRTMatchingProducer                  0.0125966     0.0449245     0.0634841     0.0518516     0.0172319       10     
reco:pandora2Track:LArPandoraTrackCreation            16.4352       39.5637       81.9254       33.2032       19.9588        10     
reco:pandora2calo:ShowerCalorimetry                   2.64272       8.44438       17.5274       6.39278       5.42395        10     
reco:pandora2calonosce:ShowerCalorimetry              2.21001       8.44985       20.1108       7.73969       5.30939        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0523357     0.122452      0.176592       0.11728      0.0377848       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0509172     0.127951      0.200508      0.119606      0.0400711       10     
reco:pandora2Shower:LArPandoraShowerCreation          4.96314       14.8098       45.2077       13.2167       11.505         10     
reco:pandora2Showercalo:ShowerCalorimetry             4.94752       19.4629       73.2222       8.87116       20.5801        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        3.55698       14.9054       47.3397       8.37119       14.1706        10     
[art]:TriggerResults:TriggerResultInserter          5.1331e-05    0.00026453    0.00166495    8.7673e-05    0.000472565      10     
end_path:out1:RootOutput                             1.087e-05    5.32089e-05   0.00029899    1.9811e-05    8.39558e-05      10     
end_path:out1:RootOutput(write)                       3.28546        5.163        8.44583       4.22742       1.84167        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6250.21 MB
  Peak resident set size usage (VmHWM): 3942.18 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_10_1_20251126T113411Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_10_1_20251126T113411Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_10_1_20251126T113411Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_10_1_20251126T113411Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_10_1_20251126T113411Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_10_1_20251126T113411Z_detsim_reco1.root.temp.json ====
.:
total 6045584
-rw-r--r-- 1 nobody nobody      40753 Nov 26 14:34 jobscript.log
-rw-r--r-- 1 nobody nobody        203 Nov 26 14:34 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 14:34 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_10_1_20251126T113411Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 14:34 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_10_1_20251126T113411Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 14:34 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_10_1_20251126T113411Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 14:34 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 14:34 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 14:34 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     204800 Nov 26 14:34 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 14:34 time.db
-rw-r--r-- 1 nobody nobody 6135780610 Nov 26 14:34 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_10_1_20251126T113411Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   54584770 Nov 26 14:25 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 12:55 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 12:45 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 12:44 detsim_single_protodunehd_hist.root
justIN time: 2026-02-04 07:21:13 UTC       justIN version: 01.06.00