justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258499.3@dunegpschedd01.fnal.gov

Jobsub ID258499.3@dunegpschedd01.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 08:53:16
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce04
Last heartbeat2025-11-26 12:20:20
From worker nodeHostnamewn121.cat.cbpf.br
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 08:55:01
Input fileshd-protodune:pdhd_prod_beam__238186_67_1_20251106T040607Z_gen_g4_IonScintPDExt.root_249735_50_1_20251122T174335Z_PDInt.root
JobscriptExit code0
Real time3h (12055s)
CPU time2h (9158s = 75%)
Max RSS bytes4267925504 (4070 MiB)
Outputting started2025-11-26 12:15:58
Output files
Finished2025-11-26 12:20:20
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

3293 tag=gauss
wclsFrameSaver: saving 85314 traces tagged "wiener"
FrameSaver: q=2.63595e+06 n=1079302 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 9388 XUs and 8224 XVs -> 70 XUVs
C:0 T:1 62216 XUs and 41827 XVs -> 1580 XUVs
C:0 T:2 11793 XUs and 8033 XVs -> 366 XUVs
C:0 T:3 5 XUs and 10 XVs -> 0 XUVs
C:0 T:4 32 XUs and 19 XVs -> 0 XUVs
C:0 T:5 59138 XUs and 36643 XVs -> 1830 XUVs
C:0 T:6 9780 XUs and 12367 XVs -> 1708 XUVs
5554 XUVs total
2715 collection wire objects
5554 potential space points
Neighbour search...
230832 tests to find 110278 neighbours
Iterating with no regularization...
Begin: 3.90149e+07
0 3.35571e+07
1 3.31901e+07
2 3.31364e+07
3 3.31193e+07
Now with regularization...
Begin: 3.11895e+07
0 3.11813e+07
26-Nov-2025 09:15:45 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258499_3_1_20251126T085507Z_detsim_reco1.root"
26-Nov-2025 09:15:45 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            197.916       287.712       457.842       277.439       74.0955        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00183439     0.0121097     0.0224934     0.0127481    0.00601316       10     
reco:opdec:Deconvolution                              1.05926       1.34215       1.62961       1.30769      0.159595        10     
reco:ophitspe:OpHitFinderDeco                        0.299314      0.411171      0.749948      0.359798      0.127423        10     
reco:opflash:OpFlashFinder                           0.0704316     0.131233      0.208428      0.134987      0.0383138       10     
reco:opslicer:OpSlicer                               0.220028      0.385355      0.488353      0.379457      0.0779152       10     
reco:wclsdatahd:WireCellToolkit                       76.6608       103.797       140.236       102.027       20.5379        10     
reco:gaushit:GausHitFinder                           0.996004       1.3831        1.8901        1.42068      0.262026        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000970398   0.00239942     0.0043636    0.00199529    0.00102148       10     
reco:reco3d:SpacePointSolver                         0.922032       2.03834       5.35333       1.57175       1.25876        10     
reco:hitpdune:DisambigFromSpacePoints                0.465561       1.11735       2.44367      0.910712      0.598433        10     
reco:pandora:StandardPandora                          21.5471       38.8599       78.2753       36.9438       15.6853        10     
reco:pandoraWriter:StandardPandora                    0.45409      0.712643      0.933352      0.737804      0.143466        10     
reco:pandoraTrack:LArPandoraTrackCreation             2.6473        5.87149       12.6523       5.34686       2.82112        10     
reco:pandoraShower:LArPandoraShowerCreation           2.0411        6.02915        18.98        5.16678       4.62467        10     
reco:pandoracalo:Calorimetry                          1.22425       2.8557        6.90527       2.53618       1.55956        10     
reco:pandoracalonosce:Calorimetry                     1.21882       2.6521        6.48323       2.46959       1.41422        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.95187       5.51509       18.1655       4.07781       4.46734        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.69553       5.23332       18.0186       4.01983       4.4568         10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0566515     0.109963      0.151891      0.117445      0.0252257       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0564624     0.109527       0.16008       0.11255      0.0262637       10     
reco:emtrkmichelid:EmTrackMichelId                    45.3532       64.812        85.6919       62.0585       12.6616        10     
reco:crttag:SingleCRTMatchingProducer                0.0105941     0.0210977     0.0393206     0.0202152    0.00903129       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00841296     0.0163164     0.0246264     0.0164975    0.00520114       10     
reco:pandora2Track:LArPandoraTrackCreation            5.38024       14.6475       34.2124       14.3882       8.12938        10     
reco:pandora2calo:ShowerCalorimetry                   1.61851       4.63329       13.6345       3.5325        3.39019        10     
reco:pandora2calonosce:ShowerCalorimetry              1.5751        4.30489       10.0814       3.6247        2.52108        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0683022     0.123505      0.270313      0.105348      0.0523616       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0692787     0.122203      0.266538      0.106757      0.0522482       10     
reco:pandora2Shower:LArPandoraShowerCreation          2.19917       5.63907       12.0329       4.67577       3.17264        10     
reco:pandora2Showercalo:ShowerCalorimetry             2.13817       5.66363       12.5061       4.72646       3.22927        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        2.01745       4.98127       10.763        3.91004       2.80106        10     
[art]:TriggerResults:TriggerResultInserter          3.6291e-05    7.3425e-05    0.000190988   6.03125e-05   4.44889e-05      10     
end_path:out1:RootOutput                             7.02e-06     1.43147e-05   5.3782e-05    1.05905e-05   1.32546e-05      10     
end_path:out1:RootOutput(write)                       2.65324       3.83818       4.48847       3.88874      0.568069        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6195.42 MB
  Peak resident set size usage (VmHWM): 3982.1 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258499_3_1_20251126T085507Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258499_3_1_20251126T085507Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258499_3_1_20251126T085507Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258499_3_1_20251126T085507Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258499_3_1_20251126T085507Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258499_3_1_20251126T085507Z_detsim_reco1.root.temp.json ====
.:
total 5849828
-rw-r--r-- 1 nobody nobody      40773 Nov 26 09:15 jobscript.log
-rw-r--r-- 1 nobody nobody        203 Nov 26 09:15 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 09:15 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258499_3_1_20251126T085507Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 09:15 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258499_3_1_20251126T085507Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 09:15 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258499_3_1_20251126T085507Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 09:15 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 09:15 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 09:15 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     204800 Nov 26 09:15 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 09:15 time.db
-rw-r--r-- 1 nobody nobody 5942087913 Nov 26 09:15 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258499_3_1_20251126T085507Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   47820870 Nov 26 09:13 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 08:28 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 08:24 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 08:24 detsim_single_protodunehd_hist.root
justIN time: 2026-02-04 07:23:10 UTC       justIN version: 01.06.00