justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 253013.3@dunegpschedd02.fnal.gov

Jobsub ID253013.3@dunegpschedd02.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 11:56:25
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce03
Last heartbeat2025-11-26 15:11:20
From worker nodeHostnamewn75
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 11:57:36
Input fileshd-protodune:pdhd_prod_beam__238162_126_1_20251106T025050Z_gen_g4_IonScintPDExt.root_255575_52_1_20251123T013533Z_PDInt.root
JobscriptExit code0
Real time3h (11338s)
CPU time2h (9506s = 83%)
Max RSS bytes4428697600 (4223 MiB)
Outputting started2025-11-26 15:06:35
Output files
Finished2025-11-26 15:11:20
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

FrameSaver: saving 90205 traces tagged "wiener"
FrameSaver: q=2.74732e+06 n=1051942 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 13742 XUs and 9692 XVs -> 109 XUVs
C:0 T:1 67546 XUs and 65620 XVs -> 1612 XUVs
C:0 T:2 23738 XUs and 15466 XVs -> 595 XUVs
C:0 T:3 22 XUs and 13 XVs -> 0 XUVs
C:0 T:4 21 XUs and 6 XVs -> 0 XUVs
C:0 T:5 39613 XUs and 31990 XVs -> 1583 XUVs
C:0 T:6 12625 XUs and 7295 XVs -> 71 XUVs
C:0 T:7 4 XUs and 6 XVs -> 0 XUVs
3970 XUVs total
2303 collection wire objects
3970 potential space points
Neighbour search...
86552 tests to find 40250 neighbours
Iterating with no regularization...
Begin: 3.38876e+07
0 3.19669e+07
1 3.1859e+07
2 3.18507e+07
Now with regularization...
Begin: 3.00545e+07
0 3.00501e+07
26-Nov-2025 12:06:26 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_3_1_20251126T115742Z_detsim_reco1.root"
26-Nov-2025 12:06:26 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            165.263       200.55        303.039       188.275       39.5588        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00138005    0.00840204     0.014338      0.0090177    0.00418066       10     
reco:opdec:Deconvolution                              1.10498       1.26844       1.38396       1.28519      0.0904002       10     
reco:ophitspe:OpHitFinderDeco                        0.220734      0.318147      0.356061      0.328605      0.034869        10     
reco:opflash:OpFlashFinder                           0.0870711     0.124163      0.157673      0.133337      0.0234912       10     
reco:opslicer:OpSlicer                               0.313972      0.387251      0.460766       0.37875      0.0498857       10     
reco:wclsdatahd:WireCellToolkit                       66.9758       69.7252       72.8798       69.4762       1.63141        10     
reco:gaushit:GausHitFinder                            1.03946       1.3814        1.86617       1.34299      0.228216        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000335063   0.000448922   0.000714837   0.000424356   0.000105878      10     
reco:reco3d:SpacePointSolver                         0.879061       1.54221       3.27208       1.17204      0.722844        10     
reco:hitpdune:DisambigFromSpacePoints                0.736602       1.13455       1.93626       1.08769      0.339933        10     
reco:pandora:StandardPandora                          20.2608       35.0764       60.342        32.3462       11.4004        10     
reco:pandoraWriter:StandardPandora                   0.385373      0.608931       1.12522       0.55854      0.193142        10     
reco:pandoraTrack:LArPandoraTrackCreation             2.20535       3.19375       6.12084       2.55115       1.3983         10     
reco:pandoraShower:LArPandoraShowerCreation            1.577        2.59282       6.13764       2.07115       1.42341        10     
reco:pandoracalo:Calorimetry                          1.01782       1.63782       3.26801       1.35878      0.666379        10     
reco:pandoracalonosce:Calorimetry                     1.01799       1.59842       3.31517       1.38213      0.630264        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.50701       2.56984       7.15939       2.00219       1.59227        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.4986        2.67042       8.41947       2.0376        1.97601        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0806073     0.103506      0.159036      0.0986276     0.0200124       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.085463      0.102217      0.161048      0.0949613     0.0206952       10     
reco:emtrkmichelid:EmTrackMichelId                    40.779        47.9385       64.2145       46.1816       6.85046        10     
reco:crttag:SingleCRTMatchingProducer               0.00607806    0.00864404     0.0124779    0.00837977    0.00186066       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00562336    0.00734159     0.0117992    0.00668584    0.00178151       10     
reco:pandora2Track:LArPandoraTrackCreation            4.46478       7.21251       17.9657       5.66556       3.84113        10     
reco:pandora2calo:ShowerCalorimetry                   1.31101       2.22152       5.99605       1.56099       1.36232        10     
reco:pandora2calonosce:ShowerCalorimetry              1.31556       2.18147       5.89464       1.62345       1.3309         10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0833185     0.100835      0.160622      0.0969832     0.0209264       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0847667     0.100594       0.15531      0.0950861     0.0197342       10     
reco:pandora2Shower:LArPandoraShowerCreation          1.73719       3.1454         8.268        2.48347       1.84124        10     
reco:pandora2Showercalo:ShowerCalorimetry             1.83276       3.16493       8.23674       2.60559       1.77033        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.98889       3.22509       9.24915       2.48891       2.07307        10     
[art]:TriggerResults:TriggerResultInserter          2.1311e-05    4.35506e-05   0.000109624   3.4431e-05    2.53465e-05      10     
end_path:out1:RootOutput                             4.02e-06     8.1804e-06    3.1251e-05    5.8755e-06    7.74794e-06      10     
end_path:out1:RootOutput(write)                       3.94439       5.03893       7.39757       4.81482      0.922503        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6263.03 MB
  Peak resident set size usage (VmHWM): 4028.37 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_3_1_20251126T115742Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_3_1_20251126T115742Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_3_1_20251126T115742Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_3_1_20251126T115742Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_3_1_20251126T115742Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_3_1_20251126T115742Z_detsim_reco1.root.temp.json ====
.:
total 6121672
-rw-r--r-- 1 nobody nobody      40615 Nov 26 12:06 jobscript.log
-rw-r--r-- 1 nobody nobody        204 Nov 26 12:06 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 12:06 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_3_1_20251126T115742Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 12:06 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_3_1_20251126T115742Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 12:06 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_3_1_20251126T115742Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 12:06 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 12:06 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 12:06 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     200704 Nov 26 12:06 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 12:06 time.db
-rw-r--r-- 1 nobody nobody 6217707001 Nov 26 12:06 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_3_1_20251126T115742Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   50580110 Nov 26 12:05 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:34 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:30 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 11:30 detsim_single_protodunehd_hist.root
justIN time: 2026-02-04 07:23:32 UTC       justIN version: 01.06.00