justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258692.3@dunegpschedd01.fnal.gov

Jobsub ID258692.3@dunegpschedd01.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 11:44:28
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce01
Last heartbeat2025-11-26 15:38:17
From worker nodeHostnamewn53
cpuinfoIntel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 11:45:49
Input fileshd-protodune:pdhd_prod_beam__243918_126_1_20251106T044555Z_gen_g4_IonScintPDExt.root_249493_21_1_20251122T002023Z_PDInt.root
JobscriptExit code0
Real time3h (13697s)
CPU time2h (10192s = 74%)
Max RSS bytes6559739904 (6255 MiB)
Outputting started2025-11-26 15:34:08
Output files
Finished2025-11-26 15:38:17
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ged "wiener"
FrameSaver: q=4.85645e+06 n=1525293 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 9931 XUs and 11632 XVs -> 112 XUVs
C:0 T:1 87431 XUs and 95267 XVs -> 3119 XUVs
C:0 T:2 67526 XUs and 189472 XVs -> 7777 XUVs
C:0 T:3 88 XUs and 375 XVs -> 27 XUVs
C:0 T:4 20 XUs and 27 XVs -> 0 XUVs
C:0 T:5 28086 XUs and 25012 XVs -> 685 XUVs
C:0 T:6 90742 XUs and 242941 XVs -> 14855 XUVs
C:0 T:7 169 XUs and 550 XVs -> 56 XUVs
26631 XUVs total
6416 collection wire objects
26631 potential space points
Neighbour search...
2861999 tests to find 1438672 neighbours
Iterating with no regularization...
Begin: 1.18225e+08
0 9.73838e+07
1 9.61393e+07
2 9.59477e+07
3 9.58844e+07
Now with regularization...
Begin: 9.11743e+07
0 9.11306e+07
26-Nov-2025 12:33:57 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258692_3_1_20251126T114555Z_detsim_reco1.root"
26-Nov-2025 12:33:57 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            209.439       333.261       447.305       328.502       65.387         10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00386477     0.0106599     0.0175433     0.0099102    0.00446787       10     
reco:opdec:Deconvolution                              1.35097       1.77194       2.56176       1.7118       0.315206        10     
reco:ophitspe:OpHitFinderDeco                        0.426206      0.445518      0.494118      0.434324      0.0204774       10     
reco:opflash:OpFlashFinder                           0.106469      0.167601      0.303869      0.148988      0.0509876       10     
reco:opslicer:OpSlicer                               0.429976      0.575045      0.959399      0.526513       0.13946        10     
reco:wclsdatahd:WireCellToolkit                       78.0836       84.6482       90.9295       86.0954       4.65506        10     
reco:gaushit:GausHitFinder                            1.2471        1.87236       2.31932       1.92953      0.331172        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000503342   0.000798113   0.00140029    0.000725706   0.000246091      10     
reco:reco3d:SpacePointSolver                         0.464196       2.49542       5.23758       2.34346       1.22641        10     
reco:hitpdune:DisambigFromSpacePoints                 1.05796       2.36592       3.76689       2.34029      0.694843        10     
reco:pandora:StandardPandora                          26.6475       58.7917       100.839       56.6541       21.0344        10     
reco:pandoraWriter:StandardPandora                   0.446107      0.633418      0.887215      0.619495      0.133164        10     
reco:pandoraTrack:LArPandoraTrackCreation             4.79299       8.59301       14.2906       8.11421       2.7382         10     
reco:pandoraShower:LArPandoraShowerCreation           3.84142       8.84052       16.8906       8.71537       3.55627        10     
reco:pandoracalo:Calorimetry                          2.06841       3.84358       5.97342       3.88873       1.11022        10     
reco:pandoracalonosce:Calorimetry                     1.49674       3.80757       6.25389       3.85275       1.30131        10     
reco:pandoraShowercalo:ShowerCalorimetry              3.20558       7.99697       14.1384       7.73589       3.1668         10     
reco:pandoraShowercalonosce:ShowerCalorimetry         3.31151       7.81594       14.7703       7.64526       3.24078        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0760521     0.143492      0.219445      0.152961      0.0365371       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.108625      0.146266      0.216296      0.149438      0.0294414       10     
reco:emtrkmichelid:EmTrackMichelId                    47.9508       66.4126       85.9041       68.4014       10.4405        10     
reco:crttag:SingleCRTMatchingProducer                0.011119      0.0204182     0.0312499     0.0208836    0.00503239       10     
reco:crtreco:TwoCRTMatchingProducer                  0.0122446     0.0190221     0.0254886     0.0202396    0.00389205       10     
reco:pandora2Track:LArPandoraTrackCreation            10.2203       22.7392       39.4268       22.9267       7.70647        10     
reco:pandora2calo:ShowerCalorimetry                   3.03058       7.38815       13.1733       7.32771       2.90141        10     
reco:pandora2calonosce:ShowerCalorimetry              2.93336       7.39299       13.1739       7.30183       2.94813        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.109857      0.148324       0.21146      0.148761      0.0264603       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.105538      0.145107      0.210428      0.147238      0.0275765       10     
reco:pandora2Shower:LArPandoraShowerCreation          4.67634       10.3029       18.2212       10.7121       3.57162        10     
reco:pandora2Showercalo:ShowerCalorimetry             4.46818       9.47998       14.3196       9.37643       3.06799        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        2.32584       9.34315       18.0731       9.72969       4.0996         10     
[art]:TriggerResults:TriggerResultInserter          2.6771e-05    6.36312e-05   0.000125961   5.8111e-05     2.818e-05       10     
end_path:out1:RootOutput                             7.784e-06    1.24747e-05   3.1013e-05    1.0225e-05    6.67542e-06      10     
end_path:out1:RootOutput(write)                       3.65998       4.71176       6.92447       4.30095      0.968185        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7111.74 MB
  Peak resident set size usage (VmHWM): 4916.79 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258692_3_1_20251126T114555Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258692_3_1_20251126T114555Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258692_3_1_20251126T114555Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258692_3_1_20251126T114555Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258692_3_1_20251126T114555Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258692_3_1_20251126T114555Z_detsim_reco1.root.temp.json ====
.:
total 6576872
-rw-r--r-- 1 nobody nobody      40714 Nov 26 12:34 jobscript.log
-rw-r--r-- 1 nobody nobody        204 Nov 26 12:34 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 12:34 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258692_3_1_20251126T114555Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 12:34 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258692_3_1_20251126T114555Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 12:34 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258692_3_1_20251126T114555Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 12:34 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 12:33 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 12:33 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     204800 Nov 26 12:33 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 12:33 time.db
-rw-r--r-- 1 nobody nobody 6676374830 Nov 26 12:33 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258692_3_1_20251126T114555Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   58015230 Nov 26 12:29 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:40 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:34 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 11:33 detsim_single_protodunehd_hist.root
justIN time: 2026-02-04 07:22:25 UTC       justIN version: 01.06.00