justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258655.3@dunegpschedd01.fnal.gov

Jobsub ID258655.3@dunegpschedd01.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 11:20:24
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce04
Last heartbeat2025-11-26 15:09:51
From worker nodeHostnamewn124
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 11:21:48
Input fileshd-protodune:pdhd_prod_beam__238390_64_1_20251106T073553Z_gen_g4_IonScintPDExt.root_249858_96_1_20251122T220528Z_PDInt.root
JobscriptExit code0
Real time3h (12934s)
CPU time2h (9509s = 73%)
Max RSS bytes4222586880 (4026 MiB)
Outputting started2025-11-26 14:57:23
Output files
Finished2025-11-26 15:09:51
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

nces...
C:0 T:0 8682 XUs and 10011 XVs -> 121 XUVs
C:0 T:1 76530 XUs and 76958 XVs -> 3735 XUVs
C:0 T:2 79028 XUs and 51941 XVs -> 2604 XUVs
C:0 T:3 6 XUs and 8 XVs -> 2 XUVs
C:0 T:5 15534 XUs and 29573 XVs -> 1434 XUVs
C:0 T:6 35236 XUs and 29483 XVs -> 932 XUVs
C:0 T:7 8 XUs and 13 XVs -> 0 XUVs
8828 XUVs total
3188 collection wire objects
8828 potential space points
Neighbour search...
624772 tests to find 246854 neighbours
Iterating with no regularization...
Begin: 6.97591e+07
0 6.17319e+07
1 6.09956e+07
2 6.09265e+07
3 6.09099e+07
Now with regularization...
Begin: 5.86115e+07
0 5.86006e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 11:57:11 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258655_3_1_20251126T112155Z_detsim_reco1.root"
26-Nov-2025 11:57:11 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            187.122       256.589       371.429       247.842       53.2972        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00153549     0.0148119     0.0643629    0.00891701     0.0172125       10     
reco:opdec:Deconvolution                              1.21012       1.56339       1.8031        1.64385      0.199595        10     
reco:ophitspe:OpHitFinderDeco                        0.322609      0.358633      0.379137      0.361364       0.01599        10     
reco:opflash:OpFlashFinder                           0.122065      0.180984      0.209685      0.187425      0.0267094       10     
reco:opslicer:OpSlicer                               0.440133      0.520029      0.629981      0.501825      0.0681829       10     
reco:wclsdatahd:WireCellToolkit                       73.1513       76.3357       79.793        76.103        2.1079         10     
reco:gaushit:GausHitFinder                            1.23931       1.57087       2.10955       1.47739      0.247286        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000422749   0.000970519   0.00193449    0.000923579   0.000441197      10     
reco:reco3d:SpacePointSolver                         0.975862       2.15978       3.25648       2.20889      0.749081        10     
reco:hitpdune:DisambigFromSpacePoints                 0.87629       1.53508       2.75858       1.51873      0.539749        10     
reco:pandora:StandardPandora                          24.7907       57.3072       122.36        45.7378       32.1991        10     
reco:pandoraWriter:StandardPandora                   0.490475       0.70921       1.04634      0.650243      0.183743        10     
reco:pandoraTrack:LArPandoraTrackCreation             2.58789       4.34809       9.5812        3.59156       1.96716        10     
reco:pandoraShower:LArPandoraShowerCreation           1.64732       4.34839       10.2582       3.67197       2.31307        10     
reco:pandoracalo:Calorimetry                          1.31245       2.2734        6.12068       1.87356       1.32857        10     
reco:pandoracalonosce:Calorimetry                     1.2821        2.19766       5.19387       1.8694        1.07592        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.60862       3.81672       6.72767       3.60991       1.45786        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.52624       3.76952       6.83826       3.49137       1.51728        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0896779     0.123524      0.153842      0.130322      0.0197876       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0863739     0.122593      0.149252      0.134178      0.0235312       10     
reco:emtrkmichelid:EmTrackMichelId                    45.7234       56.1749       78.0782       54.7034       8.48868        10     
reco:crttag:SingleCRTMatchingProducer               0.00752145     0.0118624     0.0237369     0.0105912    0.00446017       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00636042    0.00881949     0.0128835    0.00810139    0.00221437       10     
reco:pandora2Track:LArPandoraTrackCreation            5.89174       10.8441       23.629        10.095        4.6837         10     
reco:pandora2calo:ShowerCalorimetry                   1.5114        3.57293       7.55834       3.30534       1.68142        10     
reco:pandora2calonosce:ShowerCalorimetry              1.50628       3.17038       5.10285       3.14066       1.13326        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.105633      0.129334      0.154387      0.130092      0.017482        10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.102631      0.127291       0.15489      0.127137      0.0195037       10     
reco:pandora2Shower:LArPandoraShowerCreation          2.34307       4.29584       6.94669       4.2518        1.32157        10     
reco:pandora2Showercalo:ShowerCalorimetry             2.35677       4.5516        9.45199       4.22144       1.88175        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        2.24899       4.4995        9.07758       4.10417       1.84043        10     
[art]:TriggerResults:TriggerResultInserter          3.3052e-05    9.88306e-05   0.000517245   4.6202e-05    0.000141487      10     
end_path:out1:RootOutput                             5.02e-06     3.31433e-05   0.000236501    7.895e-06    6.83031e-05      10     
end_path:out1:RootOutput(write)                       4.61327       5.72292       8.3702        5.34701       1.01084        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6296.81 MB
  Peak resident set size usage (VmHWM): 4048.51 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258655_3_1_20251126T112155Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258655_3_1_20251126T112155Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258655_3_1_20251126T112155Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258655_3_1_20251126T112155Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258655_3_1_20251126T112155Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258655_3_1_20251126T112155Z_detsim_reco1.root.temp.json ====
.:
total 6458364
-rw-r--r-- 1 nobody nobody      41109 Nov 26 11:57 jobscript.log
-rw-r--r-- 1 nobody nobody        203 Nov 26 11:57 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:57 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258655_3_1_20251126T112155Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:57 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258655_3_1_20251126T112155Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 11:57 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258655_3_1_20251126T112155Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 11:57 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 11:57 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 11:57 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     204800 Nov 26 11:57 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 11:57 time.db
-rw-r--r-- 1 nobody nobody 6558508573 Nov 26 11:57 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258655_3_1_20251126T112155Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   54536700 Nov 26 11:55 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:16 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:11 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 11:10 detsim_single_protodunehd_hist.root
justIN time: 2026-02-04 07:18:52 UTC       justIN version: 01.06.00