justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 253013.4@dunegpschedd02.fnal.gov

Jobsub ID253013.4@dunegpschedd02.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 11:56:25
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce03
Last heartbeat2025-11-26 15:10:37
From worker nodeHostnamewn73
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 11:57:58
Input fileshd-protodune:pdhd_prod_beam__238468_51_1_20251106T091019Z_gen_g4_IonScintPDExt.root_255814_29_1_20251123T202239Z_PDInt.root
JobscriptExit code0
Real time3h (10867s)
CPU time2h (9231s = 84%)
Max RSS bytes4377800704 (4174 MiB)
Outputting started2025-11-26 14:59:06
Output files
Finished2025-11-26 15:10:37
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

tagged "gauss"
FrameSaver: q=2.17322e+06 n=1157619 tag=gauss
wclsFrameSaver: saving 89326 traces tagged "wiener"
FrameSaver: q=2.84397e+06 n=1128416 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11229 XUs and 9790 XVs -> 106 XUVs
C:0 T:1 83846 XUs and 65041 XVs -> 2818 XUVs
C:0 T:2 40556 XUs and 57348 XVs -> 799 XUVs
C:0 T:4 19 XUs and 6 XVs -> 0 XUVs
C:0 T:5 27196 XUs and 19097 XVs -> 374 XUVs
C:0 T:6 53192 XUs and 46408 XVs -> 2220 XUVs
6317 XUVs total
2595 collection wire objects
6317 potential space points
Neighbour search...
220347 tests to find 118214 neighbours
Iterating with no regularization...
Begin: 4.56007e+07
0 4.12221e+07
1 4.10264e+07
2 4.10033e+07
Now with regularization...
Begin: 3.97353e+07
0 3.9727e+07
26-Nov-2025 11:58:56 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_4_1_20251126T115805Z_detsim_reco1.root"
26-Nov-2025 11:58:56 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            143.009       200.629       234.851       207.211       25.9025        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00195637     0.0141308     0.0570982     0.0111895     0.015168        10     
reco:opdec:Deconvolution                              1.02389       1.31777       1.57829       1.32282      0.160946        10     
reco:ophitspe:OpHitFinderDeco                        0.186093      0.317377      0.343623      0.333783      0.045198        10     
reco:opflash:OpFlashFinder                           0.0776389     0.128361      0.182193      0.126596      0.0303264       10     
reco:opslicer:OpSlicer                               0.289216      0.392672      0.587106      0.372424      0.0843015       10     
reco:wclsdatahd:WireCellToolkit                       66.4293       68.5048       71.8681       68.4057       1.76299        10     
reco:gaushit:GausHitFinder                            0.85801       1.31203       1.64028       1.37464      0.237514        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000335261   0.000581608   0.00145996    0.00045465    0.000314998      10     
reco:reco3d:SpacePointSolver                         0.430725       1.60025       2.92814       1.34753      0.777039        10     
reco:hitpdune:DisambigFromSpacePoints                 0.53634       1.12951       1.43127       1.18499      0.231625        10     
reco:pandora:StandardPandora                          12.0893       33.1768       65.7512       29.7129       13.5264        10     
reco:pandoraWriter:StandardPandora                   0.400993      0.684039       1.29264      0.590473      0.254976        10     
reco:pandoraTrack:LArPandoraTrackCreation             1.70077       3.09387       4.86392       2.71134       0.96669        10     
reco:pandoraShower:LArPandoraShowerCreation           1.5031        2.69894       3.69037       2.80863      0.661447        10     
reco:pandoracalo:Calorimetry                         0.944135       1.42649       2.01844       1.37609      0.374347        10     
reco:pandoracalonosce:Calorimetry                    0.940888       1.57704       2.68122       1.39654      0.537365        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.20656       2.89686       5.21298       2.68626       1.15984        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.04234       2.67278       4.11012       2.65598      0.876274        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0482644     0.100986      0.140017      0.107383      0.0267539       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0468893     0.0999951     0.134401      0.106261      0.0260936       10     
reco:emtrkmichelid:EmTrackMichelId                    34.589        48.8077       59.3653        50.47        6.2613         10     
reco:crttag:SingleCRTMatchingProducer               0.00719522     0.0103223     0.0153749    0.00933767    0.00258985       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00560237    0.00865415     0.0158625    0.00783028    0.00280162       10     
reco:pandora2Track:LArPandoraTrackCreation            4.04856       7.95527       12.0916       7.76498       2.33983        10     
reco:pandora2calo:ShowerCalorimetry                   1.39035       2.58759       3.85412       2.5858       0.737663        10     
reco:pandora2calonosce:ShowerCalorimetry              1.34669       2.49034       3.68156       2.37694       0.70564        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0632054     0.0976702     0.121964      0.100751      0.018632        10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0599245     0.095582      0.121023      0.0998583     0.018459        10     
reco:pandora2Shower:LArPandoraShowerCreation          1.65957       3.26843       4.75602       3.20312      0.882208        10     
reco:pandora2Showercalo:ShowerCalorimetry             1.86964       3.37433       5.07864       3.37635       0.88468        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.83172       3.2709        4.59964       3.19771      0.812475        10     
[art]:TriggerResults:TriggerResultInserter          3.0481e-05    4.55625e-05   0.000123384   3.4431e-05    2.7072e-05       10     
end_path:out1:RootOutput                             5.17e-06     1.20312e-05   3.7081e-05     6.905e-06    1.10006e-05      10     
end_path:out1:RootOutput(write)                       3.7951        5.3453        6.24258       5.38388      0.796776        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6217.39 MB
  Peak resident set size usage (VmHWM): 3979.76 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_4_1_20251126T115805Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_4_1_20251126T115805Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_4_1_20251126T115805Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_4_1_20251126T115805Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_4_1_20251126T115805Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_4_1_20251126T115805Z_detsim_reco1.root.temp.json ====
.:
total 6050644
-rw-r--r-- 1 nobody nobody      40482 Nov 26 11:59 jobscript.log
-rw-r--r-- 1 nobody nobody        203 Nov 26 11:59 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:59 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_4_1_20251126T115805Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:59 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_4_1_20251126T115805Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 11:59 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_4_1_20251126T115805Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 11:58 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 11:58 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 11:58 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     204800 Nov 26 11:58 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 11:58 time.db
-rw-r--r-- 1 nobody nobody 6144976889 Nov 26 11:58 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253013_4_1_20251126T115805Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   50570430 Nov 26 11:57 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:26 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:22 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 11:22 detsim_single_protodunehd_hist.root
justIN time: 2026-02-04 07:23:20 UTC       justIN version: 01.06.00