justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 252754.4@dunegpschedd02.fnal.gov

Jobsub ID252754.4@dunegpschedd02.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 06:51:10
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce04
Last heartbeat2025-11-26 10:27:18
From worker nodeHostnamewn114
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 06:52:34
Input fileshd-protodune:pdhd_prod_beam__240955_40_1_20251110T234302Z_gen_g4_IonScintPDExt.root_255155_128_1_20251121T231930Z_PDInt.root
JobscriptExit code0
Real time3h (12619s)
CPU time2h (8626s = 68%)
Max RSS bytes4199317504 (4004 MiB)
Outputting started2025-11-26 10:22:54
Output files
Finished2025-11-26 10:27:18
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

eSaver: saving 109058 traces tagged "wiener"
FrameSaver: q=3.41349e+06 n=1218181 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 13065 XUs and 7910 XVs -> 82 XUVs
C:0 T:1 59143 XUs and 39135 XVs -> 1105 XUVs
C:0 T:2 100687 XUs and 97238 XVs -> 2206 XUVs
C:0 T:3 39 XUs and 44 XVs -> 3 XUVs
C:0 T:4 5 XUs and 11 XVs -> 2 XUVs
C:0 T:5 9178 XUs and 6294 XVs -> 339 XUVs
C:0 T:6 40081 XUs and 55785 XVs -> 2619 XUVs
C:0 T:7 9 XUs and 37 XVs -> 6 XUVs
6362 XUVs total
2983 collection wire objects
6362 potential space points
Neighbour search...
168840 tests to find 99410 neighbours
Iterating with no regularization...
Begin: 8.32091e+07
0 7.9179e+07
1 7.90209e+07
2 7.90016e+07
Now with regularization...
Begin: 7.57438e+07
0 7.57355e+07
26-Nov-2025 07:22:40 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252754_4_1_20251126T065242Z_detsim_reco1.root"
26-Nov-2025 07:22:40 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            177.709       262.603       462.112       253.679       78.9152        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00205487     0.0111135     0.0225956    0.00944588    0.00647868       10     
reco:opdec:Deconvolution                             0.994226       1.26022       1.41623       1.27113      0.136037        10     
reco:ophitspe:OpHitFinderDeco                        0.212868      0.311542      0.340414      0.318631      0.0349117       10     
reco:opflash:OpFlashFinder                           0.0668669     0.127353      0.182286      0.122346      0.0357808       10     
reco:opslicer:OpSlicer                                0.21932      0.373145      0.469224      0.385809      0.070487        10     
reco:wclsdatahd:WireCellToolkit                       80.6673       83.7252       87.9031        82.62        2.55722        10     
reco:gaushit:GausHitFinder                            1.12941       1.42175       2.14159       1.28458      0.304047        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.00077652    0.00148043    0.00247534    0.00144362    0.000460911      10     
reco:reco3d:SpacePointSolver                          1.07219       2.43988       5.9667        2.11904       1.3963         10     
reco:hitpdune:DisambigFromSpacePoints                0.574775       1.51163       3.54645       1.4332       0.858226        10     
reco:pandora:StandardPandora                          19.2547       45.5752       98.8033       35.9699       23.9529        10     
reco:pandoraWriter:StandardPandora                   0.377199      0.599684      0.972404      0.562036      0.163958        10     
reco:pandoraTrack:LArPandoraTrackCreation             2.3176        5.08976       13.6477       3.32058       3.33314        10     
reco:pandoraShower:LArPandoraShowerCreation           2.09056       4.40588       9.51096       3.69488       2.37116        10     
reco:pandoracalo:Calorimetry                         0.927108       2.33957       5.3788        1.73102       1.41667        10     
reco:pandoracalonosce:Calorimetry                    0.882253       2.21139       4.90939       1.69791       1.26265        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.70705       4.48991       11.6853       3.23066       3.06842        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.97076       4.30255       8.70734       3.33975       2.32971        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0879787     0.114688      0.164014      0.111913      0.0251439       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0879939     0.110093      0.164551      0.0999924     0.0241274       10     
reco:emtrkmichelid:EmTrackMichelId                    40.0485       62.4869       105.277       63.7321       19.4675        10     
reco:crttag:SingleCRTMatchingProducer               0.00852426     0.0148417     0.0237667     0.0134703    0.00484152       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00748825     0.0139227     0.0325333     0.010658     0.00787333       10     
reco:pandora2Track:LArPandoraTrackCreation            5.31776       11.6263       27.5784       8.57232       7.3482         10     
reco:pandora2calo:ShowerCalorimetry                   1.68605       4.22618       13.3734       2.76358       3.53531        10     
reco:pandora2calonosce:ShowerCalorimetry              1.55772       4.10256       12.3011       2.81245       3.29625        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0486685     0.111424      0.171451      0.105379      0.0362272       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0505585     0.111163      0.178702      0.107691      0.035808        10     
reco:pandora2Shower:LArPandoraShowerCreation          2.07599       4.70195       13.0385       3.11052       3.33677        10     
reco:pandora2Showercalo:ShowerCalorimetry             2.24013       5.52495       14.8455       3.62578       4.09146        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        2.2192        5.21085       13.4529       3.43322       3.68669        10     
[art]:TriggerResults:TriggerResultInserter          3.4071e-05    5.5735e-05    0.000132085   4.85215e-05   2.67219e-05      10     
end_path:out1:RootOutput                             6.36e-06     1.08961e-05   3.6401e-05     7.63e-06     8.62468e-06      10     
end_path:out1:RootOutput(write)                       2.78052       3.78861       5.61279       3.68006      0.756602        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6181.91 MB
  Peak resident set size usage (VmHWM): 3956.42 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252754_4_1_20251126T065242Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252754_4_1_20251126T065242Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252754_4_1_20251126T065242Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252754_4_1_20251126T065242Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252754_4_1_20251126T065242Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252754_4_1_20251126T065242Z_detsim_reco1.root.temp.json ====
.:
total 5722092
-rw-r--r-- 1 nobody nobody      40987 Nov 26 07:22 jobscript.log
-rw-r--r-- 1 nobody nobody        204 Nov 26 07:22 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 07:22 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252754_4_1_20251126T065242Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 07:22 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252754_4_1_20251126T065242Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 07:22 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252754_4_1_20251126T065242Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 07:22 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 07:22 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 07:22 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     204800 Nov 26 07:22 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 07:22 time.db
-rw-r--r-- 1 nobody nobody 5805905303 Nov 26 07:22 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252754_4_1_20251126T065242Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   53201080 Nov 26 07:19 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 06:41 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 06:36 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 06:35 detsim_single_protodunehd_hist.root
justIN time: 2026-02-04 07:21:40 UTC       justIN version: 01.06.00