justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258670.9@dunegpschedd01.fnal.gov

Jobsub ID258670.9@dunegpschedd01.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 11:32:24
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce03
Last heartbeat2025-11-26 14:47:15
From worker nodeHostnamewn67
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 11:34:05
Input fileshd-protodune:pdhd_prod_beam__238225_111_1_20251106T051654Z_gen_g4_IonScintPDExt.root_255874_101_1_20251123T231028Z_PDInt.root
JobscriptExit code0
Real time3h (11310s)
CPU time2h (9290s = 82%)
Max RSS bytes4190310400 (3996 MiB)
Outputting started2025-11-26 14:42:37
Output files
Finished2025-11-26 14:47:15
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

saving 100927 traces tagged "wiener"
FrameSaver: q=3.3236e+06 n=1348092 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 13783 XUs and 9182 XVs -> 98 XUVs
C:0 T:1 89283 XUs and 64886 XVs -> 3941 XUVs
C:0 T:2 60713 XUs and 35387 XVs -> 965 XUVs
C:0 T:4 23 XUs and 17 XVs -> 3 XUVs
C:0 T:5 20606 XUs and 13711 XVs -> 518 XUVs
C:0 T:6 53305 XUs and 31653 XVs -> 531 XUVs
6056 XUVs total
2752 collection wire objects
6056 potential space points
Neighbour search...
373466 tests to find 110560 neighbours
Iterating with no regularization...
Begin: 5.6383e+07
0 5.16096e+07
1 5.12268e+07
2 5.11905e+07
Now with regularization...
Begin: 4.94396e+07
0 4.94298e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 11:42:26 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_9_1_20251126T113413Z_detsim_reco1.root"
26-Nov-2025 11:42:26 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            174.428       231.943       376.676       206.92        66.5156        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00148101    0.00858581     0.0156359    0.00846709    0.00458623       10     
reco:opdec:Deconvolution                              1.09795       1.41177       1.90059       1.3622       0.240841        10     
reco:ophitspe:OpHitFinderDeco                        0.305838      0.336884      0.353738      0.336978      0.0133652       10     
reco:opflash:OpFlashFinder                           0.0919365     0.150216      0.225584      0.141957      0.0413969       10     
reco:opslicer:OpSlicer                               0.352947      0.466577       0.5869       0.438872      0.0765514       10     
reco:wclsdatahd:WireCellToolkit                       67.478        70.8268       73.2198       70.8074       1.77262        10     
reco:gaushit:GausHitFinder                           0.751067       1.44964       2.21519       1.47115      0.440012        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000299833   0.000564854   0.00105978    0.000538362   0.000216256      10     
reco:reco3d:SpacePointSolver                          0.70632       1.63148       2.83584       1.43762      0.629874        10     
reco:hitpdune:DisambigFromSpacePoints                0.635841       1.32943       2.81787       1.0847       0.727082        10     
reco:pandora:StandardPandora                          21.6155       50.747        124.288       42.1496       29.4164        10     
reco:pandoraWriter:StandardPandora                   0.417593      0.806536       1.5444       0.817815      0.318536        10     
reco:pandoraTrack:LArPandoraTrackCreation             1.69631       3.67729       9.7146        3.07799       2.38596        10     
reco:pandoraShower:LArPandoraShowerCreation           1.80796       3.92528       9.14346       3.36258       2.30089        10     
reco:pandoracalo:Calorimetry                         0.928898       1.83861       3.84682       1.5452       0.932862        10     
reco:pandoracalonosce:Calorimetry                    0.896096       1.76201       3.50543       1.53732      0.869267        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.72679       3.58947       7.62708       3.07362       1.7734         10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.67047       3.26926       7.84719       2.56975       1.78023        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0471112     0.0971203     0.169202      0.0926193     0.0295671       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0521797     0.0979297     0.170351      0.0889718     0.0299333       10     
reco:emtrkmichelid:EmTrackMichelId                    40.5564       50.9653       70.9734       45.4444       10.8364        10     
reco:crttag:SingleCRTMatchingProducer               0.00527446    0.00977032     0.0168396    0.00917378    0.00335893       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00454363    0.00851737     0.013648     0.00788174     0.002523        10     
reco:pandora2Track:LArPandoraTrackCreation            5.15309       9.31243       21.243        6.5095        5.24654        10     
reco:pandora2calo:ShowerCalorimetry                   1.70827       3.17065       6.74996       2.22105       1.70645        10     
reco:pandora2calonosce:ShowerCalorimetry              1.70325       3.13067       6.30152       2.1596        1.60425        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.080394      0.106326       0.14042      0.101864      0.021586        10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0742175     0.106071      0.148602      0.104297      0.0230056       10     
reco:pandora2Shower:LArPandoraShowerCreation          2.14147       3.91482       7.78237       2.87844       1.92338        10     
reco:pandora2Showercalo:ShowerCalorimetry             2.09102       4.13188       8.77868       2.78924       2.38075        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.70882       4.15807       10.5659       2.71596       2.69378        10     
[art]:TriggerResults:TriggerResultInserter          2.9621e-05    4.43198e-05   0.000107285   3.34215e-05   2.40918e-05      10     
end_path:out1:RootOutput                             4.38e-06     9.4683e-06    3.3702e-05    6.3305e-06    8.39857e-06      10     
end_path:out1:RootOutput(write)                       3.55815       5.34571       7.28466       5.24529       1.05873        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6491.11 MB
  Peak resident set size usage (VmHWM): 4142.97 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_9_1_20251126T113413Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_9_1_20251126T113413Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_9_1_20251126T113413Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_9_1_20251126T113413Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_9_1_20251126T113413Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_9_1_20251126T113413Z_detsim_reco1.root.temp.json ====
.:
total 6396232
-rw-r--r-- 1 nobody nobody      46979 Nov 26 11:42 jobscript.log
-rw-r--r-- 1 nobody nobody        205 Nov 26 11:42 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:42 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_9_1_20251126T113413Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:42 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_9_1_20251126T113413Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 11:42 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_9_1_20251126T113413Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 11:42 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 11:42 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 11:42 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     204800 Nov 26 11:42 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 11:42 time.db
-rw-r--r-- 1 nobody nobody 6495150109 Nov 26 11:42 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258670_9_1_20251126T113413Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   54268850 Nov 26 11:40 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:04 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:01 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 11:00 detsim_single_protodunehd_hist.root
justIN time: 2026-02-04 07:19:53 UTC       justIN version: 01.06.00