justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 253026.5@dunegpschedd02.fnal.gov

Jobsub ID253026.5@dunegpschedd02.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 12:06:26
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce03
Last heartbeat2025-11-26 14:58:32
From worker nodeHostnamewn70
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 12:07:33
Input fileshd-protodune:pdhd_prod_beam__238099_4_1_20251106T001200Z_gen_g4_IonScintPDExt.root_255826_78_1_20251123T215640Z_PDInt.root
JobscriptExit code0
Real time2h (9956s)
CPU time2h (7845s = 78%)
Max RSS bytes4181147648 (3987 MiB)
Outputting started2025-11-26 14:53:31
Output files
Finished2025-11-26 14:58:32
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

FrameSaver: q=2.86219e+06 n=1065291 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 10334 XUs and 8845 XVs -> 61 XUVs
C:0 T:1 96306 XUs and 92913 XVs -> 7822 XUVs
C:0 T:2 22355 XUs and 20345 XVs -> 538 XUVs
C:0 T:3 11 XUs and 20 XVs -> 0 XUVs
C:0 T:4 6 XUs and 16 XVs -> 1 XUVs
C:0 T:5 52621 XUs and 76927 XVs -> 1650 XUVs
C:0 T:6 3736 XUs and 7685 XVs -> 233 XUVs
10305 XUVs total
2367 collection wire objects
10305 potential space points
Neighbour search...
954175 tests to find 533692 neighbours
Iterating with no regularization...
Begin: 4.28219e+07
0 3.77763e+07
1 3.74818e+07
2 3.74471e+07
Now with regularization...
Begin: 3.6241e+07
0 3.62279e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 11:53:19 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_5_1_20251126T120739Z_detsim_reco1.root"
26-Nov-2025 11:53:19 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            140.429       193.63        240.242       190.114       32.5927        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00174983    0.00875385     0.015691      0.008817     0.00454247       10     
reco:opdec:Deconvolution                              1.03983       1.23671       1.50338       1.24227      0.137348        10     
reco:ophitspe:OpHitFinderDeco                        0.315325       0.34094      0.416948      0.326119      0.0342241       10     
reco:opflash:OpFlashFinder                           0.0704064     0.116372      0.180754      0.116583      0.0292751       10     
reco:opslicer:OpSlicer                               0.261857       0.37848      0.485167      0.381394      0.0687103       10     
reco:wclsdatahd:WireCellToolkit                       65.2777       69.5151       72.1358       69.8637       1.82731        10     
reco:gaushit:GausHitFinder                           0.922484       1.24511       1.62495       1.23005      0.228593        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000343034   0.000505378   0.000742731   0.000488494   0.00011567       10     
reco:reco3d:SpacePointSolver                         0.633826       1.68135       3.41616       1.41158      0.798533        10     
reco:hitpdune:DisambigFromSpacePoints                0.425906       1.00552       2.03474      0.983661      0.433489        10     
reco:pandora:StandardPandora                          17.3001       38.7021       58.9095       35.7579       13.3834        10     
reco:pandoraWriter:StandardPandora                   0.344037      0.557855      0.830188      0.529135       0.15294        10     
reco:pandoraTrack:LArPandoraTrackCreation             1.0801        2.52444       3.60135       2.80297      0.825173        10     
reco:pandoraShower:LArPandoraShowerCreation           0.9289        2.21101       3.31399       2.04783      0.779798        10     
reco:pandoracalo:Calorimetry                          0.61605       1.24407       1.69446       1.32943      0.326404        10     
reco:pandoracalonosce:Calorimetry                    0.590261       1.24334       1.74755       1.32755      0.380905        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.941917       2.12969       3.32415       2.00385       0.70757        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.920452       2.01791       3.23143       1.92589      0.736417        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0711978     0.0967197     0.125769      0.0943873     0.0156464       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0472881     0.0925202     0.122911      0.0918189     0.0206334       10     
reco:emtrkmichelid:EmTrackMichelId                    32.6693       44.2832       54.4023       43.4969       7.3892         10     
reco:crttag:SingleCRTMatchingProducer               0.00662922    0.00937923     0.0113774    0.00934108    0.00155054       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00545174    0.00779365     0.0107255    0.00755805     0.0019143       10     
reco:pandora2Track:LArPandoraTrackCreation            3.47247       6.21605       10.2777       5.43334       2.34228        10     
reco:pandora2calo:ShowerCalorimetry                  0.943968       1.82784       2.95356       1.7233       0.720716        10     
reco:pandora2calonosce:ShowerCalorimetry             0.973905       1.77405       3.04009       1.56663      0.732366        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0594389     0.0890481     0.124796      0.0846924     0.0209322       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0652049     0.0921657     0.126526      0.0859023     0.0176742       10     
reco:pandora2Shower:LArPandoraShowerCreation          1.27983       2.60058       5.01552       2.2806        1.04838        10     
reco:pandora2Showercalo:ShowerCalorimetry             1.39556       2.77156       5.40331       2.36715       1.18429        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.3784        2.63744       4.19454       2.4367       0.881887        10     
[art]:TriggerResults:TriggerResultInserter          3.1342e-05    5.04682e-05   0.000147416   4.12365e-05   3.25643e-05      10     
end_path:out1:RootOutput                             5.65e-06     9.3095e-06    3.2982e-05     6.726e-06    7.92319e-06      10     
end_path:out1:RootOutput(write)                       2.95916       4.80939       6.12332       4.85077      0.981148        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6134.51 MB
  Peak resident set size usage (VmHWM): 3907.16 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_5_1_20251126T120739Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_5_1_20251126T120739Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_5_1_20251126T120739Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_5_1_20251126T120739Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_5_1_20251126T120739Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_5_1_20251126T120739Z_detsim_reco1.root.temp.json ====
.:
total 5470256
-rw-r--r-- 1 nobody nobody      41949 Nov 26 11:53 jobscript.log
-rw-r--r-- 1 nobody nobody        203 Nov 26 11:53 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:53 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_5_1_20251126T120739Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:53 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_5_1_20251126T120739Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 11:53 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_5_1_20251126T120739Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 11:53 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 11:53 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 11:53 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     200704 Nov 26 11:53 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 11:53 time.db
-rw-r--r-- 1 nobody nobody 5555361666 Nov 26 11:53 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_5_1_20251126T120739Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   45869910 Nov 26 11:51 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:21 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:18 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 11:17 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 07:22:36 UTC       justIN version: 01.05.03