justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258421.20@dunegpschedd01.fnal.gov

Jobsub ID258421.20@dunegpschedd01.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 06:53:11
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce03
Last heartbeat2025-11-26 08:40:51
From worker nodeHostnameccwcondor0366
cpuinfoAMD EPYC 9334 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 06:54:51
Input fileshd-protodune:pdhd_prod_beam__244004_24_1_20251106T073152Z_gen_g4_IonScintPDExt.root_249507_172_1_20251122T010345Z_PDInt.root
JobscriptExit code0
Real time1h (6333s)
CPU time1h (5787s = 91%)
Max RSS bytes8409198592 (8019 MiB)
Outputting started2025-11-26 08:40:25
Output files
Finished2025-11-26 08:40:51
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ing XUV coincidences...
C:0 T:0 17751 XUs and 14008 XVs -> 176 XUVs
C:0 T:1 161955 XUs and 137294 XVs -> 5734 XUVs
C:0 T:2 35662 XUs and 39996 XVs -> 935 XUVs
C:0 T:3 15 XUs and 36 XVs -> 1 XUVs
C:0 T:5 25037 XUs and 30501 XVs -> 684 XUVs
C:0 T:6 17239 XUs and 23293 XVs -> 919 XUVs
C:0 T:7 9 XUs and 23 XVs -> 0 XUVs
8449 XUVs total
4083 collection wire objects
8449 potential space points
Neighbour search...
445479 tests to find 132212 neighbours
Iterating with no regularization...
Begin: 6.92732e+07
0 6.24925e+07
1 6.19859e+07
2 6.19623e+07
Now with regularization...
Begin: 5.9704e+07
0 5.96976e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 09:40:19 CET  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258421_20_1_20251126T065455Z_detsim_reco1.root"
26-Nov-2025 09:40:19 CET  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            80.2674       139.624       277.969       119.991       60.989         10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00135142    0.00703775     0.0155131    0.00539853    0.00430021       10     
reco:opdec:Deconvolution                              0.63658      0.836138       1.09445      0.766148      0.169427        10     
reco:ophitspe:OpHitFinderDeco                        0.121576      0.174418      0.213626       0.17735      0.0290424       10     
reco:opflash:OpFlashFinder                           0.0573275     0.101331      0.165541      0.0849188     0.0375246       10     
reco:opslicer:OpSlicer                                0.15303      0.242419      0.322976      0.268764      0.0560804       10     
reco:wclsdatahd:WireCellToolkit                       38.3898       41.7777       45.6127       41.1494       2.48196        10     
reco:gaushit:GausHitFinder                           0.631809      0.915369       1.3353       0.923304      0.254923        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000215677   0.000347864   0.000481659   0.000342934   8.13921e-05      10     
reco:reco3d:SpacePointSolver                          0.3976        1.54609       4.37466       1.43646       1.0829         10     
reco:hitpdune:DisambigFromSpacePoints                0.229004       0.99812       3.11538      0.701241      0.813249        10     
reco:pandora:StandardPandora                          8.83002       33.6966       78.1595       25.606        23.5362        10     
reco:pandoraWriter:StandardPandora                   0.173897      0.304046      0.430432      0.320402      0.0861894       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.828562       3.00264       9.5663        2.41439       2.41976        10     
reco:pandoraShower:LArPandoraShowerCreation          0.929078       3.06735       10.9934       1.89771       2.94792        10     
reco:pandoracalo:Calorimetry                         0.413193       1.39218       3.96185      0.946206       1.08446        10     
reco:pandoracalonosce:Calorimetry                    0.463899       1.31281       2.59619       1.1445       0.703811        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.590051       2.49618       8.08409       1.41065       2.19267        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.578594       2.48111       9.87466       1.15296       2.66769        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0255906     0.049305      0.110053      0.0482943     0.0227006       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0270933     0.0498379     0.112921      0.0439038     0.0229866       10     
reco:emtrkmichelid:EmTrackMichelId                    13.6655       21.454        36.4216       20.671        6.50498        10     
reco:crttag:SingleCRTMatchingProducer               0.00380216    0.00804933     0.0126719    0.00776484    0.00284103       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00280783    0.00661513     0.0127067    0.00558851    0.00299836       10     
reco:pandora2Track:LArPandoraTrackCreation            2.09019       7.72141       24.0936       4.82493       6.29157        10     
reco:pandora2calo:ShowerCalorimetry                  0.563987       2.12653       7.20381       1.37261       1.95727        10     
reco:pandora2calonosce:ShowerCalorimetry             0.542326       2.36412       9.11087       1.21415       2.49606        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0269709     0.0560787     0.109881      0.0489384     0.0261887       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0268719     0.0552082     0.105973      0.0469353     0.0259055       10     
reco:pandora2Shower:LArPandoraShowerCreation          0.63167       3.25729       11.3447       2.29496       3.20236        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.626608       2.85067       8.57532       1.92078       2.44397        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        0.63659       2.79416       8.85905       2.06904       2.3753         10     
[art]:TriggerResults:TriggerResultInserter          2.2324e-05    4.33658e-05    8.549e-05    4.27245e-05   1.78099e-05      10     
end_path:out1:RootOutput                             5.237e-06    1.11648e-05   3.5063e-05    8.3925e-06    8.1493e-06       10     
end_path:out1:RootOutput(write)                       1.47062       2.27585       3.38603       2.2949       0.573891        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7496.67 MB
  Peak resident set size usage (VmHWM): 4884.03 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258421_20_1_20251126T065455Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258421_20_1_20251126T065455Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258421_20_1_20251126T065455Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258421_20_1_20251126T065455Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258421_20_1_20251126T065455Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258421_20_1_20251126T065455Z_detsim_reco1.root.temp.json ====
.:
total 6566760
-rw-r--r-- 1 dune001 lbno      41413 Nov 26 09:40 jobscript.log
-rw-r--r-- 1 dune001 lbno        204 Nov 26 09:40 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno          0 Nov 26 09:40 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258421_20_1_20251126T065455Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno          0 Nov 26 09:40 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258421_20_1_20251126T065455Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno         31 Nov 26 09:40 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258421_20_1_20251126T065455Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno         25 Nov 26 09:40 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno        519 Nov 26 09:40 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno       3574 Nov 26 09:40 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno     204800 Nov 26 09:40 mem.db
-rw-r--r-- 1 dune001 lbno      36864 Nov 26 09:40 time.db
-rw-r--r-- 1 dune001 lbno 6664861036 Nov 26 09:40 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258421_20_1_20251126T065455Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno   59186510 Nov 26 09:39 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno          0 Nov 26 09:17 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno          0 Nov 26 09:15 debugprod.log
-rw-r--r-- 1 dune001 lbno        519 Nov 26 09:15 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 23:27:08 UTC       justIN version: 01.05.03