justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258498.1@dunegpschedd01.fnal.gov

Jobsub ID258498.1@dunegpschedd01.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 08:49:16
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce03
Last heartbeat2025-11-26 11:02:14
From worker nodeHostnameccwcondor0697
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 08:49:58
Input fileshd-protodune:pdhd_prod_beam__243875_39_1_20251106T034414Z_gen_g4_IonScintPDExt.root_255575_147_1_20251123T013544Z_PDInt.root
JobscriptExit code0
Real time2h (7887s)
CPU time1h (6662s = 84%)
Max RSS bytes6645948416 (6338 MiB)
Outputting started2025-11-26 11:01:26
Output files
Finished2025-11-26 11:02:14
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ng 110260 traces tagged "wiener"
FrameSaver: q=3.44302e+06 n=1350377 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 9264 XUs and 8649 XVs -> 72 XUVs
C:0 T:1 45675 XUs and 41094 XVs -> 1331 XUVs
C:0 T:2 38234 XUs and 42412 XVs -> 2137 XUVs
C:0 T:3 18 XUs and 28 XVs -> 11 XUVs
C:0 T:5 46569 XUs and 35883 XVs -> 3362 XUVs
C:0 T:6 26745 XUs and 32326 XVs -> 1292 XUVs
8205 XUVs total
3429 collection wire objects
8205 potential space points
Neighbour search...
309011 tests to find 188876 neighbours
Iterating with no regularization...
Begin: 3.68618e+07
0 3.42514e+07
1 3.39756e+07
2 3.39372e+07
3 3.39265e+07
Now with regularization...
Begin: 3.11746e+07
0 3.11684e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 12:01:18 CET  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258498_1_1_20251126T085004Z_detsim_reco1.root"
26-Nov-2025 12:01:18 CET  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            127.353       179.315       260.667       168.838       41.006         10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00213744     0.0147811     0.0386345    0.00983456     0.012992        10     
reco:opdec:Deconvolution                             0.926815       1.09037       1.24713       1.08993      0.100892        10     
reco:ophitspe:OpHitFinderDeco                        0.213646      0.226666      0.267437      0.223393      0.0151953       10     
reco:opflash:OpFlashFinder                           0.0668599     0.096332      0.124392      0.0969869     0.0176223       10     
reco:opslicer:OpSlicer                               0.247288      0.327489      0.393241      0.332881      0.0473545       10     
reco:wclsdatahd:WireCellToolkit                       53.3033       57.8946       68.1697       56.2584       4.96336        10     
reco:gaushit:GausHitFinder                           0.767398       1.15421       1.36529       1.18927      0.173548        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000419142   0.000592589   0.000872898   0.00053748    0.000141847      10     
reco:reco3d:SpacePointSolver                         0.671039       1.23245       1.71533        1.16        0.355409        10     
reco:hitpdune:DisambigFromSpacePoints                0.548761       1.05456       1.69196       1.01461      0.328857        10     
reco:pandora:StandardPandora                          16.0044       34.7899       84.7067       29.6693       18.6425        10     
reco:pandoraWriter:StandardPandora                   0.282172       0.36894      0.438217      0.368828      0.0482562       10     
reco:pandoraTrack:LArPandoraTrackCreation             1.63048       2.74368       4.60699       2.51617       1.04423        10     
reco:pandoraShower:LArPandoraShowerCreation           1.39691       2.89582       6.15521       2.2451        1.43927        10     
reco:pandoracalo:Calorimetry                         0.866501       1.35779       2.10807       1.29481      0.450024        10     
reco:pandoracalonosce:Calorimetry                    0.832861       1.39882       2.13936       1.2542       0.489059        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.24409       2.84846       5.97224       2.15953       1.52318        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.16556       2.60668       5.1965        2.11135       1.36434        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0460927     0.0813665     0.115936      0.0802864     0.0212917       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0457562     0.0791773     0.110697      0.0756025     0.0201048       10     
reco:emtrkmichelid:EmTrackMichelId                    29.9374       41.8079       54.9856       39.9176       6.94401        10     
reco:crttag:SingleCRTMatchingProducer                0.0058485    0.00863213     0.0121835    0.00857741    0.00166409       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00483538    0.00723503     0.0107896    0.00699672    0.00171757       10     
reco:pandora2Track:LArPandoraTrackCreation            4.1058        7.59942       12.9576       6.49838       3.08305        10     
reco:pandora2calo:ShowerCalorimetry                   1.35143       2.48375       4.40886       2.15514      0.982349        10     
reco:pandora2calonosce:ShowerCalorimetry              1.31826       2.38384       4.23778       2.02257      0.959858        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0603512     0.0831348     0.105171      0.0799306     0.0142226       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0587699     0.0843716     0.119332      0.0781767     0.0188149       10     
reco:pandora2Shower:LArPandoraShowerCreation          1.74692       3.22119       6.16916       2.5336        1.41319        10     
reco:pandora2Showercalo:ShowerCalorimetry             1.81906       3.27141       5.56588       2.70802       1.27628        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.79107       3.10303       5.2486        2.49382       1.24584        10     
[art]:TriggerResults:TriggerResultInserter          3.8903e-05    6.97637e-05   0.000183196   5.54295e-05   3.93893e-05      10     
end_path:out1:RootOutput                            1.3395e-05    2.00457e-05   4.4033e-05    1.46625e-05   1.10916e-05      10     
end_path:out1:RootOutput(write)                       2.1382        2.83464       3.42271       2.85462      0.409178        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6051.86 MB
  Peak resident set size usage (VmHWM): 3756.05 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258498_1_1_20251126T085004Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258498_1_1_20251126T085004Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258498_1_1_20251126T085004Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258498_1_1_20251126T085004Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258498_1_1_20251126T085004Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258498_1_1_20251126T085004Z_detsim_reco1.root.temp.json ====
.:
total 6154356
-rw-r--r-- 1 dune001 lbno      41523 Nov 26 12:01 jobscript.log
-rw-r--r-- 1 dune001 lbno        204 Nov 26 12:01 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno          0 Nov 26 12:01 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258498_1_1_20251126T085004Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno          0 Nov 26 12:01 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258498_1_1_20251126T085004Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno         31 Nov 26 12:01 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258498_1_1_20251126T085004Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno         25 Nov 26 12:01 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno        519 Nov 26 12:01 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno       3574 Nov 26 12:01 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno     200704 Nov 26 12:01 mem.db
-rw-r--r-- 1 dune001 lbno      36864 Nov 26 12:01 time.db
-rw-r--r-- 1 dune001 lbno 6247406356 Nov 26 12:01 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258498_1_1_20251126T085004Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno   54343760 Nov 26 11:59 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno          0 Nov 26 11:32 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno          0 Nov 26 11:29 debugprod.log
-rw-r--r-- 1 dune001 lbno        519 Nov 26 11:29 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 07:14:43 UTC       justIN version: 01.05.03