justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 252800.31@dunegpschedd02.fnal.gov

Jobsub ID252800.31@dunegpschedd02.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 08:21:15
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce01
Last heartbeat2025-11-26 10:39:16
From worker nodeHostnameccwcondor0689
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 08:23:28
Input fileshd-protodune:pdhd_prod_beam__243791_10_1_20251106T003403Z_gen_g4_IonScintPDExt.root_249477_137_1_20251121T231120Z_PDInt.root
JobscriptExit code0
Real time2h (8099s)
CPU time1h (7088s = 87%)
Max RSS bytes7771226112 (7411 MiB)
Outputting started2025-11-26 10:38:28
Output files
Finished2025-11-26 10:39:16
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

97653 traces tagged "wiener"
FrameSaver: q=2.75195e+06 n=1116813 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 10477 XUs and 9203 XVs -> 52 XUVs
C:0 T:1 34578 XUs and 29540 XVs -> 596 XUVs
C:0 T:2 39634 XUs and 120679 XVs -> 5276 XUVs
C:0 T:3 27 XUs and 308 XVs -> 11 XUVs
C:0 T:4 5 XUs and 11 XVs -> 3 XUVs
C:0 T:5 10834 XUs and 9809 XVs -> 1748 XUVs
C:0 T:6 43389 XUs and 76240 XVs -> 2173 XUVs
C:0 T:7 16 XUs and 33 XVs -> 3 XUVs
9862 XUVs total
2716 collection wire objects
9862 potential space points
Neighbour search...
1338020 tests to find 604444 neighbours
Iterating with no regularization...
Begin: 3.55216e+07
0 3.13522e+07
1 3.10329e+07
2 3.09904e+07
3 3.09759e+07
Now with regularization...
Begin: 2.8781e+07
0 2.87712e+07
26-Nov-2025 11:38:20 CET  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252800_31_1_20251126T082333Z_detsim_reco1.root"
26-Nov-2025 11:38:20 CET  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            143.666       191.42        262.036       176.962       43.055         10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00151571     0.0140797     0.0407099     0.011397      0.0119946       10     
reco:opdec:Deconvolution                              0.80891       1.11228       1.34112       1.13891      0.165072        10     
reco:ophitspe:OpHitFinderDeco                        0.216915      0.236612      0.271899      0.234579      0.0145597       10     
reco:opflash:OpFlashFinder                           0.0581355     0.121091       0.18828      0.116073      0.0425252       10     
reco:opslicer:OpSlicer                               0.277986      0.365496      0.458207      0.383087      0.0575787       10     
reco:wclsdatahd:WireCellToolkit                       52.9197       62.171         68.11        61.736        4.17929        10     
reco:gaushit:GausHitFinder                           0.665121       1.04271       1.41183       1.05166      0.207933        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000356834   0.000581085   0.000976712   0.000531553   0.000164971      10     
reco:reco3d:SpacePointSolver                         0.807908       1.52847       2.50599       1.34421      0.503929        10     
reco:hitpdune:DisambigFromSpacePoints                0.508635       1.1487        1.74656       1.19531      0.388783        10     
reco:pandora:StandardPandora                          17.8612       36.1986       77.6004       30.3206       17.4946        10     
reco:pandoraWriter:StandardPandora                   0.282354      0.393863      0.495293      0.401447      0.0735293       10     
reco:pandoraTrack:LArPandoraTrackCreation             1.13222       2.99793       4.91919       2.64904       1.19184        10     
reco:pandoraShower:LArPandoraShowerCreation          0.869476       3.09394       4.84858       3.18261       1.31799        10     
reco:pandoracalo:Calorimetry                         0.691904       1.43825       2.24665       1.39173      0.474591        10     
reco:pandoracalonosce:Calorimetry                    0.624674       1.37968       1.97692       1.35892      0.426976        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.00767       2.98991       4.94176       3.23417       1.25186        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.959009       2.91325       4.93207       3.13366       1.22051        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0511428     0.0855355     0.108905      0.0902779     0.0190723       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0527386     0.0839889     0.107543      0.0852373     0.0156734       10     
reco:emtrkmichelid:EmTrackMichelId                    31.8032       44.9286       63.5357       42.5331       9.83846        10     
reco:crttag:SingleCRTMatchingProducer               0.00578857    0.00866515     0.0110407    0.00922717    0.00168451       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00456445    0.00752414     0.0103882    0.00802541    0.00179927       10     
reco:pandora2Track:LArPandoraTrackCreation            2.44028       8.20186       13.7135       8.03092       3.6035         10     
reco:pandora2calo:ShowerCalorimetry                   0.76048       2.71065       4.58098       2.80634       1.27689        10     
reco:pandora2calonosce:ShowerCalorimetry             0.733994       2.61878       4.41742       2.72557       1.2299         10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0551565     0.0870513      0.10903      0.0882301     0.0154633       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0515661     0.0828824     0.105328      0.083772      0.0150582       10     
reco:pandora2Shower:LArPandoraShowerCreation         0.952927       3.32492       5.59146       3.54313       1.48861        10     
reco:pandora2Showercalo:ShowerCalorimetry             1.05894       3.46868       5.45164       3.69575       1.4838         10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.06867       3.45061       5.32318       3.63513       1.47327        10     
[art]:TriggerResults:TriggerResultInserter          3.5117e-05    5.08781e-05   0.000149082   3.86025e-05   3.30284e-05      10     
end_path:out1:RootOutput                             7.925e-06    1.25347e-05    3.722e-05    1.00735e-05   8.27157e-06      10     
end_path:out1:RootOutput(write)                       2.1193        3.0392        3.8364        3.03563      0.539274        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6528.6 MB
  Peak resident set size usage (VmHWM): 4313.06 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252800_31_1_20251126T082333Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252800_31_1_20251126T082333Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252800_31_1_20251126T082333Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252800_31_1_20251126T082333Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252800_31_1_20251126T082333Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252800_31_1_20251126T082333Z_detsim_reco1.root.temp.json ====
.:
total 6683948
-rw-r--r-- 1 dune001 lbno      45523 Nov 26 11:38 jobscript.log
-rw-r--r-- 1 dune001 lbno        204 Nov 26 11:38 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno          0 Nov 26 11:38 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252800_31_1_20251126T082333Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno          0 Nov 26 11:38 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252800_31_1_20251126T082333Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno         31 Nov 26 11:38 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252800_31_1_20251126T082333Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno         25 Nov 26 11:38 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno        519 Nov 26 11:38 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno       3574 Nov 26 11:38 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno     200704 Nov 26 11:38 mem.db
-rw-r--r-- 1 dune001 lbno      36864 Nov 26 11:38 time.db
-rw-r--r-- 1 dune001 lbno 6787861068 Nov 26 11:38 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252800_31_1_20251126T082333Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno   56185600 Nov 26 11:37 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno          0 Nov 26 11:07 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno          0 Nov 26 11:04 debugprod.log
-rw-r--r-- 1 dune001 lbno        519 Nov 26 11:03 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 07:22:30 UTC       justIN version: 01.05.03