justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 252825.31@dunegpschedd02.fnal.gov

Jobsub ID252825.31@dunegpschedd02.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 08:43:20
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce01
Last heartbeat2025-11-26 10:32:24
From worker nodeHostnameccwcondor0635
cpuinfoAMD EPYC 7453 28-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 08:44:43
Input fileshd-protodune:pdhd_prod_beam__238288_178_1_20251106T063209Z_gen_g4_IonScintPDExt.root_249886_108_1_20251122T234828Z_PDInt.root
JobscriptExit code0
Real time1h (6433s)
CPU time1h (5782s = 89%)
Max RSS bytes6451093504 (6152 MiB)
Outputting started2025-11-26 10:31:57
Output files
Finished2025-11-26 10:32:24
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

2.9999e+06 n=1106287 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 15024 XUs and 13150 XVs -> 192 XUVs
C:0 T:1 85599 XUs and 78730 XVs -> 4384 XUVs
C:0 T:2 61603 XUs and 127761 XVs -> 5870 XUVs
C:0 T:3 9 XUs and 135 XVs -> 0 XUVs
C:0 T:4 3 XUs and 6 XVs -> 0 XUVs
C:0 T:5 8919 XUs and 12491 XVs -> 582 XUVs
C:0 T:6 21569 XUs and 16818 XVs -> 799 XUVs
11827 XUVs total
2766 collection wire objects
11827 potential space points
Neighbour search...
1737655 tests to find 693256 neighbours
Iterating with no regularization...
Begin: 4.94436e+07
0 4.05169e+07
1 3.98383e+07
2 3.97726e+07
3 3.97551e+07
Now with regularization...
Begin: 3.82967e+07
0 3.82853e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 11:31:50 CET  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_31_1_20251126T084453Z_detsim_reco1.root"
26-Nov-2025 11:31:50 CET  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            102.893       164.764       222.611       168.866       31.9016        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                               0.001276     0.00767059     0.0124694    0.00789995    0.00325335       10     
reco:opdec:Deconvolution                              0.71555      0.901102       1.09538      0.936194      0.126421        10     
reco:ophitspe:OpHitFinderDeco                         0.16741      0.229897      0.250089      0.234701      0.0222105       10     
reco:opflash:OpFlashFinder                           0.0452694     0.0866011     0.141215      0.0846865     0.0250337       10     
reco:opslicer:OpSlicer                               0.215991      0.280867      0.343976      0.274754      0.0396281       10     
reco:wclsdatahd:WireCellToolkit                       51.0957       53.775        59.7594       52.6964       2.77004        10     
reco:gaushit:GausHitFinder                           0.701683       1.00846       1.32737      0.976882      0.170907        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000330107   0.00045728    0.000679008   0.000433224   0.000101096      10     
reco:reco3d:SpacePointSolver                          0.41155       1.3872        2.75298       1.29528      0.713841        10     
reco:hitpdune:DisambigFromSpacePoints                0.226799      0.782299       1.32752      0.792072       0.30293        10     
reco:pandora:StandardPandora                          11.1045       31.9888       59.1584       32.8812       14.2531        10     
reco:pandoraWriter:StandardPandora                   0.238278      0.335909      0.401723      0.357314      0.0538239       10     
reco:pandoraTrack:LArPandoraTrackCreation             1.19847       3.24289       6.52752       2.90093       1.50084        10     
reco:pandoraShower:LArPandoraShowerCreation           1.04754       3.28454       6.55337       2.93256       1.61887        10     
reco:pandoracalo:Calorimetry                         0.644554       1.46715       2.8167        1.29623      0.613923        10     
reco:pandoracalonosce:Calorimetry                    0.582318       1.42257       2.65691       1.31379       0.5903         10     
reco:pandoraShowercalo:ShowerCalorimetry             0.996467       2.72472       5.39831       2.42494       1.22442        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.928883       2.68508       4.86223       2.34059       1.16125        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0531764     0.0747393     0.0988071     0.0728104     0.0156718       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0492479     0.0714877     0.094635      0.0718174     0.0149039       10     
reco:emtrkmichelid:EmTrackMichelId                    21.9256       35.4065       48.9956       35.2085       7.29319        10     
reco:crttag:SingleCRTMatchingProducer               0.00602946     0.0101192     0.0140557     0.0100886    0.00237109       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00498541    0.00836326     0.0129496    0.00837409    0.00210847       10     
reco:pandora2Track:LArPandoraTrackCreation            2.54896       7.29344       11.9156       6.88349       2.80856        10     
reco:pandora2calo:ShowerCalorimetry                  0.733445       2.16712       3.99327       1.94172      0.995517        10     
reco:pandora2calonosce:ShowerCalorimetry             0.718465       2.22835       3.96947       2.18386       1.02612        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0298848     0.0725628     0.091162      0.0789291     0.0188605       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0322043     0.0735988     0.105755      0.0761685     0.0204665       10     
reco:pandora2Shower:LArPandoraShowerCreation          1.12367       3.44558       6.16689       3.44654       1.65252        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.959108       2.9588        5.24146       2.78983       1.36458        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        0.9467        2.74883       4.40053       2.63915       1.10293        10     
[art]:TriggerResults:TriggerResultInserter          3.8662e-05    6.67707e-05   0.000138588   6.1059e-05    2.81645e-05      10     
end_path:out1:RootOutput                             4.318e-06    1.46175e-05   4.5866e-05    1.11865e-05   1.11383e-05      10     
end_path:out1:RootOutput(write)                       1.82535       2.39739       2.95995       2.36055      0.353716        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6344.42 MB
  Peak resident set size usage (VmHWM): 4114.83 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_31_1_20251126T084453Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_31_1_20251126T084453Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_31_1_20251126T084453Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_31_1_20251126T084453Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_31_1_20251126T084453Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_31_1_20251126T084453Z_detsim_reco1.root.temp.json ====
.:
total 5599696
-rw-r--r-- 1 dune001 lbno      40702 Nov 26 11:31 jobscript.log
-rw-r--r-- 1 dune001 lbno        205 Nov 26 11:31 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno          0 Nov 26 11:31 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_31_1_20251126T084453Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno          0 Nov 26 11:31 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_31_1_20251126T084453Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno         31 Nov 26 11:31 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_31_1_20251126T084453Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno         25 Nov 26 11:31 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno        519 Nov 26 11:31 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno       3574 Nov 26 11:31 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno     204800 Nov 26 11:31 mem.db
-rw-r--r-- 1 dune001 lbno      36864 Nov 26 11:31 time.db
-rw-r--r-- 1 dune001 lbno 5685409209 Nov 26 11:31 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_31_1_20251126T084453Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno   48366470 Nov 26 11:30 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno          0 Nov 26 11:05 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno          0 Nov 26 11:02 debugprod.log
-rw-r--r-- 1 dune001 lbno        519 Nov 26 11:02 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 07:28:04 UTC       justIN version: 01.05.03