justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 252825.3@dunegpschedd02.fnal.gov

Jobsub ID252825.3@dunegpschedd02.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 08:43:20
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce02
Last heartbeat2025-11-26 11:05:47
From worker nodeHostnameccwcondor0583
cpuinfoAMD EPYC 7453 28-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 08:44:44
Input fileshd-protodune:pdhd_prod_beam__238253_125_1_20251106T055122Z_gen_g4_IonScintPDExt.root_255554_176_1_20251123T001328Z_PDInt.root
JobscriptExit code0
Real time2h (7832s)
CPU time1h (7073s = 90%)
Max RSS bytes6691811328 (6381 MiB)
Outputting started2025-11-26 10:55:18
Output files
Finished2025-11-26 11:05:47
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

d channels
Finding XUV coincidences...
C:0 T:0 8360 XUs and 7966 XVs -> 46 XUVs
C:0 T:1 58099 XUs and 36357 XVs -> 866 XUVs
C:0 T:2 42319 XUs and 66518 XVs -> 960 XUVs
C:0 T:3 20 XUs and 190 XVs -> 2 XUVs
C:0 T:5 16958 XUs and 20146 XVs -> 461 XUVs
C:0 T:6 40157 XUs and 46034 XVs -> 3754 XUVs
C:0 T:7 0 XUs and 4 XVs -> 0 XUVs
6089 XUVs total
1924 collection wire objects
6089 potential space points
Neighbour search...
328767 tests to find 206824 neighbours
Iterating with no regularization...
Begin: 5.23348e+07
0 4.92262e+07
1 4.90872e+07
2 4.90549e+07
Now with regularization...
Begin: 4.74667e+07
0 4.74512e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 11:55:08 CET  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_3_1_20251126T084453Z_detsim_reco1.root"
26-Nov-2025 11:55:08 CET  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            167.964       220.712       293.869       211.383       39.2384        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00130532    0.00779486     0.0114631    0.00793815    0.00328294       10     
reco:opdec:Deconvolution                             0.945478       1.06219       1.26615       1.03705      0.0947754       10     
reco:ophitspe:OpHitFinderDeco                         0.23327       0.24551      0.265644      0.241054      0.0113525       10     
reco:opflash:OpFlashFinder                           0.0775505     0.108607      0.157106      0.110725      0.0207106       10     
reco:opslicer:OpSlicer                               0.278062      0.354733      0.488244      0.357625      0.0587904       10     
reco:wclsdatahd:WireCellToolkit                       50.7634       52.3658       54.3936       52.1509       1.23359        10     
reco:gaushit:GausHitFinder                            1.05625       1.30681       1.75341       1.25281      0.205449        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000376092   0.000485428   0.000766423   0.000477117   0.000104415      10     
reco:reco3d:SpacePointSolver                         0.860597       1.7846         2.903        1.52977      0.684845        10     
reco:hitpdune:DisambigFromSpacePoints                0.698555       1.25263       1.96669       1.14728      0.373695        10     
reco:pandora:StandardPandora                          25.0892       51.8635       100.506       39.1103       25.8359        10     
reco:pandoraWriter:StandardPandora                   0.347647      0.427406      0.524677      0.419841      0.0544615       10     
reco:pandoraTrack:LArPandoraTrackCreation             3.24304       5.29818       7.0032        5.52348       1.12088        10     
reco:pandoraShower:LArPandoraShowerCreation           3.68784       5.68262       7.65705       5.38841       1.26687        10     
reco:pandoracalo:Calorimetry                          1.70102       2.87766       4.01231       2.80901      0.770728        10     
reco:pandoracalonosce:Calorimetry                     1.67139       2.84709       3.78561       2.71559      0.655022        10     
reco:pandoraShowercalo:ShowerCalorimetry              2.94852       5.84198       8.85276       5.8031        1.98063        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.86598       5.11056       8.56723       5.02817       1.91135        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.076417      0.0939118     0.112514      0.0931119     0.0107791       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0712827     0.0907087     0.108885      0.0905117     0.0111792       10     
reco:emtrkmichelid:EmTrackMichelId                    35.7893       40.9489       48.6903       40.5504       3.98105        10     
reco:crttag:SingleCRTMatchingProducer                0.0102363     0.0127442     0.0153977     0.0125352    0.00182725       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00808698     0.0129121     0.0266822     0.010982     0.00506631       10     
reco:pandora2Track:LArPandoraTrackCreation            8.09104       12.2738       16.2118       12.0294       2.77677        10     
reco:pandora2calo:ShowerCalorimetry                   2.30491       4.4849        6.33997       4.46318       1.40357        10     
reco:pandora2calonosce:ShowerCalorimetry              1.95442       4.4005        6.91249       4.19826       1.56662        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0689496     0.0876138     0.115619      0.0840227     0.0139114       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0632847     0.0854679     0.114123      0.0850094     0.0144959       10     
reco:pandora2Shower:LArPandoraShowerCreation          3.6397        5.63781       8.49218       5.14352       1.41954        10     
reco:pandora2Showercalo:ShowerCalorimetry             3.4215        5.80801       9.46149       5.2631        2.1822         10     
reco:pandora2Showercalonosce:ShowerCalorimetry        3.40462       5.24272       8.47885       4.75913       1.62779        10     
[art]:TriggerResults:TriggerResultInserter          3.3593e-05    6.38353e-05   0.000197459    4.805e-05    4.58716e-05      10     
end_path:out1:RootOutput                             8.456e-06    1.4458e-05    5.4031e-05     9.603e-06    1.3269e-05       10     
end_path:out1:RootOutput(write)                       2.45346       2.86897       3.41576       2.82049       0.32049        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6263.51 MB
  Peak resident set size usage (VmHWM): 4021.05 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_3_1_20251126T084453Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_3_1_20251126T084453Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_3_1_20251126T084453Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_3_1_20251126T084453Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_3_1_20251126T084453Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_3_1_20251126T084453Z_detsim_reco1.root.temp.json ====
.:
total 6402284
-rw-r--r-- 1 dune001 lbno      53782 Nov 26 11:55 jobscript.log
-rw-r--r-- 1 dune001 lbno        205 Nov 26 11:55 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno          0 Nov 26 11:55 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_3_1_20251126T084453Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno          0 Nov 26 11:55 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_3_1_20251126T084453Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno         31 Nov 26 11:55 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_3_1_20251126T084453Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno         25 Nov 26 11:55 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno        519 Nov 26 11:55 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno       3574 Nov 26 11:55 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno     204800 Nov 26 11:55 mem.db
-rw-r--r-- 1 dune001 lbno      36864 Nov 26 11:55 time.db
-rw-r--r-- 1 dune001 lbno 6496659463 Nov 26 11:55 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252825_3_1_20251126T084453Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno   58953530 Nov 26 11:53 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno          0 Nov 26 11:21 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno          0 Nov 26 11:16 debugprod.log
-rw-r--r-- 1 dune001 lbno        519 Nov 26 11:16 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 07:14:19 UTC       justIN version: 01.05.03