justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 252752.19@dunegpschedd02.fnal.gov

Jobsub ID252752.19@dunegpschedd02.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 06:49:10
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce03
Last heartbeat2025-11-26 08:42:17
From worker nodeHostnameccwcondor0039
cpuinfoAMD EPYC 9334 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 06:52:29
Input fileshd-protodune:pdhd_prod_beam__238093_129_1_20251106T000319Z_gen_g4_IonScintPDExt.root_255824_162_1_20251123T215004Z_PDInt.root
JobscriptExit code0
Real time1h (6546s)
CPU time1h (6200s = 94%)
Max RSS bytes8287653888 (7903 MiB)
Outputting started2025-11-26 08:41:37
Output files
Finished2025-11-26 08:42:17
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 12838 XUs and 9260 XVs -> 237 XUVs
C:0 T:1 133350 XUs and 90084 XVs -> 5161 XUVs
C:0 T:2 13051 XUs and 12277 XVs -> 976 XUVs
C:0 T:3 4 XUs and 11 XVs -> 3 XUVs
C:0 T:4 14 XUs and 18 XVs -> 0 XUVs
C:0 T:5 29654 XUs and 23023 XVs -> 790 XUVs
C:0 T:6 18788 XUs and 25794 XVs -> 651 XUVs
7818 XUVs total
2878 collection wire objects
7818 potential space points
Neighbour search...
332338 tests to find 139568 neighbours
Iterating with no regularization...
Begin: 3.24382e+07
0 2.93631e+07
1 2.91165e+07
2 2.90915e+07
Now with regularization...
Begin: 2.74127e+07
0 2.74062e+07
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
26-Nov-2025 09:41:28 CET  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252752_19_1_20251126T065236Z_detsim_reco1.root"
26-Nov-2025 09:41:28 CET  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            124.394       160.706       276.613       149.013       42.0321        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00117845     0.0109166     0.026433      0.0107872    0.00712733       10     
reco:opdec:Deconvolution                             0.790882       1.09216       1.39911       1.11618      0.168984        10     
reco:ophitspe:OpHitFinderDeco                         0.2052       0.239616      0.271631      0.242777      0.0234192       10     
reco:opflash:OpFlashFinder                           0.0867223     0.124792      0.236172       0.11721      0.040164        10     
reco:opslicer:OpSlicer                               0.171933      0.343396      0.565185      0.323687      0.0941358       10     
reco:wclsdatahd:WireCellToolkit                       49.7921       54.0974       61.2598       53.2304       3.22516        10     
reco:gaushit:GausHitFinder                            0.7985        1.01516       1.38867      0.995705      0.159302        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000272503   0.000434283   0.000637136   0.000407407   0.000116134      10     
reco:reco3d:SpacePointSolver                         0.650331       1.63275       3.76774       1.57333       0.84494        10     
reco:hitpdune:DisambigFromSpacePoints                0.637726      0.929118       1.94512      0.772781      0.366191        10     
reco:pandora:StandardPandora                          14.1866       30.033        65.0132       27.2003       13.4662        10     
reco:pandoraWriter:StandardPandora                   0.228518      0.387589      0.598827      0.377973      0.0859011       10     
reco:pandoraTrack:LArPandoraTrackCreation             1.71881       3.66486       8.32302       3.13513       1.78478        10     
reco:pandoraShower:LArPandoraShowerCreation           2.00218       3.77604       10.044        3.00131       2.29232        10     
reco:pandoracalo:Calorimetry                         0.973562       1.9104        4.17099       1.57121       0.88006        10     
reco:pandoracalonosce:Calorimetry                    0.879091       1.64261       3.52844       1.48246       0.74759        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.86752       3.44224       9.50399       2.82198       2.16381        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.61569       3.0948        7.84994       2.43817       1.74386        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0676016     0.0841971     0.118315      0.0848226     0.0140163       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0611395     0.0805979     0.112982      0.0801205      0.01387        10     
reco:emtrkmichelid:EmTrackMichelId                    18.9649       25.1177       33.2551       23.6839       4.17454        10     
reco:crttag:SingleCRTMatchingProducer                0.0075551     0.0105439     0.0170061    0.00958395     0.0030408       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00527316    0.00812678     0.0166194    0.00690282    0.00344815       10     
reco:pandora2Track:LArPandoraTrackCreation            4.36271       8.76129       18.8867       7.36889       4.0596         10     
reco:pandora2calo:ShowerCalorimetry                   1.18978       2.93247       7.84003       2.25207       1.82854        10     
reco:pandora2calonosce:ShowerCalorimetry              1.12079       2.75756       7.31927       2.07702       1.73135        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0595162     0.0978048     0.207708      0.0880321     0.0403879       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.058861      0.0778903     0.102661      0.0787411     0.0127668       10     
reco:pandora2Shower:LArPandoraShowerCreation          1.3567        3.58211       8.84398       2.99234       1.99077        10     
reco:pandora2Showercalo:ShowerCalorimetry             1.45839       3.39268       8.44305        2.649        1.9462         10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.37137       3.20045       7.2123        2.5472        1.74508        10     
[art]:TriggerResults:TriggerResultInserter          4.1423e-05    6.70176e-05   0.000106983   5.7212e-05    2.32489e-05      10     
end_path:out1:RootOutput                             8.974e-06    1.69065e-05   4.7752e-05    1.1808e-05    1.13798e-05      10     
end_path:out1:RootOutput(write)                       2.27614       2.92209       4.16889       2.7925       0.475223        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6949.1 MB
  Peak resident set size usage (VmHWM): 4517.87 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252752_19_1_20251126T065236Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252752_19_1_20251126T065236Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252752_19_1_20251126T065236Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252752_19_1_20251126T065236Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252752_19_1_20251126T065236Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252752_19_1_20251126T065236Z_detsim_reco1.root.temp.json ====
.:
total 6559700
-rw-r--r-- 1 dune001 lbno      40672 Nov 26 09:41 jobscript.log
-rw-r--r-- 1 dune001 lbno        177 Nov 26 09:41 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno          0 Nov 26 09:41 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252752_19_1_20251126T065236Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno          0 Nov 26 09:41 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252752_19_1_20251126T065236Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno         31 Nov 26 09:41 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252752_19_1_20251126T065236Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno         25 Nov 26 09:41 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno        519 Nov 26 09:41 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno       3574 Nov 26 09:41 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno     200704 Nov 26 09:41 mem.db
-rw-r--r-- 1 dune001 lbno      36864 Nov 26 09:41 time.db
-rw-r--r-- 1 dune001 lbno 6659167617 Nov 26 09:41 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252752_19_1_20251126T065236Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno   57655310 Nov 26 09:40 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno          0 Nov 26 09:15 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno          0 Nov 26 09:13 debugprod.log
-rw-r--r-- 1 dune001 lbno        519 Nov 26 09:12 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 23:17:59 UTC       justIN version: 01.05.03