justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 252713.0@dunegpschedd02.fnal.gov

Jobsub ID252713.0@dunegpschedd02.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 05:43:12
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce03
Last heartbeat2025-11-26 07:22:20
From worker nodeHostnameccwcondor0045
cpuinfoAMD EPYC 9334 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 05:44:21
Input fileshd-protodune:pdhd_prod_beam__238490_29_1_20251106T092818Z_gen_g4_IonScintPDExt.root_255586_45_1_20251123T024429Z_PDInt.root
JobscriptExit code0
Real time1h (5850s)
CPU time1h (5326s = 91%)
Max RSS bytes6478675968 (6178 MiB)
Outputting started2025-11-26 07:21:53
Output files
Finished2025-11-26 07:22:20
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

=1321159 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 12592 XUs and 10404 XVs -> 78 XUVs
C:0 T:1 49875 XUs and 46433 XVs -> 1377 XUVs
C:0 T:2 24636 XUs and 20768 XVs -> 758 XUVs
C:0 T:4 2 XUs and 7 XVs -> 0 XUVs
C:0 T:5 26482 XUs and 15268 XVs -> 405 XUVs
C:0 T:6 43896 XUs and 41306 XVs -> 1405 XUVs
C:0 T:7 8 XUs and 9 XVs -> 0 XUVs
4023 XUVs total
2322 collection wire objects
4023 potential space points
Neighbour search...
92679 tests to find 48068 neighbours
Iterating with no regularization...
Begin: 4.16255e+07
0 4.01003e+07
1 4.00204e+07
2 4.00147e+07
Now with regularization...
Begin: 3.80438e+07
0 3.80409e+07
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
26-Nov-2025 08:21:47 CET  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252713_0_1_20251126T054428Z_detsim_reco1.root"
26-Nov-2025 08:21:47 CET  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            96.1232       125.581       156.867       125.412       17.9123        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00115573     0.0106897     0.0256685     0.0109762    0.00632245       10     
reco:opdec:Deconvolution                             0.748415      0.877625      0.988816      0.864402      0.0671305       10     
reco:ophitspe:OpHitFinderDeco                        0.175899      0.195988      0.209891      0.196191     0.00930384       10     
reco:opflash:OpFlashFinder                           0.0689488     0.0992984     0.127691      0.0961259     0.0162384       10     
reco:opslicer:OpSlicer                               0.187774      0.259735      0.305332      0.270199      0.037458        10     
reco:wclsdatahd:WireCellToolkit                       44.7406       46.2521       51.5556       45.6315       1.9141         10     
reco:gaushit:GausHitFinder                           0.693779      0.903799       1.20055      0.850991      0.151268        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000205351   0.000331332   0.000580831   0.000299484   0.000107568      10     
reco:reco3d:SpacePointSolver                          0.52627       1.05849       1.8627       0.975225      0.355451        10     
reco:hitpdune:DisambigFromSpacePoints                0.371323      0.731682       1.04996      0.733455      0.212231        10     
reco:pandora:StandardPandora                          17.3856       23.8291        43.23        21.8798       7.26546        10     
reco:pandoraWriter:StandardPandora                   0.247906      0.326147      0.376523       0.32979      0.037144        10     
reco:pandoraTrack:LArPandoraTrackCreation             1.22852       2.30363       3.35337       2.32268      0.685868        10     
reco:pandoraShower:LArPandoraShowerCreation          0.867225       2.3875        4.10904       2.25376       1.09941        10     
reco:pandoracalo:Calorimetry                         0.584203       1.12993        1.637        1.10619      0.352478        10     
reco:pandoracalonosce:Calorimetry                    0.629091       1.16768       1.87476       1.12826      0.389029        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.944046       2.37285       4.49975       2.23851       1.16814        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.862284       2.14601       3.68778       1.88686      0.947856        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0440857     0.0701132     0.101807      0.0660423     0.0162376       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0429754     0.0677144     0.099246      0.0654034     0.0154777       10     
reco:emtrkmichelid:EmTrackMichelId                    14.3739       20.2066       23.7699       20.3644       2.88885        10     
reco:crttag:SingleCRTMatchingProducer               0.00477133    0.00786717     0.0134324    0.00734279    0.00255619       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00374205    0.00644479     0.0136803    0.00573865    0.00270988       10     
reco:pandora2Track:LArPandoraTrackCreation            2.41562       5.64672       9.01292       5.74057       2.06104        10     
reco:pandora2calo:ShowerCalorimetry                  0.719107       1.94981       4.0653        1.66458      0.999265        10     
reco:pandora2calonosce:ShowerCalorimetry             0.678995       1.82869       3.42399       1.68636      0.819561        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0423265     0.0673254     0.0870944     0.066332      0.0138788       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0423017     0.0648698     0.0849573     0.0629252     0.0134579       10     
reco:pandora2Shower:LArPandoraShowerCreation          1.01965       2.24037       3.68599       2.11813       0.85928        10     
reco:pandora2Showercalo:ShowerCalorimetry             1.05627       2.4079        3.89469       2.29169      0.908551        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.04904       2.40252       3.94635       2.17678      0.962183        10     
[art]:TriggerResults:TriggerResultInserter          2.7182e-05    4.98567e-05   0.00013819    3.5033e-05    3.33309e-05      10     
end_path:out1:RootOutput                             7.251e-06    1.12893e-05   3.1578e-05     9.259e-06    6.85934e-06      10     
end_path:out1:RootOutput(write)                       1.83049       2.38445       2.71787       2.40625      0.271204        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6831.55 MB
  Peak resident set size usage (VmHWM): 4260.56 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252713_0_1_20251126T054428Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252713_0_1_20251126T054428Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252713_0_1_20251126T054428Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252713_0_1_20251126T054428Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252713_0_1_20251126T054428Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252713_0_1_20251126T054428Z_detsim_reco1.root.temp.json ====
.:
total 5747724
-rw-r--r-- 1 dune001 lbno      40872 Nov 26 08:21 jobscript.log
-rw-r--r-- 1 dune001 lbno        203 Nov 26 08:21 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno          0 Nov 26 08:21 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252713_0_1_20251126T054428Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno          0 Nov 26 08:21 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252713_0_1_20251126T054428Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno         31 Nov 26 08:21 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252713_0_1_20251126T054428Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno         25 Nov 26 08:21 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno        519 Nov 26 08:21 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno       3574 Nov 26 08:21 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno     204800 Nov 26 08:21 mem.db
-rw-r--r-- 1 dune001 lbno      36864 Nov 26 08:21 time.db
-rw-r--r-- 1 dune001 lbno 5832352523 Nov 26 08:21 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252713_0_1_20251126T054428Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno   53006050 Nov 26 08:20 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno          0 Nov 26 08:02 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno          0 Nov 26 07:59 debugprod.log
-rw-r--r-- 1 dune001 lbno        519 Nov 26 07:59 detsim_single_protodunehd_hist.root
justIN time: 2025-12-20 01:16:58 UTC       justIN version: 01.05.03