justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 252844.13@dunegpschedd02.fnal.gov

Jobsub ID252844.13@dunegpschedd02.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 09:29:18
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce03
Last heartbeat2025-11-26 11:39:21
From worker nodeHostnameccwcondor0551
cpuinfoAMD EPYC 7453 28-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 09:30:03
Input fileshd-protodune:pdhd_prod_beam__238266_8_1_20251106T055437Z_gen_g4_IonScintPDExt.root_255828_165_1_20251123T220843Z_PDInt.root
JobscriptExit code0
Real time2h (7711s)
CPU time1h (7126s = 92%)
Max RSS bytes8006164480 (7635 MiB)
Outputting started2025-11-26 11:38:35
Output files
Finished2025-11-26 11:39:21
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

and 12204 XVs -> 148 XUVs
C:0 T:1 75307 XUs and 83206 XVs -> 2547 XUVs
C:0 T:2 24902 XUs and 14159 XVs -> 201 XUVs
C:0 T:4 16 XUs and 19 XVs -> 1 XUVs
C:0 T:5 26555 XUs and 29592 XVs -> 634 XUVs
C:0 T:6 52783 XUs and 21847 XVs -> 513 XUVs
C:0 T:7 8 XUs and 5 XVs -> 0 XUVs
4044 XUVs total
2046 collection wire objects
4044 potential space points
Neighbour search...
108930 tests to find 46930 neighbours
Iterating with no regularization...
Begin: 3.07087e+07
0 2.87063e+07
1 2.85998e+07
2 2.85925e+07
Now with regularization...
Begin: 2.75724e+07
0 2.75704e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
26-Nov-2025 12:38:26 CET  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252844_13_1_20251126T093014Z_detsim_reco1.root"
26-Nov-2025 12:38:26 CET  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            152.193       205.406       312.261       194.99        45.8061        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00126117    0.00877471     0.0170814     0.0090337     0.004982        10     
reco:opdec:Deconvolution                             0.913898       1.08809       1.40728       1.04837      0.137572        10     
reco:ophitspe:OpHitFinderDeco                         0.22974      0.248655      0.272234       0.24895      0.0112986       10     
reco:opflash:OpFlashFinder                           0.0838003     0.125566      0.176992      0.125739      0.0264488       10     
reco:opslicer:OpSlicer                               0.287801      0.369675      0.568354      0.349591      0.0770579       10     
reco:wclsdatahd:WireCellToolkit                       48.887        55.1358       61.112        55.4698       3.75686        10     
reco:gaushit:GausHitFinder                           0.597647       1.21982       1.92028       1.17276      0.326589        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000323856   0.00050682    0.000755292   0.000460671   0.000131319      10     
reco:reco3d:SpacePointSolver                         0.822664       1.4964        2.23213       1.48473      0.430574        10     
reco:hitpdune:DisambigFromSpacePoints                0.744352       1.02433       1.46375      0.992651      0.206629        10     
reco:pandora:StandardPandora                          22.979        34.5199       53.8912       33.5038       9.11273        10     
reco:pandoraWriter:StandardPandora                    0.2866        0.41321      0.543755      0.414088      0.0738326       10     
reco:pandoraTrack:LArPandoraTrackCreation             2.67658       4.82725       8.60465       4.9035        1.57143        10     
reco:pandoraShower:LArPandoraShowerCreation           3.35499       5.81721       10.6317       5.33127       2.13357        10     
reco:pandoracalo:Calorimetry                          1.45552       2.64119       5.44766       2.42305       1.17582        10     
reco:pandoracalonosce:Calorimetry                     1.46543       2.5186        4.95722       2.26294      0.993885        10     
reco:pandoraShowercalo:ShowerCalorimetry              2.79803       5.46509       12.2202       4.75766       2.7536         10     
reco:pandoraShowercalonosce:ShowerCalorimetry         2.42628       5.11091       12.0724       4.23499       2.75303        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0701647     0.095405      0.135254      0.0970047     0.0183701       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0688846     0.0931052     0.136446      0.0902965     0.0192997       10     
reco:emtrkmichelid:EmTrackMichelId                    31.1009       41.8928       54.1322       39.3187       7.39852        10     
reco:crttag:SingleCRTMatchingProducer               0.00938238     0.0130006     0.0170706     0.0117944    0.00263962       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00845869     0.0119898     0.017575      0.0115894    0.00287592       10     
reco:pandora2Track:LArPandoraTrackCreation            7.17444       11.8741       24.3844       10.1819       4.83058        10     
reco:pandora2calo:ShowerCalorimetry                   2.30846       4.79782       11.2027       3.83176       2.58238        10     
reco:pandora2calonosce:ShowerCalorimetry              2.31839       4.54924       9.86768       3.69835       2.34804        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0740907     0.0958039     0.155733      0.087458      0.0233976       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0671611     0.085591      0.116049      0.0817111     0.0132308       10     
reco:pandora2Shower:LArPandoraShowerCreation          3.2406        5.46615       10.8801       5.02909       2.13889        10     
reco:pandora2Showercalo:ShowerCalorimetry             3.18181       5.78426       12.475        5.24911       2.64602        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        3.04393       5.46579       11.5402       4.68013       2.55001        10     
[art]:TriggerResults:TriggerResultInserter          4.5304e-05    7.34552e-05   0.00016601    6.7922e-05    3.26029e-05      10     
end_path:out1:RootOutput                             7.794e-06    1.52766e-05    4.252e-05    1.15115e-05   9.61291e-06      10     
end_path:out1:RootOutput(write)                       2.55001       2.89186       3.98788       2.76307       0.41481        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6529.21 MB
  Peak resident set size usage (VmHWM): 4303.88 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252844_13_1_20251126T093014Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252844_13_1_20251126T093014Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252844_13_1_20251126T093014Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252844_13_1_20251126T093014Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252844_13_1_20251126T093014Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252844_13_1_20251126T093014Z_detsim_reco1.root.temp.json ====
.:
total 6651920
-rw-r--r-- 1 dune001 lbno      47529 Nov 26 12:38 jobscript.log
-rw-r--r-- 1 dune001 lbno        204 Nov 26 12:38 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno          0 Nov 26 12:38 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252844_13_1_20251126T093014Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno          0 Nov 26 12:38 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252844_13_1_20251126T093014Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno         31 Nov 26 12:38 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252844_13_1_20251126T093014Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno         25 Nov 26 12:38 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno        519 Nov 26 12:38 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno       3574 Nov 26 12:38 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno     204800 Nov 26 12:38 mem.db
-rw-r--r-- 1 dune001 lbno      36864 Nov 26 12:38 time.db
-rw-r--r-- 1 dune001 lbno 6755153550 Nov 26 12:38 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252844_13_1_20251126T093014Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno   56093970 Nov 26 12:36 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno          0 Nov 26 12:05 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno          0 Nov 26 12:02 debugprod.log
-rw-r--r-- 1 dune001 lbno        519 Nov 26 12:01 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 07:13:01 UTC       justIN version: 01.05.03