justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 252827.7@dunegpschedd02.fnal.gov

Jobsub ID252827.7@dunegpschedd02.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 08:47:16
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce04
Last heartbeat2025-11-26 12:43:15
From worker nodeHostnamewn123
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 08:48:16
Input fileshd-protodune:pdhd_prod_beam__238300_35_1_20251106T064103Z_gen_g4_IonScintPDExt.root_255164_14_1_20251122T001751Z_PDInt.root
JobscriptExit code0
Real time3h (13835s)
CPU time2h (9054s = 65%)
Max RSS bytes4502024192 (4293 MiB)
Outputting started2025-11-26 12:38:53
Output files
Finished2025-11-26 12:43:15
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0 XUs and 94908 XVs -> 4479 XUVs
C:0 T:2 17960 XUs and 24436 XVs -> 690 XUVs
C:0 T:3 7 XUs and 9 XVs -> 0 XUVs
C:0 T:5 57523 XUs and 24693 XVs -> 918 XUVs
C:0 T:6 13696 XUs and 19824 XVs -> 971 XUVs
C:0 T:7 3 XUs and 12 XVs -> 0 XUVs
7216 XUVs total
2919 collection wire objects
7216 potential space points
Neighbour search...
207668 tests to find 112100 neighbours
Iterating with no regularization...
Begin: 7.00906e+07
0 5.80281e+07
1 5.78774e+07
2 5.78645e+07
Now with regularization...
Begin: 5.60032e+07
0 5.59995e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 09:38:41 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252827_7_1_20251126T084823Z_detsim_reco1.root"
26-Nov-2025 09:38:41 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            140.714       208.683       282.867       208.02        40.8694        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00178274    0.00776225     0.0156976    0.00668221    0.00476296       10     
reco:opdec:Deconvolution                              1.18357       1.37077       1.72501       1.35799      0.150842        10     
reco:ophitspe:OpHitFinderDeco                        0.283951      0.345968      0.416741      0.343948      0.0306333       10     
reco:opflash:OpFlashFinder                           0.0836629     0.125905      0.164966      0.126633      0.0248502       10     
reco:opslicer:OpSlicer                               0.352512      0.432963      0.562377      0.419447      0.0544855       10     
reco:wclsdatahd:WireCellToolkit                       70.0045       72.8895       74.9766       72.6993       1.52033        10     
reco:gaushit:GausHitFinder                           0.754948       1.38236       1.92189       1.4099       0.361354        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.00041534    0.000733633    0.0014512     0.0006123    0.000358432      10     
reco:reco3d:SpacePointSolver                         0.413395       2.22481       4.33641       2.54805       1.08207        10     
reco:hitpdune:DisambigFromSpacePoints                0.374879       1.21193       2.26996       1.23744      0.581715        10     
reco:pandora:StandardPandora                          14.9569       34.7025       64.7448       35.7033       15.1013        10     
reco:pandoraWriter:StandardPandora                   0.329281      0.586868      0.889333      0.565837       0.16466        10     
reco:pandoraTrack:LArPandoraTrackCreation            0.852937       3.13577       4.87974       3.23598       1.18327        10     
reco:pandoraShower:LArPandoraShowerCreation          0.918241       3.1137        6.3086        2.92056       1.42411        10     
reco:pandoracalo:Calorimetry                         0.526129       1.61896       3.47624       1.51518      0.752696        10     
reco:pandoracalonosce:Calorimetry                    0.503135       1.70558       3.70535       1.53835      0.844896        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.851308       3.0143        5.50984       2.88461       1.34236        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.823514       2.90302       5.77256       2.82178       1.34678        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0718885     0.102569      0.139073      0.104446      0.0202844       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0681544     0.102798      0.135031      0.103741      0.0208456       10     
reco:emtrkmichelid:EmTrackMichelId                    34.0355       49.122        62.6499       48.1368       8.49235        10     
reco:crttag:SingleCRTMatchingProducer               0.00619864    0.00908356     0.0162031    0.00788272    0.00282094       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00420967    0.00742311     0.0125787    0.00702467    0.00218315       10     
reco:pandora2Track:LArPandoraTrackCreation            2.30254       8.06701       13.7204       7.47079       3.62375        10     
reco:pandora2calo:ShowerCalorimetry                   0.74968       2.63927       5.04715       2.37686       1.25788        10     
reco:pandora2calonosce:ShowerCalorimetry             0.777009       2.63556       4.91917       2.44805       1.18987        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0540332     0.0966565     0.144323      0.0957942     0.0278628       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0545219     0.0958166     0.138786      0.0975412     0.0287031       10     
reco:pandora2Shower:LArPandoraShowerCreation          1.04814       3.28126       5.88308       3.14825       1.39496        10     
reco:pandora2Showercalo:ShowerCalorimetry             1.1095        3.47853       7.09402       3.33306       1.6897         10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.03363       3.34077       7.16231       3.39754       1.59722        10     
[art]:TriggerResults:TriggerResultInserter          3.2021e-05    5.02052e-05   0.000109435   4.1647e-05    2.21765e-05      10     
end_path:out1:RootOutput                             4.801e-06    9.0764e-06    2.9462e-05     6.53e-06     7.01126e-06      10     
end_path:out1:RootOutput(write)                       2.95376       4.74684       6.36108       4.87577      0.948493        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6158.51 MB
  Peak resident set size usage (VmHWM): 3966.99 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252827_7_1_20251126T084823Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252827_7_1_20251126T084823Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252827_7_1_20251126T084823Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252827_7_1_20251126T084823Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252827_7_1_20251126T084823Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252827_7_1_20251126T084823Z_detsim_reco1.root.temp.json ====
.:
total 5920880
-rw-r--r-- 1 nobody nobody      47771 Nov 26 09:38 jobscript.log
-rw-r--r-- 1 nobody nobody        203 Nov 26 09:38 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 09:38 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252827_7_1_20251126T084823Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 09:38 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252827_7_1_20251126T084823Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 09:38 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252827_7_1_20251126T084823Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 09:38 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 09:38 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 09:38 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     204800 Nov 26 09:38 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 09:38 time.db
-rw-r--r-- 1 nobody nobody 6013178780 Nov 26 09:38 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252827_7_1_20251126T084823Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   49480660 Nov 26 09:36 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 09:04 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 09:01 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 09:00 detsim_single_protodunehd_hist.root
justIN time: 2026-02-04 07:21:32 UTC       justIN version: 01.06.00