justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258765.3@dunegpschedd01.fnal.gov

Jobsub ID258765.3@dunegpschedd01.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 13:22:29
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce03
Last heartbeat2025-11-26 16:16:29
From worker nodeHostnamewn75
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 13:23:26
Input fileshd-protodune:pdhd_prod_beam__238336_124_1_20251106T071957Z_gen_g4_IonScintPDExt.root_255543_76_1_20251122T234032Z_PDInt.root
JobscriptExit code0
Real time2h (10121s)
CPU time2h (8647s = 85%)
Max RSS bytes4502990848 (4294 MiB)
Outputting started2025-11-26 16:12:08
Output files
Finished2025-11-26 16:16:29
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

: saving 121676 traces tagged "wiener"
FrameSaver: q=3.89708e+06 n=1451248 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 12232 XUs and 10365 XVs -> 100 XUVs
C:0 T:1 60189 XUs and 51578 XVs -> 1478 XUVs
C:0 T:2 39342 XUs and 44551 XVs -> 1362 XUVs
C:0 T:3 15 XUs and 20 XVs -> 0 XUVs
C:0 T:4 40 XUs and 43 XVs -> 1 XUVs
C:0 T:5 90756 XUs and 96009 XVs -> 5834 XUVs
C:0 T:6 20998 XUs and 25634 XVs -> 682 XUVs
C:0 T:7 22 XUs and 23 XVs -> 0 XUVs
9457 XUVs total
3590 collection wire objects
9457 potential space points
Neighbour search...
315193 tests to find 187008 neighbours
Iterating with no regularization...
Begin: 4.8379e+07
0 4.42523e+07
1 4.39179e+07
2 4.38825e+07
Now with regularization...
Begin: 4.07166e+07
0 4.07007e+07
26-Nov-2025 13:11:55 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_3_1_20251126T132333Z_detsim_reco1.root"
26-Nov-2025 13:11:55 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            158.562       201.248       291.457       191.703       35.7437        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00140189    0.00892547     0.0150977    0.00890085    0.00434252       10     
reco:opdec:Deconvolution                              1.03605       1.34013       1.67461       1.30934      0.189995        10     
reco:ophitspe:OpHitFinderDeco                        0.178826      0.318465      0.353605      0.328506      0.0479777       10     
reco:opflash:OpFlashFinder                           0.100813      0.129731      0.206134      0.121873      0.0280216       10     
reco:opslicer:OpSlicer                               0.303224      0.428152      0.601109       0.43617      0.0987589       10     
reco:wclsdatahd:WireCellToolkit                       68.7164       70.0841       73.0796       69.7204       1.14175        10     
reco:gaushit:GausHitFinder                           0.929213       1.35864       1.66433       1.33802      0.226336        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000367494   0.000461432   0.000629154   0.000440127   8.56294e-05      10     
reco:reco3d:SpacePointSolver                         0.770479        1.601        2.69129       1.51481      0.609055        10     
reco:hitpdune:DisambigFromSpacePoints                0.818235       1.16675       2.41438       1.04612      0.447722        10     
reco:pandora:StandardPandora                          16.3751       36.5622       56.3043       33.9617       11.4568        10     
reco:pandoraWriter:StandardPandora                   0.382035      0.697539       1.11428      0.676267      0.245588        10     
reco:pandoraTrack:LArPandoraTrackCreation             1.27022       2.99805       5.67024       2.71382       1.19886        10     
reco:pandoraShower:LArPandoraShowerCreation           1.19717       2.70282       6.08709       2.4066        1.30952        10     
reco:pandoracalo:Calorimetry                         0.805311       1.51466       2.72872       1.39025      0.560793        10     
reco:pandoracalonosce:Calorimetry                    0.773392       1.48192       3.11802       1.32862      0.663069        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.896787        2.512        5.81759       2.25911       1.26108        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.825968       2.36068       5.59931       2.19572       1.20525        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0424472     0.0987551     0.146317      0.105459      0.0294026       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0490656     0.100216      0.142411      0.106782      0.0263136       10     
reco:emtrkmichelid:EmTrackMichelId                    36.6788       48.1911       66.7706       46.3843       7.88149        10     
reco:crttag:SingleCRTMatchingProducer                0.006214     0.00844064     0.0138071    0.00806456    0.00202391       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00565124    0.00727575     0.0118792    0.00678098    0.00164701       10     
reco:pandora2Track:LArPandoraTrackCreation            3.57198       6.6405        16.5881       5.89859       3.51051        10     
reco:pandora2calo:ShowerCalorimetry                  0.924253       2.29467       5.88232       2.11283       1.34016        10     
reco:pandora2calonosce:ShowerCalorimetry              1.06718       2.28162       5.85358       1.94038       1.28548        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0508998     0.100759      0.152887      0.100766      0.0260496       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0532575     0.0987952      0.15287      0.101135      0.0263964       10     
reco:pandora2Shower:LArPandoraShowerCreation          1.52811       2.88254       7.15328       2.35842       1.55601        10     
reco:pandora2Showercalo:ShowerCalorimetry             1.43389       3.07945       7.9881        2.40599       1.76855        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.18241       3.02142       7.63326       2.38159       1.67565        10     
[art]:TriggerResults:TriggerResultInserter          2.0991e-05    4.31969e-05   0.000116694   3.16465e-05   2.79131e-05      10     
end_path:out1:RootOutput                             4.24e-06     8.0861e-06    3.0581e-05     6.04e-06     7.54161e-06      10     
end_path:out1:RootOutput(write)                       3.35469       5.02369       7.22463       4.96791      0.992117        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6078.87 MB
  Peak resident set size usage (VmHWM): 3899.74 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_3_1_20251126T132333Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_3_1_20251126T132333Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_3_1_20251126T132333Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_3_1_20251126T132333Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_3_1_20251126T132333Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_3_1_20251126T132333Z_detsim_reco1.root.temp.json ====
.:
total 5763152
-rw-r--r-- 1 nobody nobody      40801 Nov 26 13:12 jobscript.log
-rw-r--r-- 1 nobody nobody        204 Nov 26 13:12 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 13:12 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_3_1_20251126T132333Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 13:12 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_3_1_20251126T132333Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 13:12 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_3_1_20251126T132333Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 13:11 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 13:11 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 13:11 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     204800 Nov 26 13:11 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 13:11 time.db
-rw-r--r-- 1 nobody nobody 5851085059 Nov 26 13:11 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_3_1_20251126T132333Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   50071030 Nov 26 13:09 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 12:39 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 12:36 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 12:35 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 07:15:59 UTC       justIN version: 01.05.03