justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258662.11@dunegpschedd01.fnal.gov

Jobsub ID258662.11@dunegpschedd01.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 11:26:24
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce03
Last heartbeat2025-11-26 15:03:29
From worker nodeHostnamewn84
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 11:27:48
Input fileshd-protodune:pdhd_prod_beam__238564_0_1_20251106T105019Z_gen_g4_IonScintPDExt.root_255530_138_1_20251122T223148Z_PDInt.root
JobscriptExit code0
Real time3h (12649s)
CPU time2h (10665s = 84%)
Max RSS bytes4515127296 (4305 MiB)
Outputting started2025-11-26 14:58:39
Output files
Finished2025-11-26 15:03:29
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

channels
Finding XUV coincidences...
C:0 T:0 11889 XUs and 10982 XVs -> 148 XUVs
C:0 T:1 80612 XUs and 80332 XVs -> 2247 XUVs
C:0 T:2 19517 XUs and 35338 XVs -> 3651 XUVs
C:0 T:3 5 XUs and 12 XVs -> 0 XUVs
C:0 T:4 11 XUs and 20 XVs -> 5 XUVs
C:0 T:5 8114 XUs and 17237 XVs -> 1547 XUVs
C:0 T:6 11609 XUs and 13996 XVs -> 937 XUVs
C:0 T:7 39 XUs and 51 XVs -> 6 XUVs
8541 XUVs total
2662 collection wire objects
8541 potential space points
Neighbour search...
362839 tests to find 216190 neighbours
Iterating with no regularization...
Begin: 2.16526e+07
0 1.91191e+07
1 1.88512e+07
2 1.88098e+07
3 1.87974e+07
Now with regularization...
Begin: 1.68042e+07
0 1.67953e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 11:58:28 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258662_11_1_20251126T112756Z_detsim_reco1.root"
26-Nov-2025 11:58:28 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            180.739       241.928       347.935       218.179       59.3583        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00183298     0.0112447     0.0224918     0.0108238    0.00660693       10     
reco:opdec:Deconvolution                              1.08442       1.47506       1.71733       1.49971      0.165344        10     
reco:ophitspe:OpHitFinderDeco                        0.209036       0.32747      0.367241      0.341079      0.0434518       10     
reco:opflash:OpFlashFinder                           0.111732      0.160463      0.216745       0.15136      0.0342006       10     
reco:opslicer:OpSlicer                               0.328138      0.454929      0.547494       0.46807      0.0751218       10     
reco:wclsdatahd:WireCellToolkit                       67.2523       69.3255       70.7751       69.9487       1.39396        10     
reco:gaushit:GausHitFinder                            1.07312       1.54956       2.37033       1.49672      0.400027        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.00039564    0.000736923   0.00119758    0.000588264   0.000293636      10     
reco:reco3d:SpacePointSolver                         0.720222       1.6182         2.847        1.63835      0.609315        10     
reco:hitpdune:DisambigFromSpacePoints                0.520554       1.37484       2.4817        1.22316      0.695962        10     
reco:pandora:StandardPandora                          21.6122       47.7863       88.6106       36.1445       23.5979        10     
reco:pandoraWriter:StandardPandora                   0.340095      0.866162       1.27122      0.894283      0.303394        10     
reco:pandoraTrack:LArPandoraTrackCreation             2.2757        4.11866       6.92493       3.20205       1.72022        10     
reco:pandoraShower:LArPandoraShowerCreation           1.91811       4.47314       8.49479       3.43213       2.36734        10     
reco:pandoracalo:Calorimetry                          1.26697       2.11784       4.06801       1.56965       1.02605        10     
reco:pandoracalonosce:Calorimetry                     1.26234       2.05425       3.73951       1.5772       0.915927        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.91625       4.30888       8.06312       3.22784       2.26142        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         2.28492       4.01963       7.6843        2.78327       2.21219        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0621592     0.112083      0.166106      0.110331      0.0326314       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0577169     0.110162      0.165206      0.112108      0.033327        10     
reco:emtrkmichelid:EmTrackMichelId                    44.5616       57.2434       75.3761       53.167        10.643         10     
reco:crttag:SingleCRTMatchingProducer               0.00738151    0.00996948     0.0132522    0.00972684    0.00189253       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00693463    0.00919709     0.0132354    0.00872646    0.00188072       10     
reco:pandora2Track:LArPandoraTrackCreation            5.80417       10.8555       21.2313       8.50652       5.05075        10     
reco:pandora2calo:ShowerCalorimetry                   1.79736       3.75979       7.53542       2.93958       1.80742        10     
reco:pandora2calonosce:ShowerCalorimetry              1.45743       3.58722       7.35881       2.67058       1.89024        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0833091      0.11913       0.16787      0.113277      0.0260036       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0835904      0.11726      0.164499      0.110639      0.0251394       10     
reco:pandora2Shower:LArPandoraShowerCreation          2.34362       4.58927       8.18599        3.695        2.10511        10     
reco:pandora2Showercalo:ShowerCalorimetry             2.47708       4.7653        8.33714       3.68864       2.16754        10     
reco:pandora2Showercalonosce:ShowerCalorimetry         2.005        4.44843        8.777        3.50504       2.10107        10     
[art]:TriggerResults:TriggerResultInserter          2.7501e-05    4.55303e-05   0.000130527   3.7647e-05    2.86993e-05      10     
end_path:out1:RootOutput                              5.1e-06     9.2636e-06    3.7452e-05     6.111e-06    9.41149e-06      10     
end_path:out1:RootOutput(write)                       4.52995       5.95144       7.63413       5.83206       1.08612        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6465.29 MB
  Peak resident set size usage (VmHWM): 4235.83 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258662_11_1_20251126T112756Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258662_11_1_20251126T112756Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258662_11_1_20251126T112756Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258662_11_1_20251126T112756Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258662_11_1_20251126T112756Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258662_11_1_20251126T112756Z_detsim_reco1.root.temp.json ====
.:
total 6460364
-rw-r--r-- 1 nobody nobody      40665 Nov 26 11:58 jobscript.log
-rw-r--r-- 1 nobody nobody        203 Nov 26 11:58 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:58 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258662_11_1_20251126T112756Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:58 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258662_11_1_20251126T112756Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 11:58 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258662_11_1_20251126T112756Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 11:58 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 11:58 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 11:58 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     204800 Nov 26 11:58 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 11:58 time.db
-rw-r--r-- 1 nobody nobody 6556700034 Nov 26 11:58 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258662_11_1_20251126T112756Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   58398800 Nov 26 11:56 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:19 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:15 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 11:14 detsim_single_protodunehd_hist.root
justIN time: 2026-02-04 07:20:03 UTC       justIN version: 01.06.00