justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258826.4@dunegpschedd01.fnal.gov

Jobsub ID258826.4@dunegpschedd01.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 15:14:35
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce03
Last heartbeat2025-11-26 18:11:03
From worker nodeHostnamewn78
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 15:15:41
Input fileshd-protodune:pdhd_prod_beam__243896_64_1_20251106T041841Z_gen_g4_IonScintPDExt.root_249776_144_1_20251122T195710Z_PDInt.root
JobscriptExit code0
Real time2h (10260s)
CPU time2h (8509s = 82%)
Max RSS bytes4433485824 (4228 MiB)
Outputting started2025-11-26 18:06:43
Output files
Finished2025-11-26 18:11:03
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

"
FrameSaver: q=2.69961e+06 n=1090133 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 14024 XUs and 9010 XVs -> 90 XUVs
C:0 T:1 94902 XUs and 81016 XVs -> 2457 XUVs
C:0 T:2 19584 XUs and 11015 XVs -> 323 XUVs
C:0 T:3 2 XUs and 1 XVs -> 0 XUVs
C:0 T:4 21 XUs and 22 XVs -> 2 XUVs
C:0 T:5 40971 XUs and 28140 XVs -> 1177 XUVs
C:0 T:6 27390 XUs and 15984 XVs -> 531 XUVs
4580 XUVs total
2128 collection wire objects
4580 potential space points
Neighbour search...
124212 tests to find 73068 neighbours
Iterating with no regularization...
Begin: 3.85338e+07
0 3.61934e+07
1 3.6026e+07
2 3.60161e+07
Now with regularization...
Begin: 3.50317e+07
0 3.50288e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 15:06:32 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258826_4_1_20251126T151549Z_detsim_reco1.root"
26-Nov-2025 15:06:32 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            157.327       214.387       311.081       200.606       43.1643        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00180494    0.00984079     0.0185916     0.0110792    0.00560647       10     
reco:opdec:Deconvolution                              1.18153       1.39253       1.76283       1.31415      0.192673        10     
reco:ophitspe:OpHitFinderDeco                         0.30219      0.334978      0.362787      0.332839      0.0182632       10     
reco:opflash:OpFlashFinder                           0.0901755     0.153638      0.235408      0.144176      0.0409965       10     
reco:opslicer:OpSlicer                                0.32802      0.451784      0.612103      0.434951      0.0910649       10     
reco:wclsdatahd:WireCellToolkit                       65.3116       69.9105       73.5757       69.9806       2.50366        10     
reco:gaushit:GausHitFinder                            1.0233        1.48989       1.95558       1.49711      0.299334        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000346945   0.000565173   0.00104189    0.000489776   0.000192806      10     
reco:reco3d:SpacePointSolver                         0.702035       1.65813       3.96886       1.42111      0.856754        10     
reco:hitpdune:DisambigFromSpacePoints                0.314541       1.20605       3.03204       1.06804      0.702342        10     
reco:pandora:StandardPandora                          18.9046       42.1765       66.8445       37.5629       15.8935        10     
reco:pandoraWriter:StandardPandora                   0.248549      0.696247       1.33132      0.588917       0.33698        10     
reco:pandoraTrack:LArPandoraTrackCreation             1.66541       3.50172       7.45217       3.21365       1.63651        10     
reco:pandoraShower:LArPandoraShowerCreation           1.54558       3.25533       7.69844       2.62919       1.87797        10     
reco:pandoracalo:Calorimetry                          1.07965       1.78478       3.7016        1.46479       0.81881        10     
reco:pandoracalonosce:Calorimetry                     1.05293       1.74896       3.60651       1.42823      0.764578        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.48197       3.06166       7.30051       2.60286       1.66261        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.42933       2.84684       7.05119       2.33069       1.62899        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0580771     0.0947228     0.148242      0.0916936     0.0268276       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0519143     0.0927837     0.151921      0.0909101     0.0274299       10     
reco:emtrkmichelid:EmTrackMichelId                    37.2977       49.5435       70.3838       47.8709       8.99619        10     
reco:crttag:SingleCRTMatchingProducer               0.00678352    0.00851436     0.0146424    0.00837216     0.0021962       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00560274    0.00723579     0.0115584    0.00717285     0.0015928       10     
reco:pandora2Track:LArPandoraTrackCreation            5.03361       7.98062       13.4243       7.13205       3.00019        10     
reco:pandora2calo:ShowerCalorimetry                   1.30959       2.58702       5.16595       2.19413       1.24594        10     
reco:pandora2calonosce:ShowerCalorimetry              1.26537       2.48712       4.47872       2.24584       1.08447        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0731185     0.0997507     0.156146      0.096109      0.024551        10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0579981     0.0974811     0.147516      0.0954135     0.0248019       10     
reco:pandora2Shower:LArPandoraShowerCreation          2.0788        3.42464       7.15192       2.78077       1.53267        10     
reco:pandora2Showercalo:ShowerCalorimetry             2.04465       3.19113       5.72886       2.75774       1.15661        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        2.04396       3.38458       6.76449       2.88518       1.46642        10     
[art]:TriggerResults:TriggerResultInserter          2.7461e-05    5.19135e-05   0.000160238   3.6287e-05    3.9446e-05       10     
end_path:out1:RootOutput                             4.96e-06     9.2152e-06    3.2001e-05     6.35e-06     7.72681e-06      10     
end_path:out1:RootOutput(write)                       3.81785       5.52892       7.23414       5.44248      0.965668        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6181.24 MB
  Peak resident set size usage (VmHWM): 3960.24 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258826_4_1_20251126T151549Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258826_4_1_20251126T151549Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258826_4_1_20251126T151549Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258826_4_1_20251126T151549Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258826_4_1_20251126T151549Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258826_4_1_20251126T151549Z_detsim_reco1.root.temp.json ====
.:
total 5989076
-rw-r--r-- 1 nobody nobody      41128 Nov 26 15:06 jobscript.log
-rw-r--r-- 1 nobody nobody        204 Nov 26 15:06 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 15:06 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258826_4_1_20251126T151549Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 15:06 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258826_4_1_20251126T151549Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 15:06 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258826_4_1_20251126T151549Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 15:06 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 15:06 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 15:06 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     204800 Nov 26 15:06 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 15:06 time.db
-rw-r--r-- 1 nobody nobody 6081078213 Nov 26 15:06 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258826_4_1_20251126T151549Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   51420730 Nov 26 15:05 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 14:31 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 14:28 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 14:27 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 07:14:38 UTC       justIN version: 01.05.03