justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 253055.4@dunegpschedd02.fnal.gov

Jobsub ID253055.4@dunegpschedd02.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 12:44:30
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce04
Last heartbeat2025-11-26 17:15:49
From worker nodeHostnamewn123
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 12:46:02
Input fileshd-protodune:pdhd_prod_beam__238233_17_1_20251106T053629Z_gen_g4_IonScintPDExt.root_255523_24_1_20251122T220716Z_PDInt.root
JobscriptExit code0
Real time4h (15917s)
CPU time2h (10287s = 64%)
Max RSS bytes4514299904 (4305 MiB)
Outputting started2025-11-26 17:11:20
Output files
Finished2025-11-26 17:15:49
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

68 XUs and 54932 XVs -> 1959 XUVs
C:0 T:2 14522 XUs and 13797 XVs -> 396 XUVs
C:0 T:3 4 XUs and 8 XVs -> 1 XUVs
C:0 T:4 28 XUs and 22 XVs -> 0 XUVs
C:0 T:5 24320 XUs and 21487 XVs -> 624 XUVs
C:0 T:6 11110 XUs and 12021 XVs -> 770 XUVs
3829 XUVs total
2255 collection wire objects
3829 potential space points
Neighbour search...
65445 tests to find 33720 neighbours
Iterating with no regularization...
Begin: 4.88925e+07
0 4.63798e+07
1 4.62193e+07
2 4.62123e+07
Now with regularization...
Begin: 4.43351e+07
0 4.43332e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 14:11:08 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253055_4_1_20251126T124609Z_detsim_reco1.root"
26-Nov-2025 14:11:08 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            173.06        260.545       346.608       266.667       55.4164        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00176271    0.00989916     0.0207601    0.00953212    0.00576237       10     
reco:opdec:Deconvolution                              1.06662       1.43601       1.79039       1.35755      0.208083        10     
reco:ophitspe:OpHitFinderDeco                        0.315676      0.364909      0.433927      0.359693      0.0349525       10     
reco:opflash:OpFlashFinder                           0.0913571     0.142425      0.206935      0.124445      0.0406244       10     
reco:opslicer:OpSlicer                               0.273731      0.461641      0.588693      0.482821      0.0864516       10     
reco:wclsdatahd:WireCellToolkit                       78.6816       83.3532       89.8594       82.6424       3.97387        10     
reco:gaushit:GausHitFinder                           0.881962       1.50151       1.90867       1.60419      0.328351        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000456642   0.00151406     0.0024567    0.00137111    0.000554811      10     
reco:reco3d:SpacePointSolver                         0.500584       1.69527       2.88981       1.77952      0.850442        10     
reco:hitpdune:DisambigFromSpacePoints                0.473474       1.10172       1.95003       1.01636      0.494955        10     
reco:pandora:StandardPandora                          13.4051       40.1493       65.8779       45.4509       16.2602        10     
reco:pandoraWriter:StandardPandora                   0.509241      0.825567       1.71081      0.778242      0.317784        10     
reco:pandoraTrack:LArPandoraTrackCreation             2.48484       6.35167       13.4538       4.88803       3.46533        10     
reco:pandoraShower:LArPandoraShowerCreation           1.84273       6.00428       12.9506       4.17744       3.58076        10     
reco:pandoracalo:Calorimetry                          1.10987       2.40298       4.79116       1.86981       1.22328        10     
reco:pandoracalonosce:Calorimetry                     1.11881       2.55437       6.2229        1.93307       1.55396        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.39187       4.50934       10.9925       3.53875       2.75412        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.66725       4.90393       9.1666        4.15597       2.69653        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0789079     0.142793      0.194099      0.143129      0.0350287       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0832508     0.146526      0.189573      0.152219      0.034644        10     
reco:emtrkmichelid:EmTrackMichelId                    38.9222       57.5844       70.772        61.5512       10.4791        10     
reco:crttag:SingleCRTMatchingProducer                0.0101882     0.0165236     0.0238011     0.0159244    0.00449335       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00807896     0.0154032     0.0283396     0.0145236    0.00525779       10     
reco:pandora2Track:LArPandoraTrackCreation            8.93434       15.4475       31.803        13.113        7.10687        10     
reco:pandora2calo:ShowerCalorimetry                   2.70729       4.13676       6.00892       3.88478       1.20852        10     
reco:pandora2calonosce:ShowerCalorimetry              2.40232       3.8666        6.94881       3.30551       1.58402        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0751745      0.12087      0.171657      0.110148      0.0339675       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0937529     0.132938      0.190313      0.124932      0.0304374       10     
reco:pandora2Shower:LArPandoraShowerCreation          2.63641       5.71105       11.5749       4.69083       2.70876        10     
reco:pandora2Showercalo:ShowerCalorimetry             2.46901       4.82123       8.6456        4.21259       1.82189        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        2.10262       4.79282       8.17536       4.18648       1.9686         10     
[art]:TriggerResults:TriggerResultInserter          3.9002e-05    8.21219e-05   0.000282054   5.65825e-05   6.96174e-05      10     
end_path:out1:RootOutput                             6.571e-06    1.6665e-05    8.6295e-05     9.166e-06    2.32812e-05      10     
end_path:out1:RootOutput(write)                       3.81623       5.51686       6.62876       5.63766      0.875141        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6109.63 MB
  Peak resident set size usage (VmHWM): 3889.92 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253055_4_1_20251126T124609Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253055_4_1_20251126T124609Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253055_4_1_20251126T124609Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253055_4_1_20251126T124609Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253055_4_1_20251126T124609Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253055_4_1_20251126T124609Z_detsim_reco1.root.temp.json ====
.:
total 5979056
-rw-r--r-- 1 nobody nobody      42807 Nov 26 14:11 jobscript.log
-rw-r--r-- 1 nobody nobody        203 Nov 26 14:11 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 14:11 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253055_4_1_20251126T124609Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 14:11 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253055_4_1_20251126T124609Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 14:11 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253055_4_1_20251126T124609Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 14:11 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 14:11 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 14:11 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     204800 Nov 26 14:11 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 14:11 time.db
-rw-r--r-- 1 nobody nobody 6072858110 Nov 26 14:11 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253055_4_1_20251126T124609Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   49378140 Nov 26 14:09 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 13:29 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 13:24 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 13:23 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 09:19:12 UTC       justIN version: 01.05.03