justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303658.29@dunegpschedd01.fnal.gov

Jobsub ID303658.29@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:56:27
SiteIT_CNAF
EntryCMSHTPC_T1_IT_CNAF_condor_ce02
Last heartbeat2026-02-04 01:45:36
From worker nodeHostnamewn-f01-15-05-03.cr.cnaf.infn.it
cpuinfoIntel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:15:14
Input filesvd-protodune:np02vd_raw_run042433_0618_df-s03-d1_dw_0_20260203T223701.hdf5
JobscriptExit code0
Real time1h (5303s)
CPU time1h (5033s = 94%)
Max RSS bytes3346894848 (3191 MiB)
Outputting started2026-02-04 01:43:38
Output files
Finished2026-02-04 01:45:36
Saved logsjustin-logs:303658.29-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e point ptr... 0x38757d50, 0x5d46aa30
Caught exception in looking up space point ptr... 0x5d46aa30, 0x6acc9400
Caught exception in looking up space point ptr... 0x6acc9400, 0x41c7bc20
Caught exception in looking up space point ptr... 0x41c7bc20, 0x3ec14b20
Caught exception in looking up space point ptr... 0x3ec14b20, 0x7e72b2f0
Caught exception in looking up space point ptr... 0x5d341d50, 0x34f4eac0
Caught exception in looking up space point ptr... 0x34f4eac0, 0xa003930
Caught exception in looking up space point ptr... 0x362a7690, 0x28e43850
Caught exception in looking up space point ptr... 0x28e43850, 0x3a319370
Caught exception in looking up space point ptr... 0x69934ea0, 0x362a7690
Caught exception in looking up space point ptr... 0x6538f900, 0x6538d030
Caught exception in looking up space point ptr... 0x6538d030, 0x14715f30
Caught exception in looking up space point ptr... 0x14715f30, 0x754c86e0
Caught exception in looking up space point ptr... 0x754c86e0, 0x51a9d790
Caught exception in looking up space point ptr... 0x39bb2290, 0x52aafea0
Caught exception in looking up space point ptr... 0x52aafea0, 0xf771cd0
Caught exception in looking up space point ptr... 0x765943f0, 0x765944f0
Caught exception in looking up space point ptr... 0x765944f0, 0x11333b90
Caught exception in looking up space point ptr... 0xf5b19e0, 0x37ddb0e0
Caught exception in looking up space point ptr... 0x37ddb0e0, 0x51427350
Caught exception in looking up space point ptr... 0x10f74710, 0x3fa25b30
Caught exception in looking up space point ptr... 0x3a033cf0, 0x10f74710
Caught exception in looking up space point ptr... 0x34442410, 0x31e25e60
Caught exception in looking up space point ptr... 0x31e25e60, 0x34442410
Caught exception in looking up space point ptr... 0x3543eef0, 0x3a1edf20
Caught exception in looking up space point ptr... 0x3a1edf20, 0xc44fe30
Caught exception in looking up space point ptr... 0x31f4cd20, 0x6aafaec0
Caught exception in looking up space point ptr... 0x6aafaec0, 0x3eca20a0
Caught exception in looking up space point ptr... 0x68afeca0, 0x181f7010
Caught exception in looking up space point ptr... 0x181f7010, 0x68afeca0
Caught exception in looking up space point ptr... 0x68afeca0, 0x4e41aab0
Caught exception in looking up space point ptr... 0x4e41aab0, 0x68afeca0
Caught exception in looking up space point ptr... 0x13bb08d0, 0x781f5890
Caught exception in looking up space point ptr... 0x781f5890, 0x6e4411a0
Caught exception in looking up space point ptr... 0x6e440ba0, 0x13bb08d0
Caught exception in looking up space point ptr... 0x765449b0, 0x765436d0
Caught exception in looking up space point ptr... 0x765436d0, 0x765449b0
Caught exception in looking up space point ptr... 0x5003f220, 0x5003f360
Caught exception in looking up space point ptr... 0x5003f360, 0x124e6300
Caught exception in looking up space point ptr... 0x3c701370, 0x384ae1a0
Caught exception in looking up space point ptr... 0x384ae1a0, 0xa11ad40
Caught exception in looking up space point ptr... 0x35fec390, 0x366d8810
Caught exception in looking up space point ptr... 0x366d8810, 0x35fec390
Caught exception in looking up space point ptr... 0x135cb4d0, 0x135cb5d0
Caught exception in looking up space point ptr... 0x135cb5d0, 0x135cb4d0
Caught exception in looking up space point ptr... 0x345435d0, 0x3a8ffa30
Caught exception in looking up space point ptr... 0x3a8ffa30, 0x6b562b60
Caught exception in looking up space point ptr... 0xcc74e50, 0x41b695c0
Caught exception in looking up space point ptr... 0x41b695c0, 0xcc74e50
Caught exception in looking up space point ptr... 0xcc74e50, 0x4e3e5a30
Caught exception in looking up space point ptr... 0x4e3e5a30, 0xcc74e50
Caught exception in looking up space point ptr... 0x6aae9310, 0x38443960
Caught exception in looking up space point ptr... 0x38443960, 0x6aae9310
Caught exception in looking up space point ptr... 0x6cec3320, 0x2d26eb90
Caught exception in looking up space point ptr... 0x2d26eb90, 0x6cec3320
Caught exception in looking up space point ptr... 0x650724a0, 0x6ad77e80
Caught exception in looking up space point ptr... 0x6ad77e80, 0x650724a0
Caught exception in looking up space point ptr... 0x3eccfe80, 0x36e49370
Caught exception in looking up space point ptr... 0x14e70510, 0x3eccfe80
Caught exception in looking up space point ptr... 0x4e442ce0, 0x13e70c40
Caught exception in looking up space point ptr... 0x13e70c40, 0x4e442ce0
Caught exception in looking up space point ptr... 0xf545110, 0xf545210
Caught exception in looking up space point ptr... 0xf545210, 0xf545110
Caught exception in looking up space point ptr... 0x6fae57a0, 0x61a96f00
Caught exception in looking up space point ptr... 0x61a96f00, 0x124ea600
Caught exception in looking up space point ptr... 0x3ad864d0, 0x3c3df140
Caught exception in looking up space point ptr... 0x3c3df140, 0x4e2fad20
++++>>>> total num hits: 38168, num free: 36932
04-Feb-2026 02:43:27 CET  Closed output file "np02vd_raw_run042433_0618_df-s03-d1_dw_0_20260203T223701_reco_stage0_20260204T014327_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         235.025       272.701       316.06        270.774       20.5459        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.8891e-05    0.000121759   0.000306102   0.000103698   6.08036e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                15.0317       16.3135       20.8697       15.9989       1.32327        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.306194      0.327278      0.383555      0.317418      0.0255716       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0103608     0.0104969     0.0108978     0.0104803    0.00010696       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.010361      0.0105027     0.0110547     0.0104735    0.000146811      19     
produce:beamevent:BeamEvent                      8.7299e-05    0.000213393   0.00138455    0.00015538    0.000277625      19     
produce:pdvddaphne:DAPHNEReaderPDVD                5.31972       6.46671       7.09839       6.55433      0.429382        19     
produce:ophit:OpHitFinder                         0.0429595     0.0529038     0.0619292     0.0533542    0.00514073       19     
produce:wclsdatavd:WireCellToolkit                 185.597       216.929       249.371       217.057       15.3546        19     
produce:gaushit:GausHitFinder                      1.32367       2.19373       4.22174       1.87824      0.765094        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000393531   0.00054033    0.000817854   0.000518815   0.000121345      19     
produce:reco3d:SpacePointSolver                    11.3355       18.4227       35.3301       16.3916       5.60765        19     
produce:hitpdune:DisambigFromSpacePoints          0.141213      0.306111      0.550835      0.273242      0.104024        19     
produce:cluster3d:Cluster3D                        1.78278       3.53842       8.53814       2.98962       1.64939        19     
[art]:TriggerResults:TriggerResultInserter       2.6976e-05    4.66426e-05   0.000125936   4.0621e-05    2.1674e-05       19     
end_path:out1:RootOutput                          5.037e-06    1.07416e-05   3.5188e-05     8.602e-06    6.82204e-06      19     
end_path:out1:RootOutput(write)                    5.92676       8.08941       10.5901       7.54868       1.47832        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5303.87 MB
  Peak resident set size usage (VmHWM): 3346.89 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0618_df-s03-d1_dw_0_20260203T223701_reco_stage0_20260204T014327_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0618_df-s03-d1_dw_0_20260203T223701_reco_stage0_20260204T014327_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0618_df-s03-d1_dw_0_20260203T223701_reco_stage0_20260204T014327_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770164134.0,
    "core.end_time": 1770169409.0,
    "core.events": [
      70461,
      70467,
      70473,
      70479,
      70485,
      70491,
      70497,
      70503,
      70509,
      70515,
      70521,
      70527,
      70533,
      70539,
      70545,
      70551,
      70557,
      70563,
      70569
    ],
    "core.event_count": 19,
    "core.first_event_number": 70461,
    "core.last_event_number": 70569,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0618_df-s03-d1_dw_0_20260203T223701.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:53:17 UTC       justIN version: 01.06.00