justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.88@dunegpschedd02.fnal.gov

Jobsub ID297923.88@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteIT_CNAF
EntryCMSHTPC_T1_IT_CNAF_condor_ce02
Last heartbeat2026-02-04 01:25:47
From worker nodeHostnamewn-f01-15-03-09.cr.cnaf.infn.it
cpuinfoIntel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:00:37
Input filesvd-protodune:np02vd_raw_run042433_0417_df-s03-d1_dw_0_20260203T212036.hdf5
JobscriptExit code0
Real time1h (4984s)
CPU time1h (4749s = 95%)
Max RSS bytes3400826880 (3243 MiB)
Outputting started2026-02-04 01:23:44
Output files
Finished2026-02-04 01:25:47
Saved logsjustin-logs:297923.88-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

oint ptr... 0x69ad9d10, 0x69ad9540
Caught exception in looking up space point ptr... 0x6eb292a0, 0x6eb29150
Caught exception in looking up space point ptr... 0x6eb29150, 0x6eb292a0
Caught exception in looking up space point ptr... 0x5b001c90, 0x5b001dd0
Caught exception in looking up space point ptr... 0x5b001dd0, 0x5b00a7c0
Caught exception in looking up space point ptr... 0x14f39700, 0x14f393e0
Caught exception in looking up space point ptr... 0x14f393e0, 0x14f39700
Caught exception in looking up space point ptr... 0x36df18b0, 0x36df17b0
Caught exception in looking up space point ptr... 0x36df3aa0, 0x36df18b0
Caught exception in looking up space point ptr... 0x11755c80, 0x11755e10
Caught exception in looking up space point ptr... 0x11755e10, 0x117562c0
Caught exception in looking up space point ptr... 0xeb8a060, 0xeb8a1f0
Caught exception in looking up space point ptr... 0xeb8a1f0, 0xeb8a060
Caught exception in looking up space point ptr... 0x546fca10, 0x546fbba0
Caught exception in looking up space point ptr... 0x546fbba0, 0x546fc040
Caught exception in looking up space point ptr... 0x597b86a0, 0x597b81a0
Caught exception in looking up space point ptr... 0x597b81a0, 0x597b86a0
Caught exception in looking up space point ptr... 0x625417d0, 0x625427f0
Caught exception in looking up space point ptr... 0x62541870, 0x625417d0
Caught exception in looking up space point ptr... 0x69cbc8c0, 0x69cbc5a0
Caught exception in looking up space point ptr... 0x69cbc5a0, 0x69cbc730
Caught exception in looking up space point ptr... 0x68bd6420, 0x68bd6710
Caught exception in looking up space point ptr... 0x68bd6710, 0x68bd6420
Caught exception in looking up space point ptr... 0x69b17ba0, 0x69b183d0
Caught exception in looking up space point ptr... 0x69b183d0, 0x69b1a6d0
Caught exception in looking up space point ptr... 0x36b2ec30, 0x36b35670
Caught exception in looking up space point ptr... 0x36b35670, 0x36b2ea90
Caught exception in looking up space point ptr... 0xfc27150, 0xfc1c7e0
Caught exception in looking up space point ptr... 0xfc1c7e0, 0xfc21890
Caught exception in looking up space point ptr... 0x6b3734d0, 0x6b3733a0
Caught exception in looking up space point ptr... 0x6b3733a0, 0x6b3734d0
Caught exception in looking up space point ptr... 0xd3da980, 0xd3dad80
Caught exception in looking up space point ptr... 0xd3dad80, 0xd3da590
Caught exception in looking up space point ptr... 0x72eb3830, 0x4aef0760
Caught exception in looking up space point ptr... 0x4aef0760, 0x72eb3830
Caught exception in looking up space point ptr... 0x547fb120, 0x547f8350
Caught exception in looking up space point ptr... 0x547f8350, 0x547fb120
Caught exception in looking up space point ptr... 0x18a65ba0, 0x18a4d680
Caught exception in looking up space point ptr... 0x18a4d680, 0x18a65ba0
Caught exception in looking up space point ptr... 0x17cc9e60, 0x17cc9020
Caught exception in looking up space point ptr... 0x17cc9020, 0x17cc9e60
Caught exception in looking up space point ptr... 0x6a6a6820, 0x6a6a6700
Caught exception in looking up space point ptr... 0x6a6a6700, 0x6a6a6820
Caught exception in looking up space point ptr... 0x57aa39d0, 0x57aab450
Caught exception in looking up space point ptr... 0x57aab450, 0x57aac4c0
Caught exception in looking up space point ptr... 0x7ea70850, 0x7ea70a60
Caught exception in looking up space point ptr... 0x7ea70a60, 0x7ea6f840
Caught exception in looking up space point ptr... 0x6f700a70, 0x368f0cb0
Caught exception in looking up space point ptr... 0x368f0cb0, 0x6f700a70
Caught exception in looking up space point ptr... 0x597bf680, 0x597bf2c0
Caught exception in looking up space point ptr... 0x597bf2c0, 0x597bf4a0
Caught exception in looking up space point ptr... 0x6ec59710, 0x6ec595e0
Caught exception in looking up space point ptr... 0x6ec595e0, 0x6ec59710
Caught exception in looking up space point ptr... 0x53c3d450, 0x53c3f510
Caught exception in looking up space point ptr... 0x53c3f510, 0x53c3d450
Caught exception in looking up space point ptr... 0x729fcaa0, 0x6317e980
Caught exception in looking up space point ptr... 0x6317e980, 0x729fcaa0
Caught exception in looking up space point ptr... 0x18a27920, 0x18a27bc0
Caught exception in looking up space point ptr... 0x18a27bc0, 0x18a27ec0
Caught exception in looking up space point ptr... 0x70cd6da0, 0x12f88190
Caught exception in looking up space point ptr... 0x12f88190, 0x423677d0
Caught exception in looking up space point ptr... 0x17d20480, 0x17d200c0
Caught exception in looking up space point ptr... 0x17d200c0, 0x17d20480
Caught exception in looking up space point ptr... 0x60781ab0, 0x60780f30
Caught exception in looking up space point ptr... 0x60780f30, 0x6077e3e0
Caught exception in looking up space point ptr... 0x6ec5b2e0, 0x6ec5b150
Caught exception in looking up space point ptr... 0x6ec5b150, 0x6ec5a770
++++>>>> total num hits: 189348, num free: 181761
04-Feb-2026 02:23:30 CET  Closed output file "np02vd_raw_run042433_0417_df-s03-d1_dw_0_20260203T212036_reco_stage0_20260204T012330_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         226.297       256.155       316.102       250.199       24.2395        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.1649e-05    0.000117791   0.000228751   0.000105649   4.50524e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                12.7892       15.4873       17.4101       15.5012       1.15306        19     
produce:triggerrawdecoder:PDVDTriggerReader4       0.30995      0.345203      0.465364      0.333077      0.0474574       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0105803     0.0156979     0.0879449     0.0107411     0.0171716       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0105391     0.0107257     0.0109919     0.0106694    0.000126963      19     
produce:beamevent:BeamEvent                      0.000122252   0.000218208   0.00106224    0.000171937   0.000201379      19     
produce:pdvddaphne:DAPHNEReaderPDVD                5.20239       6.50157       8.58999       6.18129       1.00747        19     
produce:ophit:OpHitFinder                         0.0378331     0.0527501     0.0723729     0.0568943     0.0104858       19     
produce:wclsdatavd:WireCellToolkit                 163.965       192.931       231.551       185.989       20.8213        19     
produce:gaushit:GausHitFinder                      1.42845       2.27891       3.97422       2.24111      0.572817        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000565649   0.000712531   0.00113047    0.000629519   0.000161924      19     
produce:reco3d:SpacePointSolver                    15.8946       26.1616       38.2331       25.3388       6.3571         19     
produce:hitpdune:DisambigFromSpacePoints          0.186323      0.325154      0.588897       0.29372      0.0950343       19     
produce:cluster3d:Cluster3D                        1.96956       4.42655       10.4243       3.46787       2.12888        19     
[art]:TriggerResults:TriggerResultInserter       2.4951e-05    3.92397e-05   9.3224e-05    3.3485e-05    1.52152e-05      19     
end_path:out1:RootOutput                          4.764e-06    9.56126e-06   2.4866e-05     8.145e-06    4.55383e-06      19     
end_path:out1:RootOutput(write)                    5.54366       7.5641        10.9612       6.43282       1.89511        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5335.58 MB
  Peak resident set size usage (VmHWM): 3400.83 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0417_df-s03-d1_dw_0_20260203T212036_reco_stage0_20260204T012330_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0417_df-s03-d1_dw_0_20260203T212036_reco_stage0_20260204T012330_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0417_df-s03-d1_dw_0_20260203T212036_reco_stage0_20260204T012330_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163253.0,
    "core.end_time": 1770168213.0,
    "core.events": [
      47547,
      47553,
      47559,
      47565,
      47571,
      47577,
      47583,
      47589,
      47595,
      47601,
      47607,
      47613,
      47619,
      47625,
      47631,
      47637,
      47643,
      47649,
      47655
    ],
    "core.event_count": 19,
    "core.first_event_number": 47547,
    "core.last_event_number": 47655,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0417_df-s03-d1_dw_0_20260203T212036.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:53:26 UTC       justIN version: 01.06.00