justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.90@dunegpschedd02.fnal.gov

Jobsub ID297921.90@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteIT_CNAF
EntryCMSHTPC_T1_IT_CNAF_condor_ce02
Last heartbeat2026-02-04 01:18:04
From worker nodeHostnamewn-s04-06-11-04.cr.cnaf.infn.it
cpuinfoAMD EPYC 7351 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:54:19
Input filesvd-protodune:np02vd_raw_run042433_0155_df-s04-d0_dw_0_20260203T194102.hdf5
JobscriptExit code0
Real time1h (4884s)
CPU time1h (4324s = 88%)
Max RSS bytes3431878656 (3272 MiB)
Outputting started2026-02-04 01:15:44
Output files
Finished2026-02-04 01:18:04
Saved logsjustin-logs:297921.90-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

up space point ptr... 0x704f40b0, 0x6d643410
Caught exception in looking up space point ptr... 0x6d643410, 0x704f43a0
Caught exception in looking up space point ptr... 0x5b995ba0, 0xc951cf0
Caught exception in looking up space point ptr... 0xc951cf0, 0x5b995ba0
Caught exception in looking up space point ptr... 0x5c7b19d0, 0x5c7b18d0
Caught exception in looking up space point ptr... 0x5c7b18d0, 0x3197ece0
Caught exception in looking up space point ptr... 0x2a2e74c0, 0x17f86180
Caught exception in looking up space point ptr... 0x3d757640, 0x38ff1250
Caught exception in looking up space point ptr... 0x38ff1250, 0x2a2e74c0
Caught exception in looking up space point ptr... 0x643cda00, 0x643bf5b0
Caught exception in looking up space point ptr... 0x643bf5b0, 0x643bf420
Caught exception in looking up space point ptr... 0x5e79ee90, 0x5e79ec40
Caught exception in looking up space point ptr... 0x5e7a2500, 0x5e79ee90
Caught exception in looking up space point ptr... 0x15758180, 0x15759de0
Caught exception in looking up space point ptr... 0x15751cb0, 0x15758180
Caught exception in looking up space point ptr... 0x17576bf0, 0x17576a60
Caught exception in looking up space point ptr... 0x175716f0, 0x17576bf0
Caught exception in looking up space point ptr... 0x5b18c460, 0x64a43ce0
Caught exception in looking up space point ptr... 0x64a43ce0, 0x64a43bd0
Caught exception in looking up space point ptr... 0x58d6f9c0, 0x58d725a0
Caught exception in looking up space point ptr... 0x58d725a0, 0x58d6f9c0
Caught exception in looking up space point ptr... 0x497138f0, 0x52ba9d70
Caught exception in looking up space point ptr... 0x52ba9d70, 0x497138f0
Caught exception in looking up space point ptr... 0xfa27860, 0xfa28210
Caught exception in looking up space point ptr... 0xfa28210, 0x379953d0
Caught exception in looking up space point ptr... 0x6d3493a0, 0x6d3511f0
Caught exception in looking up space point ptr... 0x6d3511f0, 0x6d3493a0
Caught exception in looking up space point ptr... 0x643bdcb0, 0x643bdb20
Caught exception in looking up space point ptr... 0x643bdb20, 0x643bdcb0
Caught exception in looking up space point ptr... 0xcd222e0, 0xcd22050
Caught exception in looking up space point ptr... 0xcd23ee0, 0xcd222e0
Caught exception in looking up space point ptr... 0x3a5c4950, 0x3a5c4a50
Caught exception in looking up space point ptr... 0x3a5c4a50, 0x5ce7d890
Caught exception in looking up space point ptr... 0x575f0380, 0x575ee930
Caught exception in looking up space point ptr... 0x575ee930, 0x575ef620
Caught exception in looking up space point ptr... 0x52b518b0, 0x109f48a0
Caught exception in looking up space point ptr... 0x109f48a0, 0x52b518b0
Caught exception in looking up space point ptr... 0x3d0f3590, 0x731fe340
Caught exception in looking up space point ptr... 0x731fe340, 0x3d0f3590
Caught exception in looking up space point ptr... 0x16d1fa00, 0x10c08bf0
Caught exception in looking up space point ptr... 0x16d1f6b0, 0x16d1fa00
Caught exception in looking up space point ptr... 0x643d0ef0, 0x643d0d60
Caught exception in looking up space point ptr... 0x643d0d60, 0x643d0bd0
Caught exception in looking up space point ptr... 0x65383240, 0x65383140
Caught exception in looking up space point ptr... 0x65383140, 0x65383240
Caught exception in looking up space point ptr... 0xfcabe90, 0xfca3fd0
Caught exception in looking up space point ptr... 0xfca3fd0, 0xfcabe90
Caught exception in looking up space point ptr... 0x5d9a3840, 0x5d9a36b0
Caught exception in looking up space point ptr... 0x65364ef0, 0x5d9a3840
Caught exception in looking up space point ptr... 0x5b8d6150, 0x63d74c70
Caught exception in looking up space point ptr... 0x379aa9e0, 0x5b8d6150
Caught exception in looking up space point ptr... 0x2a184270, 0x846a640
Caught exception in looking up space point ptr... 0x846a640, 0xd902500
Caught exception in looking up space point ptr... 0x5df85490, 0x5df855c0
Caught exception in looking up space point ptr... 0x5df855c0, 0x5df881e0
Caught exception in looking up space point ptr... 0xcd52bb0, 0xcd53200
Caught exception in looking up space point ptr... 0xcd53200, 0x628cee70
Caught exception in looking up space point ptr... 0x5df8e180, 0x5df8e050
Caught exception in looking up space point ptr... 0x5df8e050, 0x5df8e180
Caught exception in looking up space point ptr... 0x3ce49f40, 0x123617e0
Caught exception in looking up space point ptr... 0x39175a70, 0x3ce49f40
Caught exception in looking up space point ptr... 0x14ed1c00, 0x38b299b0
Caught exception in looking up space point ptr... 0x38b299b0, 0xa0043d0
Caught exception in looking up space point ptr... 0x61826bd0, 0x36f24790
Caught exception in looking up space point ptr... 0x36f24790, 0x5cdd2c90
Caught exception in looking up space point ptr... 0x46875ca0, 0x6fa3fe60
Caught exception in looking up space point ptr... 0x6fa3fe60, 0x7291b050
++++>>>> total num hits: 58931, num free: 56530
04-Feb-2026 02:15:33 CET  Closed output file "np02vd_raw_run042433_0155_df-s04-d0_dw_0_20260203T194102_reco_stage0_20260204T011533_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         198.15        250.255       294.229       246.947       24.5813        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        6.999e-05    0.000135178   0.000297321   0.00011598    5.45199e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                19.5228       27.1409       44.8461       26.4158       5.50847        19     
produce:triggerrawdecoder:PDVDTriggerReader4       0.30883      0.357953      0.768294      0.328646      0.0993366       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0104293     0.0150043     0.0679918     0.0106837     0.0129628       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0103648     0.0107035     0.0112895     0.0106332    0.000253252      19     
produce:beamevent:BeamEvent                      0.000108811   0.000280364    0.0014529    0.000211801   0.000285435      19     
produce:pdvddaphne:DAPHNEReaderPDVD                4.49813       5.31304       6.19698       5.26521      0.538215        19     
produce:ophit:OpHitFinder                         0.0391933     0.0557444     0.100275      0.0543697     0.0129625       19     
produce:wclsdatavd:WireCellToolkit                 143.722       181.076       212.003       181.852       16.239         19     
produce:gaushit:GausHitFinder                      1.69045       2.1934        3.18919       2.01916      0.455785        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000443871   0.000602538   0.00118869    0.000552251   0.000167425      19     
produce:reco3d:SpacePointSolver                    14.1512       21.0311       34.1792       19.9981       4.95342        19     
produce:hitpdune:DisambigFromSpacePoints          0.136092      0.301694      0.489822      0.301649      0.0983239       19     
produce:cluster3d:Cluster3D                        1.70143       4.81046       21.9736       3.32951       4.32119        19     
[art]:TriggerResults:TriggerResultInserter        2.753e-05    4.86891e-05   0.00012173     3.649e-05    2.82098e-05      19     
end_path:out1:RootOutput                          3.85e-06     1.00401e-05    3.338e-05     7.65e-06     6.76168e-06      19     
end_path:out1:RootOutput(write)                    5.82057       7.89371       10.7469       8.06298       1.3359         19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5337.75 MB
  Peak resident set size usage (VmHWM): 3431.88 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0155_df-s04-d0_dw_0_20260203T194102_reco_stage0_20260204T011533_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0155_df-s04-d0_dw_0_20260203T194102_reco_stage0_20260204T011533_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0155_df-s04-d0_dw_0_20260203T194102_reco_stage0_20260204T011533_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162881.0,
    "core.end_time": 1770167736.0,
    "core.events": [
      17680,
      17686,
      17692,
      17698,
      17704,
      17710,
      17716,
      17722,
      17728,
      17734,
      17740,
      17746,
      17752,
      17758,
      17764,
      17770,
      17776,
      17782,
      17788
    ],
    "core.event_count": 19,
    "core.first_event_number": 17680,
    "core.last_event_number": 17788,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0155_df-s04-d0_dw_0_20260203T194102.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:03 UTC       justIN version: 01.06.00