justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.179@dunegpschedd01.fnal.gov

Jobsub ID303652.179@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce514
Last heartbeat2026-02-04 00:37:57
From worker nodeHostnameb9p06p2461.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:34:49
Input filesvd-protodune:np02vd_raw_run042433_0095_df-s03-d1_dw_0_20260203T191813.hdf5
JobscriptExit code0
Real time1h (3624s)
CPU time57m (3449s = 95%)
Max RSS bytes3609808896 (3442 MiB)
Outputting started2026-02-04 00:35:15
Output files
Finished2026-02-04 00:37:57
Saved logsjustin-logs:303652.179-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ptr... 0x3ca54040, 0x5a8abb80
Caught exception in looking up space point ptr... 0x5a8abb80, 0x5a8abc60
Caught exception in looking up space point ptr... 0x5a8abc60, 0x5d01def0
Caught exception in looking up space point ptr... 0x455fff70, 0x54203930
Caught exception in looking up space point ptr... 0x459d7920, 0x455fff70
Caught exception in looking up space point ptr... 0x329dcb60, 0x329dca10
Caught exception in looking up space point ptr... 0x3715fb50, 0x329dcb60
Caught exception in looking up space point ptr... 0x328a2fe0, 0x2d7d72e0
Caught exception in looking up space point ptr... 0x2d7d72e0, 0x328a2fe0
Caught exception in looking up space point ptr... 0x59a697e0, 0x70998250
Caught exception in looking up space point ptr... 0x70998250, 0x50a323a0
Caught exception in looking up space point ptr... 0x3a707a10, 0x3a707870
Caught exception in looking up space point ptr... 0x1a7c6d90, 0x3a707a10
Caught exception in looking up space point ptr... 0x3dd6cd00, 0x70e41960
Caught exception in looking up space point ptr... 0x2d754480, 0x3dd6cd00
Caught exception in looking up space point ptr... 0x102754c0, 0x10273bb0
Caught exception in looking up space point ptr... 0x10273bb0, 0x60417000
Caught exception in looking up space point ptr... 0x454b5870, 0x15c79100
Caught exception in looking up space point ptr... 0x454b5d20, 0x454b5870
Caught exception in looking up space point ptr... 0x5acf3ff0, 0x2d2c3790
Caught exception in looking up space point ptr... 0x2d2c3790, 0x2d2c3bf0
Caught exception in looking up space point ptr... 0x2e597000, 0x2e596c40
Caught exception in looking up space point ptr... 0x2e596c40, 0x822a32f0
Caught exception in looking up space point ptr... 0x45734aa0, 0x45734910
Caught exception in looking up space point ptr... 0x45734910, 0x457345f0
Caught exception in looking up space point ptr... 0x37a717e0, 0x473cc290
Caught exception in looking up space point ptr... 0x473cc290, 0x3cabb500
Caught exception in looking up space point ptr... 0x18b99270, 0x3a064e50
Caught exception in looking up space point ptr... 0x3a064e50, 0x4f5eee80
Caught exception in looking up space point ptr... 0x3f83a650, 0x3f83ab90
Caught exception in looking up space point ptr... 0x3f83ab90, 0x3f83ae30
Caught exception in looking up space point ptr... 0x2d7c1770, 0x2d7c1570
Caught exception in looking up space point ptr... 0x2d7c1570, 0x706c84b0
Caught exception in looking up space point ptr... 0x50f363b0, 0x50f36540
Caught exception in looking up space point ptr... 0x50f36540, 0x7110f760
Caught exception in looking up space point ptr... 0x56188190, 0x454a9eb0
Caught exception in looking up space point ptr... 0x56188520, 0x561869f0
Caught exception in looking up space point ptr... 0x561869f0, 0x54350260
Caught exception in looking up space point ptr... 0x54350260, 0x56188190
Caught exception in looking up space point ptr... 0x543a3260, 0xbd43110
Caught exception in looking up space point ptr... 0xbd43110, 0x129c11d0
Caught exception in looking up space point ptr... 0x5a9d4f60, 0x59bb95f0
Caught exception in looking up space point ptr... 0x59bb95f0, 0x45620f30
Caught exception in looking up space point ptr... 0x100fc600, 0x100fc700
Caught exception in looking up space point ptr... 0x100fc700, 0x100fc400
Caught exception in looking up space point ptr... 0x4559a430, 0x5ae52ec0
Caught exception in looking up space point ptr... 0x5ae52ec0, 0x4559a430
Caught exception in looking up space point ptr... 0x5d0445e0, 0x5d044680
Caught exception in looking up space point ptr... 0x5d044680, 0x5d0448b0
Caught exception in looking up space point ptr... 0x11836630, 0x11836c30
Caught exception in looking up space point ptr... 0x11836c30, 0x11836630
Caught exception in looking up space point ptr... 0x5d44b400, 0x5d44a600
Caught exception in looking up space point ptr... 0x5d44a600, 0x5d44b400
Caught exception in looking up space point ptr... 0x1a2f1d20, 0x4ca4d310
Caught exception in looking up space point ptr... 0x11835630, 0x1a2f1d20
Caught exception in looking up space point ptr... 0x470f4ae0, 0xd5ab870
Caught exception in looking up space point ptr... 0xd5ab870, 0x470f4ae0
Caught exception in looking up space point ptr... 0x59cecbb0, 0x60c62ad0
Caught exception in looking up space point ptr... 0x60c62ad0, 0x59cecbb0
Caught exception in looking up space point ptr... 0x13c4b560, 0x13c4b250
Caught exception in looking up space point ptr... 0x13c4b250, 0x13c4b560
Caught exception in looking up space point ptr... 0x36f03470, 0x36f03380
Caught exception in looking up space point ptr... 0x36f03380, 0x36f024b0
Caught exception in looking up space point ptr... 0xb4e24a0, 0x62db9670
Caught exception in looking up space point ptr... 0x62db9670, 0x62db8950
Caught exception in looking up space point ptr... 0x121c0660, 0x3c69e1b0
Caught exception in looking up space point ptr... 0x3c69e1b0, 0x121c0660
++++>>>> total num hits: 59110, num free: 56998
04-Feb-2026 01:35:06 CET  Closed output file "np02vd_raw_run042433_0095_df-s03-d1_dw_0_20260203T191813_reco_stage0_20260204T003506_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         125.049       187.293       240.339       190.612       34.9385        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        6.364e-05     0.0001249    0.000233622   8.9831e-05    5.73561e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                8.07278       9.49751       12.1223       9.45007      0.911444        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0161775      0.04666      0.178192      0.0240306     0.043716        19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000515165   0.00166452     0.0120042    0.000639807   0.00283317       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000488176   0.000705574   0.00121873    0.000598186   0.000199875      19     
produce:beamevent:BeamEvent                      5.8381e-05    0.000150695   0.00090586    0.000112691   0.000179915      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.07665       2.87364       3.77566       2.88555      0.429252        19     
produce:ophit:OpHitFinder                         0.0242081     0.0370389     0.0446462     0.0375754    0.00545911       19     
produce:wclsdatavd:WireCellToolkit                 89.9115       115.638       139.025       117.315       13.6172        19     
produce:gaushit:GausHitFinder                     0.763262       1.89871       3.2367        1.9759       0.638784        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000214002   0.000396058   0.000619927   0.000382684   0.00010797       19     
produce:reco3d:SpacePointSolver                    8.94003       16.1837       25.3142       13.4654       4.66983        19     
produce:hitpdune:DisambigFromSpacePoints          0.0958171     0.252047       0.59832      0.230897      0.0981444       19     
produce:cluster3d:Cluster3D                        1.28741       34.6752       67.2397       45.6213       25.9102        19     
[art]:TriggerResults:TriggerResultInserter       2.5921e-05    3.63926e-05   9.8121e-05     3.175e-05    1.54745e-05      19     
end_path:out1:RootOutput                          3.53e-06     8.10384e-06   3.1961e-05     6.94e-06     5.91392e-06      19     
end_path:out1:RootOutput(write)                    4.13209       6.15369       8.55511       6.06873       1.06686        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5597.52 MB
  Peak resident set size usage (VmHWM): 3609.81 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0095_df-s03-d1_dw_0_20260203T191813_reco_stage0_20260204T003506_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0095_df-s03-d1_dw_0_20260203T191813_reco_stage0_20260204T003506_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0095_df-s03-d1_dw_0_20260203T191813_reco_stage0_20260204T003506_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161700.0,
    "core.end_time": 1770165307.0,
    "core.events": [
      10839,
      10845,
      10851,
      10857,
      10863,
      10869,
      10875,
      10881,
      10887,
      10893,
      10899,
      10905,
      10911,
      10917,
      10923,
      10929,
      10935,
      10941,
      10947
    ],
    "core.event_count": 19,
    "core.first_event_number": 10839,
    "core.last_event_number": 10947,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0095_df-s03-d1_dw_0_20260203T191813.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:36 UTC       justIN version: 01.06.00