justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.150@dunegpschedd01.fnal.gov

Jobsub ID303652.150@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce512
Last heartbeat2026-02-04 00:36:49
From worker nodeHostnameb9g24p4713.cern.ch
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:32:19
Input filesvd-protodune:np02vd_raw_run042433_0608_df-s02-d1_dw_0_20260203T223312.hdf5
JobscriptExit code0
Real time1h (3708s)
CPU time1h (3610s = 97%)
Max RSS bytes3620474880 (3452 MiB)
Outputting started2026-02-04 00:34:09
Output files
Finished2026-02-04 00:36:49
Saved logsjustin-logs:303652.150-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

space point ptr... 0x5358b8c0, 0x4bec0de0
Caught exception in looking up space point ptr... 0x369d4c10, 0x3c25df10
Caught exception in looking up space point ptr... 0x86254a20, 0x369d4c10
Caught exception in looking up space point ptr... 0x36ba8f70, 0x36b977c0
Caught exception in looking up space point ptr... 0x36bba270, 0x36ba8f70
Caught exception in looking up space point ptr... 0x6cf63190, 0x47f655f0
Caught exception in looking up space point ptr... 0x47f655f0, 0x42a8b940
Caught exception in looking up space point ptr... 0x16b84f10, 0x16ba0430
Caught exception in looking up space point ptr... 0x16ba0430, 0x16ba05c0
Caught exception in looking up space point ptr... 0x8256330, 0x8232de0
Caught exception in looking up space point ptr... 0x82564c0, 0x8256330
Caught exception in looking up space point ptr... 0x2d26f930, 0x457ef5c0
Caught exception in looking up space point ptr... 0x2d26fac0, 0x2d26f930
Caught exception in looking up space point ptr... 0x3d9e1420, 0x54c9fcc0
Caught exception in looking up space point ptr... 0x3d9e15b0, 0x3d9e1420
Caught exception in looking up space point ptr... 0x44f29fd0, 0x146d8ec0
Caught exception in looking up space point ptr... 0x146d8ec0, 0x7a5b8900
Caught exception in looking up space point ptr... 0xabe3280, 0xabd4530
Caught exception in looking up space point ptr... 0xabe3410, 0xabe3280
Caught exception in looking up space point ptr... 0xabe2c40, 0xabd3ef0
Caught exception in looking up space point ptr... 0xabe2dd0, 0xabe2c40
Caught exception in looking up space point ptr... 0x19e787b0, 0x6408faa0
Caught exception in looking up space point ptr... 0x19e78940, 0x19e787b0
Caught exception in looking up space point ptr... 0x134159f0, 0x6108c260
Caught exception in looking up space point ptr... 0x13415b80, 0x134159f0
Caught exception in looking up space point ptr... 0x167dd900, 0x167dd770
Caught exception in looking up space point ptr... 0x167dd770, 0x167c62a0
Caught exception in looking up space point ptr... 0x50fbe770, 0x50fa7620
Caught exception in looking up space point ptr... 0x50fbe900, 0x50fbe770
Caught exception in looking up space point ptr... 0x63ebd7e0, 0x456a2670
Caught exception in looking up space point ptr... 0x63ebd970, 0x63ebd7e0
Caught exception in looking up space point ptr... 0x18fc3140, 0x18fb1800
Caught exception in looking up space point ptr... 0x18fb1800, 0x18fc2fb0
Caught exception in looking up space point ptr... 0x4d113720, 0x4d101480
Caught exception in looking up space point ptr... 0x4d124d40, 0x4d113720
Caught exception in looking up space point ptr... 0x13d8ae70, 0x625ead60
Caught exception in looking up space point ptr... 0x625ead60, 0x16c4e070
Caught exception in looking up space point ptr... 0xd4f6560, 0x60c11f90
Caught exception in looking up space point ptr... 0xd5044d0, 0xd4f6560
Caught exception in looking up space point ptr... 0x7792c640, 0x7791b7f0
Caught exception in looking up space point ptr... 0x7791b7f0, 0x77909d20
Caught exception in looking up space point ptr... 0x6c5a3950, 0x6c591cf0
Caught exception in looking up space point ptr... 0x6c591cf0, 0x6c580860
Caught exception in looking up space point ptr... 0xf4417b0, 0xf4347e0
Caught exception in looking up space point ptr... 0xf4347e0, 0xf427810
Caught exception in looking up space point ptr... 0x3c56ef10, 0x3c55cc70
Caught exception in looking up space point ptr... 0x3c55cc70, 0x3c54a840
Caught exception in looking up space point ptr... 0x33c3ad90, 0x33c3b170
Caught exception in looking up space point ptr... 0x33c3b170, 0x33c3ad90
Caught exception in looking up space point ptr... 0x2eeae9a0, 0x85277f80
Caught exception in looking up space point ptr... 0x85277f80, 0x2eeaeaa0
Caught exception in looking up space point ptr... 0x5f4edbf0, 0x4b227f60
Caught exception in looking up space point ptr... 0x2bbfe300, 0x5f4edbf0
Caught exception in looking up space point ptr... 0x3cd09a30, 0x33d8d940
Caught exception in looking up space point ptr... 0x33d8d940, 0x3cd09a30
Caught exception in looking up space point ptr... 0x3cd09a30, 0x2bf47480
Caught exception in looking up space point ptr... 0x2bf47480, 0x3cd09a30
Caught exception in looking up space point ptr... 0x7a76a6b0, 0x70346cf0
Caught exception in looking up space point ptr... 0x70346cf0, 0x7a76a6b0
Caught exception in looking up space point ptr... 0x62ada800, 0x36cc11c0
Caught exception in looking up space point ptr... 0x36cc11c0, 0x3d0df6d0
Caught exception in looking up space point ptr... 0x15dffd70, 0x7773cc90
Caught exception in looking up space point ptr... 0x7773cc90, 0x15dffd70
Caught exception in looking up space point ptr... 0x15dffd70, 0x513b0e00
Caught exception in looking up space point ptr... 0x513b0e00, 0x15dffd70
Caught exception in looking up space point ptr... 0x15dffd70, 0x2d1eb810
Caught exception in looking up space point ptr... 0x2d1eb810, 0x15dffd70
++++>>>> total num hits: 509553, num free: 501953
04-Feb-2026 01:34:00 CET  Closed output file "np02vd_raw_run042433_0608_df-s02-d1_dw_0_20260203T223312_reco_stage0_20260204T003400_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         125.451       191.985       233.982       194.638       26.596         19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.1871e-05    8.18158e-05   0.000158353   6.8871e-05    3.09288e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                4.67624       6.31372       8.22118       5.91953       1.16594        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00936863     0.0166079     0.0726871     0.0127861     0.0136997       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000353547   0.000598774    0.0010386    0.00053378    0.000198465      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000376757   0.000543518   0.00112312    0.00046489    0.000196971      19     
produce:beamevent:BeamEvent                      6.2571e-05    0.000140479   0.000714064   0.000101192   0.000143732      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.09223       2.39888       3.0249        2.34468      0.257146        19     
produce:ophit:OpHitFinder                         0.023349      0.0317725     0.041262       0.03146     0.00464066       19     
produce:wclsdatavd:WireCellToolkit                 101.539       113.676       136.331       113.079       9.67138        19     
produce:gaushit:GausHitFinder                     0.703064       1.85839       2.5527        1.92578      0.446046        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000222774   0.000451765   0.000586692   0.000440819   9.13736e-05      19     
produce:reco3d:SpacePointSolver                    7.00507       15.2644       25.2684       14.6669       4.22551        19     
produce:hitpdune:DisambigFromSpacePoints          0.0853121     0.271554      0.371277      0.282154      0.0665974       19     
produce:cluster3d:Cluster3D                        1.18193       46.0954       70.1305        52.96        19.6349        19     
[art]:TriggerResults:TriggerResultInserter        2.518e-05    3.42491e-05   9.5552e-05     2.888e-05    1.58317e-05      19     
end_path:out1:RootOutput                          5.84e-06     1.17759e-05    3.294e-05     9.421e-06    7.28641e-06      19     
end_path:out1:RootOutput(write)                    4.44421       6.0295        7.50797       6.1446       0.630336        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5567.46 MB
  Peak resident set size usage (VmHWM): 3620.47 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0608_df-s02-d1_dw_0_20260203T223312_reco_stage0_20260204T003400_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0608_df-s02-d1_dw_0_20260203T223312_reco_stage0_20260204T003400_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0608_df-s02-d1_dw_0_20260203T223312_reco_stage0_20260204T003400_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161549.0,
    "core.end_time": 1770165241.0,
    "core.events": [
      69319,
      69325,
      69331,
      69337,
      69343,
      69349,
      69355,
      69361,
      69367,
      69373,
      69379,
      69385,
      69391,
      69397,
      69403,
      69409,
      69415,
      69421,
      69427
    ],
    "core.event_count": 19,
    "core.first_event_number": 69319,
    "core.last_event_number": 69427,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0608_df-s02-d1_dw_0_20260203T223312.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:42:49 UTC       justIN version: 01.06.00