justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303656.44@dunegpschedd01.fnal.gov

Jobsub ID303656.44@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:56:25
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce503
Last heartbeat2026-02-04 00:51:10
From worker nodeHostnameb9p02p6824.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:12:00
Input filesvd-protodune:np02vd_raw_run042421_0022_df-s04-d1_dw_0_20260203T134254.hdf5
JobscriptExit code0
Real time36m (2204s)
CPU time35m (2103s = 95%)
Max RSS bytes2591162368 (2471 MiB)
Outputting started2026-02-04 00:48:45
Output files
Finished2026-02-04 00:51:10
Saved logsjustin-logs:303656.44-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

tr... 0x2bc33bb0, 0x2bc33ff0
Caught exception in looking up space point ptr... 0x361616c0, 0x57adfb10
Caught exception in looking up space point ptr... 0x57adfb10, 0x361616c0
Caught exception in looking up space point ptr... 0x28f739e0, 0x28f60d10
Caught exception in looking up space point ptr... 0x28f771a0, 0x28f739e0
Caught exception in looking up space point ptr... 0x4939eb70, 0x493992a0
Caught exception in looking up space point ptr... 0x493992a0, 0x4939b0e0
Caught exception in looking up space point ptr... 0x28deac40, 0x28deaaf0
Caught exception in looking up space point ptr... 0x28deaaf0, 0x28dea9a0
Caught exception in looking up space point ptr... 0x53270f70, 0x532772f0
Caught exception in looking up space point ptr... 0x532772f0, 0x53270f70
Caught exception in looking up space point ptr... 0x2894dd60, 0x2894df00
Caught exception in looking up space point ptr... 0x2894df00, 0x2894dd60
Caught exception in looking up space point ptr... 0x46a5e910, 0x46a5bac0
Caught exception in looking up space point ptr... 0x46a5bac0, 0x46a5e910
Caught exception in looking up space point ptr... 0x1423efb0, 0x1423d4d0
Caught exception in looking up space point ptr... 0x1423d4d0, 0x1423efb0
Caught exception in looking up space point ptr... 0x42333e00, 0x4232d480
Caught exception in looking up space point ptr... 0x4232d480, 0xe954570
Caught exception in looking up space point ptr... 0xebe00c0, 0x4c812c80
Caught exception in looking up space point ptr... 0x4c812c80, 0x3c353600
Caught exception in looking up space point ptr... 0x28df55d0, 0x28df5330
Caught exception in looking up space point ptr... 0x28df5330, 0x28df55d0
Caught exception in looking up space point ptr... 0x36820380, 0x36820680
Caught exception in looking up space point ptr... 0x36820680, 0x36820380
Caught exception in looking up space point ptr... 0x39ff9b60, 0x46a68540
Caught exception in looking up space point ptr... 0x46a647d0, 0x39ff9b60
Caught exception in looking up space point ptr... 0x2d183800, 0x2d17bcc0
Caught exception in looking up space point ptr... 0x2d17bcc0, 0x154610b0
Caught exception in looking up space point ptr... 0x36b950b0, 0x36b953a0
Caught exception in looking up space point ptr... 0x36b953a0, 0x36b950b0
Caught exception in looking up space point ptr... 0x469c4110, 0x3a017960
Caught exception in looking up space point ptr... 0x3a017960, 0x3a0178c0
Caught exception in looking up space point ptr... 0x36b93130, 0x36b91980
Caught exception in looking up space point ptr... 0x36b91980, 0x36b93130
Caught exception in looking up space point ptr... 0x33d5d480, 0x33d5d720
Caught exception in looking up space point ptr... 0x33d5d720, 0x33d5d480
Caught exception in looking up space point ptr... 0x462b1b00, 0x462b31b0
Caught exception in looking up space point ptr... 0x462b31b0, 0x462b3f90
Caught exception in looking up space point ptr... 0x2358b860, 0x2358dfe0
Caught exception in looking up space point ptr... 0x2358dfe0, 0x2358b860
Caught exception in looking up space point ptr... 0x3e6e7e30, 0x3e6e82c0
Caught exception in looking up space point ptr... 0x3e6e82c0, 0x3e6e7fd0
Caught exception in looking up space point ptr... 0x3c66bf00, 0x49713430
Caught exception in looking up space point ptr... 0x49713430, 0x3c66bf00
Caught exception in looking up space point ptr... 0x4912be90, 0x4912c130
Caught exception in looking up space point ptr... 0x4912c130, 0x4912c280
Caught exception in looking up space point ptr... 0x4912c280, 0x4912c130
Caught exception in looking up space point ptr... 0x4912c130, 0x4912c3d0
Caught exception in looking up space point ptr... 0x59e18960, 0x2ebb3e30
Caught exception in looking up space point ptr... 0x2ebb3e30, 0x2a036350
Caught exception in looking up space point ptr... 0x3e6eed80, 0x3e6ee8d0
Caught exception in looking up space point ptr... 0x3e6ee8d0, 0x3e6eed80
Caught exception in looking up space point ptr... 0x85543a0, 0x8550e20
Caught exception in looking up space point ptr... 0x8550e20, 0x85543a0
Caught exception in looking up space point ptr... 0x3c7a4170, 0x57b82b80
Caught exception in looking up space point ptr... 0x57b82b80, 0x3c7a35e0
Caught exception in looking up space point ptr... 0x3c7a35e0, 0x57b82dd0
Caught exception in looking up space point ptr... 0x462bf780, 0x462bfaa0
Caught exception in looking up space point ptr... 0x462bfaa0, 0x462bf910
Caught exception in looking up space point ptr... 0x47cd62d0, 0x47cd64c0
Caught exception in looking up space point ptr... 0x47cd64c0, 0x47cd62d0
Caught exception in looking up space point ptr... 0x248cb3d0, 0x2d35a620
Caught exception in looking up space point ptr... 0x2d35a620, 0x248cb3d0
Caught exception in looking up space point ptr... 0x3a009eb0, 0x3a010af0
Caught exception in looking up space point ptr... 0x3a010af0, 0x3a009eb0
++++>>>> total num hits: 75834, num free: 72707
04-Feb-2026 01:48:36 CET  Closed output file "np02vd_raw_run042421_0022_df-s04-d1_dw_0_20260203T134254_reco_stage0_20260204T004836_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         70.5914       79.3951       91.4186       77.3487       6.16176        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.153e-05    7.98891e-05   0.000293072   6.8521e-05    4.89355e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.7521        4.59073       5.01388       4.64325      0.255608        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00928443     0.0177708      0.06222      0.0127429     0.0128525       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000379343   0.000481755   0.000627125   0.000456774   7.58159e-05      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00143101    0.00167228    0.00256481    0.00162303    0.000226448      27     
produce:beamevent:BeamEvent                       4.968e-05    0.000102209   0.000733136    7.387e-05    0.00012497       27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.93533       2.76426       3.30707       2.85085      0.330657        27     
produce:ophit:OpHitFinder                         0.0277536     0.0365933     0.0511582     0.0360834    0.00556544       27     
produce:wclsdatavd:WireCellToolkit                 49.7437       56.5442       65.0492       56.7337       4.1556         27     
produce:gaushit:GausHitFinder                     0.488604      0.833968       1.29282      0.783585      0.212173        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000140181   0.000224236   0.000364133   0.000215662   5.77261e-05      27     
produce:reco3d:SpacePointSolver                    5.00342       9.5475        16.3869       9.78041       3.10363        27     
produce:hitpdune:DisambigFromSpacePoints          0.0454756     0.081207      0.169925      0.0737353     0.0284509       27     
produce:cluster3d:Cluster3D                       0.709239       1.54925       2.7887        1.4949       0.530589        27     
[art]:TriggerResults:TriggerResultInserter        1.707e-05    2.57398e-05   8.8781e-05     2.281e-05    1.28911e-05      27     
end_path:out1:RootOutput                           3.3e-06     6.75307e-06    3.034e-05     5.03e-06     5.14397e-06      27     
end_path:out1:RootOutput(write)                    2.83635       3.40816       4.63136       3.23171      0.460279        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4557.65 MB
  Peak resident set size usage (VmHWM): 2591.16 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0022_df-s04-d1_dw_0_20260203T134254_reco_stage0_20260204T004836_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0022_df-s04-d1_dw_0_20260203T134254_reco_stage0_20260204T004836_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0022_df-s04-d1_dw_0_20260203T134254_reco_stage0_20260204T004836_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163931.0,
    "core.end_time": 1770166117.0,
    "core.events": [
      3605,
      3611,
      3617,
      3623,
      3629,
      3635,
      3641,
      3647,
      3653,
      3659,
      3665,
      3671,
      3677,
      3683,
      3689,
      3695,
      3701,
      3707,
      3713,
      3719,
      3725,
      3731,
      3737,
      3743,
      3749,
      3755,
      3761
    ],
    "core.event_count": 27,
    "core.first_event_number": 3605,
    "core.last_event_number": 3761,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0022_df-s04-d1_dw_0_20260203T134254.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:03 UTC       justIN version: 01.06.00