justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.147@dunegpschedd02.fnal.gov

Jobsub ID297921.147@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce504
Last heartbeat2026-02-04 01:06:00
From worker nodeHostnameb9p06p4022.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:55:54
Input filesvd-protodune:np02vd_raw_run042433_0529_df-s02-d1_dw_0_20260203T220309.hdf5
JobscriptExit code0
Real time1h (4062s)
CPU time1h (3867s = 95%)
Max RSS bytes3369590784 (3213 MiB)
Outputting started2026-02-04 01:03:38
Output files
Finished2026-02-04 01:05:59
Saved logsjustin-logs:297921.147-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

. 0x5f490440, 0x152fee30
Caught exception in looking up space point ptr... 0x152fee30, 0x152fe980
Caught exception in looking up space point ptr... 0x1a8584f0, 0x1a8586d0
Caught exception in looking up space point ptr... 0x1a8586d0, 0x1a8584f0
Caught exception in looking up space point ptr... 0x31f522e0, 0x3fa4b500
Caught exception in looking up space point ptr... 0x3fa4b500, 0x31f522e0
Caught exception in looking up space point ptr... 0x31f39060, 0x2c903e20
Caught exception in looking up space point ptr... 0x2c903e20, 0x20ad7270
Caught exception in looking up space point ptr... 0x662e52a0, 0x662e4f80
Caught exception in looking up space point ptr... 0x662e4f80, 0x662e52a0
Caught exception in looking up space point ptr... 0x662e52a0, 0x662e5110
Caught exception in looking up space point ptr... 0x662e5110, 0x662e52a0
Caught exception in looking up space point ptr... 0x326ae3d0, 0x326ac910
Caught exception in looking up space point ptr... 0x326ac910, 0x326ae3d0
Caught exception in looking up space point ptr... 0x4ca8dcf0, 0x4ca8df90
Caught exception in looking up space point ptr... 0x4ca8df90, 0x4ca8dcf0
Caught exception in looking up space point ptr... 0x3f71c8c0, 0x3f71c5a0
Caught exception in looking up space point ptr... 0x3f71c5a0, 0x3f71c8c0
Caught exception in looking up space point ptr... 0x3f71c8c0, 0x3f71c3c0
Caught exception in looking up space point ptr... 0x3f71c3c0, 0x3f71c8c0
Caught exception in looking up space point ptr... 0x698337c0, 0x69833310
Caught exception in looking up space point ptr... 0x69833310, 0x698337c0
Caught exception in looking up space point ptr... 0x6d5a9650, 0x6d5a94c0
Caught exception in looking up space point ptr... 0x6d5a94c0, 0x6d5a9650
Caught exception in looking up space point ptr... 0x51c381d0, 0x51c382d0
Caught exception in looking up space point ptr... 0x51c382d0, 0x51c384d0
Caught exception in looking up space point ptr... 0x10967300, 0x10967170
Caught exception in looking up space point ptr... 0x10967170, 0x10966e50
Caught exception in looking up space point ptr... 0x10966e50, 0x10966fe0
Caught exception in looking up space point ptr... 0x3440bbb0, 0x3440b940
Caught exception in looking up space point ptr... 0x3440b940, 0x3440bbb0
Caught exception in looking up space point ptr... 0x6b2058c0, 0x6b205770
Caught exception in looking up space point ptr... 0x6b205770, 0x6b205620
Caught exception in looking up space point ptr... 0x52db8110, 0x52db3980
Caught exception in looking up space point ptr... 0x52db3980, 0x52db8110
Caught exception in looking up space point ptr... 0x2b471400, 0x2b470890
Caught exception in looking up space point ptr... 0x2b470890, 0x2b479520
Caught exception in looking up space point ptr... 0x34947fb0, 0x4bcb0810
Caught exception in looking up space point ptr... 0x4bcb0810, 0x34947fb0
Caught exception in looking up space point ptr... 0x336d6ad0, 0x336d6940
Caught exception in looking up space point ptr... 0x336d6940, 0x336d6ad0
Caught exception in looking up space point ptr... 0x33b9b3d0, 0x33b9af20
Caught exception in looking up space point ptr... 0x33b9af20, 0x33b9b3d0
Caught exception in looking up space point ptr... 0x662ef220, 0x662ef0f0
Caught exception in looking up space point ptr... 0x662ef0f0, 0x662ef220
Caught exception in looking up space point ptr... 0x336d48c0, 0x336d4730
Caught exception in looking up space point ptr... 0x336d4730, 0x336d48c0
Caught exception in looking up space point ptr... 0x375953a0, 0x37594b80
Caught exception in looking up space point ptr... 0x37594b80, 0x37593b60
Caught exception in looking up space point ptr... 0x336da900, 0x336daa90
Caught exception in looking up space point ptr... 0x336daa90, 0x336da900
Caught exception in looking up space point ptr... 0x336d87e0, 0x336d8e20
Caught exception in looking up space point ptr... 0x336d8e20, 0x336d87e0
Caught exception in looking up space point ptr... 0x336d87e0, 0x336d8970
Caught exception in looking up space point ptr... 0x336d8970, 0x336d87e0
Caught exception in looking up space point ptr... 0x2aa9cf50, 0x2f30dc70
Caught exception in looking up space point ptr... 0x2f30dc70, 0x4795ca60
Caught exception in looking up space point ptr... 0x336dbfd0, 0x336dbcb0
Caught exception in looking up space point ptr... 0x336dbcb0, 0x336dbe40
Caught exception in looking up space point ptr... 0x4181e170, 0x4181e210
Caught exception in looking up space point ptr... 0x4181e210, 0x400e8310
Caught exception in looking up space point ptr... 0x10956560, 0x10956880
Caught exception in looking up space point ptr... 0x10956880, 0x10956560
Caught exception in looking up space point ptr... 0x3af8fbc0, 0x31f3c330
Caught exception in looking up space point ptr... 0x31f3c330, 0x34a26700
Caught exception in looking up space point ptr... 0x94802e0, 0x6b207b20
Caught exception in looking up space point ptr... 0x6b207b20, 0x94802e0
++++>>>> total num hits: 121146, num free: 116305
04-Feb-2026 02:03:30 CET  Closed output file "np02vd_raw_run042433_0529_df-s02-d1_dw_0_20260203T220309_reco_stage0_20260204T010330_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         124.237       207.617       246.571       224.184       38.5904        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.5851e-05    0.000243357   0.000820592   0.000179713   0.000187383      19     
produce:tpcrawdecoder:PDVDTPCReader                6.95988       10.6094       14.8261       10.5548       1.61942        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0153367     0.109388      0.299489      0.0813215     0.0804396       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000608049     0.0127       0.106249     0.00485001     0.0242517       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000438616   0.00124667    0.00231055    0.00121153    0.000538257      19     
produce:beamevent:BeamEvent                      6.5131e-05    0.000287281   0.00124716    0.000229513   0.000253411      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.26263       3.45902       4.18446       3.5312       0.496111        19     
produce:ophit:OpHitFinder                         0.0318838     0.0506019     0.0868025     0.0479202     0.0132591       19     
produce:wclsdatavd:WireCellToolkit                 89.8832       162.043       197.48        174.294       31.7841        19     
produce:gaushit:GausHitFinder                      1.02339       2.00047       2.78048       1.94765      0.581077        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000242763   0.00183635    0.00424755    0.00204287    0.00131346       19     
produce:reco3d:SpacePointSolver                    11.3294       19.0791       27.1444       19.9961       4.83027        19     
produce:hitpdune:DisambigFromSpacePoints          0.119788      0.279303       0.45919      0.303853      0.0960318       19     
produce:cluster3d:Cluster3D                        1.45853       3.6997        7.13262       3.57786       1.48758        19     
[art]:TriggerResults:TriggerResultInserter       2.1031e-05    7.04779e-05   0.000224334   5.6371e-05    5.22396e-05      19     
end_path:out1:RootOutput                          8.331e-06    2.82735e-05   0.000133392    2.046e-05    2.89787e-05      19     
end_path:out1:RootOutput(write)                    4.19874       6.19596       7.07195       6.41359       0.86379        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5346.44 MB
  Peak resident set size usage (VmHWM): 3369.59 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0529_df-s02-d1_dw_0_20260203T220309_reco_stage0_20260204T010330_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0529_df-s02-d1_dw_0_20260203T220309_reco_stage0_20260204T010330_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0529_df-s02-d1_dw_0_20260203T220309_reco_stage0_20260204T010330_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162977.0,
    "core.end_time": 1770167012.0,
    "core.events": [
      60313,
      60319,
      60325,
      60331,
      60337,
      60343,
      60349,
      60355,
      60361,
      60367,
      60373,
      60379,
      60385,
      60391,
      60397,
      60403,
      60409,
      60415,
      60421
    ],
    "core.event_count": 19,
    "core.first_event_number": 60313,
    "core.last_event_number": 60421,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0529_df-s02-d1_dw_0_20260203T220309.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:44:12 UTC       justIN version: 01.06.00