justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.147@dunegpschedd01.fnal.gov

Jobsub ID303652.147@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce510
Last heartbeat2026-02-04 00:35:48
From worker nodeHostnameb9p09p3861.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:32:25
Input filesvd-protodune:np02vd_raw_run042433_0116_df-s03-d1_dw_0_20260203T192612.hdf5
JobscriptExit code0
Real time1h (3630s)
CPU time58m (3513s = 96%)
Max RSS bytes3609681920 (3442 MiB)
Outputting started2026-02-04 00:32:58
Output files
Finished2026-02-04 00:35:48
Saved logsjustin-logs:303652.147-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

pace point ptr... 0x720d90f0, 0x16274af0
Caught exception in looking up space point ptr... 0x3ad7f300, 0x698d54a0
Caught exception in looking up space point ptr... 0x698d54a0, 0x7ff89950
Caught exception in looking up space point ptr... 0x3aeb50f0, 0x585e0f90
Caught exception in looking up space point ptr... 0x585e0f90, 0x585e6840
Caught exception in looking up space point ptr... 0x74b12780, 0x74b124f0
Caught exception in looking up space point ptr... 0x74b124f0, 0x74b1e020
Caught exception in looking up space point ptr... 0xaffd110, 0xaffcec0
Caught exception in looking up space point ptr... 0xaffcec0, 0x4bd5ba30
Caught exception in looking up space point ptr... 0x2c2bf730, 0x15854e20
Caught exception in looking up space point ptr... 0x59c7fa90, 0x2c2bf730
Caught exception in looking up space point ptr... 0xfb26cd0, 0xfb148a0
Caught exception in looking up space point ptr... 0xfb26e60, 0xfb26cd0
Caught exception in looking up space point ptr... 0x2c7ccdb0, 0x2c7da970
Caught exception in looking up space point ptr... 0x2c7cd080, 0x2c7ccdb0
Caught exception in looking up space point ptr... 0x37226fa0, 0x8c55fd0
Caught exception in looking up space point ptr... 0x8c55fd0, 0x8c55e40
Caught exception in looking up space point ptr... 0x181d8d40, 0x181e1eb0
Caught exception in looking up space point ptr... 0x181d9010, 0x181d8d40
Caught exception in looking up space point ptr... 0x127e3a80, 0x127e3c10
Caught exception in looking up space point ptr... 0x127e3c10, 0x127e3da0
Caught exception in looking up space point ptr... 0x466556f0, 0x2c002760
Caught exception in looking up space point ptr... 0x2c002760, 0x2c0025d0
Caught exception in looking up space point ptr... 0x11a5b5a0, 0xb9dd090
Caught exception in looking up space point ptr... 0xb9dd090, 0x11a5b5a0
Caught exception in looking up space point ptr... 0x43c90300, 0x43b6b1d0
Caught exception in looking up space point ptr... 0x43b6b1d0, 0x43c90300
Caught exception in looking up space point ptr... 0x37a436e0, 0x37a55f80
Caught exception in looking up space point ptr... 0x37a55f80, 0x37a56110
Caught exception in looking up space point ptr... 0x32554770, 0x32554810
Caught exception in looking up space point ptr... 0x613cb860, 0x32554770
Caught exception in looking up space point ptr... 0x7378e7f0, 0x7378f7d0
Caught exception in looking up space point ptr... 0x73739980, 0x7378e7f0
Caught exception in looking up space point ptr... 0x46fb88d0, 0x16cf0050
Caught exception in looking up space point ptr... 0x46fd8e60, 0x46fb88d0
Caught exception in looking up space point ptr... 0x19592680, 0x42f77b90
Caught exception in looking up space point ptr... 0x42f77b90, 0x4749c9d0
Caught exception in looking up space point ptr... 0x6942f680, 0x1454ceb0
Caught exception in looking up space point ptr... 0x1454ceb0, 0x693c4500
Caught exception in looking up space point ptr... 0x35d9fd60, 0x49953610
Caught exception in looking up space point ptr... 0x49953610, 0x499589d0
Caught exception in looking up space point ptr... 0x1acb1670, 0x1ac9f880
Caught exception in looking up space point ptr... 0x1acc3f50, 0x1acb1670
Caught exception in looking up space point ptr... 0x43b8b150, 0x43b8b270
Caught exception in looking up space point ptr... 0x43b8b270, 0x43b8b150
Caught exception in looking up space point ptr... 0x43b8b150, 0x797a8970
Caught exception in looking up space point ptr... 0x797a8970, 0x43b8b150
Caught exception in looking up space point ptr... 0x16373500, 0x4cb99870
Caught exception in looking up space point ptr... 0x4cb9ab40, 0x16373500
Caught exception in looking up space point ptr... 0x68738b00, 0x4cb7a3f0
Caught exception in looking up space point ptr... 0x4cb7a3f0, 0x29ebca40
Caught exception in looking up space point ptr... 0x3ef285c0, 0x3ef28750
Caught exception in looking up space point ptr... 0x3ef28750, 0x3ef285c0
Caught exception in looking up space point ptr... 0xe92d290, 0xe92ce10
Caught exception in looking up space point ptr... 0xe92ce10, 0xe92d290
Caught exception in looking up space point ptr... 0x105bdac0, 0x4cc94b00
Caught exception in looking up space point ptr... 0x4cc94b00, 0x4cc94c60
Caught exception in looking up space point ptr... 0x55d5d890, 0x55d5dd10
Caught exception in looking up space point ptr... 0x55d5dd10, 0x55d5d980
Caught exception in looking up space point ptr... 0x4cc19dc0, 0x4cc19b70
Caught exception in looking up space point ptr... 0x4cc19b70, 0x4cc19dc0
Caught exception in looking up space point ptr... 0x461778c0, 0x5ef09970
Caught exception in looking up space point ptr... 0x5ef09970, 0x461778c0
Caught exception in looking up space point ptr... 0x9d26320, 0x29f158e0
Caught exception in looking up space point ptr... 0x29f158e0, 0x9d26320
Caught exception in looking up space point ptr... 0x416c9a40, 0x416c9950
Caught exception in looking up space point ptr... 0x416c9950, 0x11f6ccb0
++++>>>> total num hits: 516135, num free: 507659
04-Feb-2026 01:32:47 CET  Closed output file "np02vd_raw_run042433_0116_df-s03-d1_dw_0_20260203T192612_reco_stage0_20260204T003247_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         131.045       187.693       224.844       193.334       22.8402        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.4281e-05    0.000132646   0.000398145   0.000106842   8.38091e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.53917       6.93396       14.7462       6.63549       1.88316        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0115126     0.0263553     0.138747      0.015364      0.0281122       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000433417   0.000609562   0.00168759    0.000525638    0.0002693       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000408006   0.000531742   0.000853285   0.000490427   0.000112058      19     
produce:beamevent:BeamEvent                      6.6041e-05    0.000142254   0.000926144   9.6212e-05    0.000185752      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.21551       2.85076       3.37265       2.90038      0.349494        19     
produce:ophit:OpHitFinder                         0.0271505     0.0378071     0.0442357     0.0391119    0.00458584       19     
produce:wclsdatavd:WireCellToolkit                 93.5047       108.727       121.186       110.405       7.16267        19     
produce:gaushit:GausHitFinder                     0.733975       2.11572       3.27546       2.12385      0.569654        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000238214   0.000378223   0.000501939   0.000375896   6.21478e-05      19     
produce:reco3d:SpacePointSolver                    9.42496       18.5534       35.0093        17.94        6.55238        19     
produce:hitpdune:DisambigFromSpacePoints          0.134047      0.291908      0.556546      0.231591      0.115124        19     
produce:cluster3d:Cluster3D                        1.5021        41.8533       63.6404       44.9162       18.3886        19     
[art]:TriggerResults:TriggerResultInserter       2.4391e-05    3.91196e-05   0.000111702   3.4671e-05    1.77463e-05      19     
end_path:out1:RootOutput                          4.17e-06     8.18532e-06   3.4521e-05     6.56e-06     6.38045e-06      19     
end_path:out1:RootOutput(write)                    4.39884       6.27355       8.41244       5.95237       1.07044        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5583.62 MB
  Peak resident set size usage (VmHWM): 3609.68 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0116_df-s03-d1_dw_0_20260203T192612_reco_stage0_20260204T003247_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0116_df-s03-d1_dw_0_20260203T192612_reco_stage0_20260204T003247_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0116_df-s03-d1_dw_0_20260203T192612_reco_stage0_20260204T003247_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161558.0,
    "core.end_time": 1770165168.0,
    "core.events": [
      13233,
      13239,
      13245,
      13251,
      13257,
      13263,
      13269,
      13275,
      13281,
      13287,
      13293,
      13299,
      13305,
      13311,
      13317,
      13323,
      13329,
      13335,
      13341
    ],
    "core.event_count": 19,
    "core.first_event_number": 13233,
    "core.last_event_number": 13341,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0116_df-s03-d1_dw_0_20260203T192612.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:58:03 UTC       justIN version: 01.06.00