justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.162@dunegpschedd01.fnal.gov

Jobsub ID303652.162@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce512
Last heartbeat2026-02-04 00:22:21
From worker nodeHostnameb9g24p4713.cern.ch
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:33:09
Input filesvd-protodune:np02vd_raw_run042433_0012_df-s02-d1_dw_0_20260203T184641.hdf5
JobscriptExit code0
Real time46m (2810s)
CPU time45m (2734s = 97%)
Max RSS bytes3431292928 (3272 MiB)
Outputting started2026-02-04 00:20:01
Output files
Finished2026-02-04 00:22:21
Saved logsjustin-logs:303652.162-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

in looking up space point ptr... 0x6c86f670, 0x8ee6c00
Caught exception in looking up space point ptr... 0x463146c0, 0x46311f00
Caught exception in looking up space point ptr... 0x462faca0, 0x463146c0
Caught exception in looking up space point ptr... 0x52e46990, 0x52e46a90
Caught exception in looking up space point ptr... 0x52e46a90, 0x5b2436b0
Caught exception in looking up space point ptr... 0x19f50880, 0x62fd3b40
Caught exception in looking up space point ptr... 0x62fd3b40, 0x5aa23540
Caught exception in looking up space point ptr... 0x146efb90, 0x146efe90
Caught exception in looking up space point ptr... 0x146efe90, 0x146efb90
Caught exception in looking up space point ptr... 0x6aa53400, 0x6aa52f50
Caught exception in looking up space point ptr... 0x6aa52f50, 0x6aa53400
Caught exception in looking up space point ptr... 0x103194e0, 0x10319170
Caught exception in looking up space point ptr... 0x10319170, 0x10318e50
Caught exception in looking up space point ptr... 0x5f661480, 0x5f665ed0
Caught exception in looking up space point ptr... 0x5f665ed0, 0x5f661480
Caught exception in looking up space point ptr... 0xeaa6cb0, 0x629f24a0
Caught exception in looking up space point ptr... 0x629e09e0, 0xeaa6cb0
Caught exception in looking up space point ptr... 0x54d5bbd0, 0x54d5b930
Caught exception in looking up space point ptr... 0x54d5b930, 0x54d5bbd0
Caught exception in looking up space point ptr... 0x3452bb90, 0x3452bd20
Caught exception in looking up space point ptr... 0x3452bd20, 0x3452bb90
Caught exception in looking up space point ptr... 0x168325d0, 0x54d82290
Caught exception in looking up space point ptr... 0x168300f0, 0x168325d0
Caught exception in looking up space point ptr... 0x6bcbb850, 0x6bcbb670
Caught exception in looking up space point ptr... 0x6bcbb670, 0x6bcbb850
Caught exception in looking up space point ptr... 0x5f2e2f70, 0x5f2e30e0
Caught exception in looking up space point ptr... 0x5f2e30e0, 0x5f2e2f70
Caught exception in looking up space point ptr... 0x5f2e2f70, 0x5f2e2e20
Caught exception in looking up space point ptr... 0x5f2e2e20, 0x5f2e2f70
Caught exception in looking up space point ptr... 0x55f1fe70, 0x55f1f590
Caught exception in looking up space point ptr... 0x55f1f590, 0x55f1c890
Caught exception in looking up space point ptr... 0x1b8aff80, 0x45607560
Caught exception in looking up space point ptr... 0x45607560, 0xe9ecdc0
Caught exception in looking up space point ptr... 0x1ac61df0, 0x1ac614e0
Caught exception in looking up space point ptr... 0x1ac614e0, 0x1ac615e0
Caught exception in looking up space point ptr... 0x4dee1350, 0x6f2e89a0
Caught exception in looking up space point ptr... 0x6f2e89a0, 0x4dee1350
Caught exception in looking up space point ptr... 0x5b0bbfe0, 0x5b08ff70
Caught exception in looking up space point ptr... 0x5b08ff70, 0x1a710ba0
Caught exception in looking up space point ptr... 0x78878400, 0x7886c880
Caught exception in looking up space point ptr... 0x7886c880, 0x788716f0
Caught exception in looking up space point ptr... 0x470f9080, 0x470f8460
Caught exception in looking up space point ptr... 0x470f8460, 0x470f8590
Caught exception in looking up space point ptr... 0x54c1a6a0, 0x54c02f00
Caught exception in looking up space point ptr... 0x54c02f00, 0x54c02520
Caught exception in looking up space point ptr... 0x6aa76300, 0x6aa7b610
Caught exception in looking up space point ptr... 0x6aa7b610, 0x6aa76300
Caught exception in looking up space point ptr... 0xd937060, 0xd936ab0
Caught exception in looking up space point ptr... 0xd936ab0, 0xd936e60
Caught exception in looking up space point ptr... 0x380ca330, 0x44691a30
Caught exception in looking up space point ptr... 0x44691a30, 0x380ca330
Caught exception in looking up space point ptr... 0x7faaae70, 0x7faac270
Caught exception in looking up space point ptr... 0x7faac270, 0x7faaae70
Caught exception in looking up space point ptr... 0x197f7b90, 0x197f7a40
Caught exception in looking up space point ptr... 0x197f7a40, 0x197f7870
Caught exception in looking up space point ptr... 0x14271310, 0xd92efa0
Caught exception in looking up space point ptr... 0xd92efa0, 0x14271310
Caught exception in looking up space point ptr... 0x6f5f5360, 0x6aa5ce80
Caught exception in looking up space point ptr... 0x6aa5ce80, 0x6f5f5680
Caught exception in looking up space point ptr... 0x94febf0, 0x77c7a100
Caught exception in looking up space point ptr... 0x77c7a100, 0x950d140
Caught exception in looking up space point ptr... 0x950d140, 0x139185b0
Caught exception in looking up space point ptr... 0x139185b0, 0x6038ee40
Caught exception in looking up space point ptr... 0x19c818f0, 0x3492b780
Caught exception in looking up space point ptr... 0x3492b780, 0x346458b0
Caught exception in looking up space point ptr... 0x161b5d80, 0x161add40
Caught exception in looking up space point ptr... 0x161add40, 0x161b5d80
++++>>>> total num hits: 147654, num free: 142470
04-Feb-2026 01:19:52 CET  Closed output file "np02vd_raw_run042433_0012_df-s02-d1_dw_0_20260203T184641_reco_stage0_20260204T001952_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         131.858       144.487       167.965       143.508       9.39179        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.9591e-05    7.9162e-05    0.000159803   7.0012e-05    2.81491e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                4.56559       5.91139       9.40898       5.59461        1.219         19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00743587     0.0136311     0.0443336    0.00957505     0.0099702       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000305346   0.000460246   0.000712194   0.000431429   0.000115499      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000289815   0.000366539    0.0005257    0.000355037   5.37925e-05      19     
produce:beamevent:BeamEvent                      6.1911e-05    0.000117705   0.000652373   8.2622e-05    0.000127584      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.11544       2.48444       3.04243       2.47294      0.232961        19     
produce:ophit:OpHitFinder                         0.0249989     0.0333379     0.0423458     0.0323583    0.00474941       19     
produce:wclsdatavd:WireCellToolkit                 96.2421       103.102       123.691       100.443       7.51401        19     
produce:gaushit:GausHitFinder                     0.956858       1.27181       1.73323        1.265       0.210349        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000391728   0.000474094   0.000704313   0.000456459   7.07955e-05      19     
produce:reco3d:SpacePointSolver                    18.728        23.5145       33.0706       22.6477       3.77979        19     
produce:hitpdune:DisambigFromSpacePoints          0.161347      0.217635       0.35419      0.185602      0.0549723       19     
produce:cluster3d:Cluster3D                        1.78571       2.74199       4.13549       2.59185      0.629584        19     
[art]:TriggerResults:TriggerResultInserter        2.413e-05    3.25228e-05   8.7822e-05    2.7591e-05    1.40115e-05      19     
end_path:out1:RootOutput                          3.66e-06     7.94658e-06    2.369e-05     7.61e-06     5.51158e-06      19     
end_path:out1:RootOutput(write)                     4.677        5.16789       6.6527        5.04527      0.513207        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5378.52 MB
  Peak resident set size usage (VmHWM): 3431.29 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0012_df-s02-d1_dw_0_20260203T184641_reco_stage0_20260204T001952_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0012_df-s02-d1_dw_0_20260203T184641_reco_stage0_20260204T001952_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0012_df-s02-d1_dw_0_20260203T184641_reco_stage0_20260204T001952_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161601.0,
    "core.end_time": 1770164393.0,
    "core.events": [
      1375,
      1381,
      1387,
      1393,
      1399,
      1405,
      1411,
      1417,
      1423,
      1429,
      1435,
      1441,
      1447,
      1453,
      1459,
      1465,
      1471,
      1477,
      1483
    ],
    "core.event_count": 19,
    "core.first_event_number": 1375,
    "core.last_event_number": 1483,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0012_df-s02-d1_dw_0_20260203T184641.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:15 UTC       justIN version: 01.06.00