justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.6@dunegpschedd01.fnal.gov

Jobsub ID303653.6@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce514
Last heartbeat2026-02-04 00:22:49
From worker nodeHostnameb9p08p6650.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:35:48
Input filesvd-protodune:np02vd_raw_run042421_0087_df-s02-d1_dw_0_20260203T141802.hdf5
JobscriptExit code0
Real time44m (2673s)
CPU time42m (2561s = 95%)
Max RSS bytes2591232000 (2471 MiB)
Outputting started2026-02-04 00:20:22
Output files
Finished2026-02-04 00:22:49
Saved logsjustin-logs:303653.6-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

*********************************************************************
******************************************************************************************************************
****> vertexList containted 0 vertices
Caught exception in looking up space point ptr... 0x4dbdc2e0, 0x2a4759c0
Caught exception in looking up space point ptr... 0x2a4759c0, 0x2ee7adb0
Caught exception in looking up space point ptr... 0x2ee7adb0, 0x2ee7b0d0
Caught exception in looking up space point ptr... 0x2ee7b0d0, 0x2a475f60
Caught exception in looking up space point ptr... 0x2a475f60, 0x4dbdc4e0
Caught exception in looking up space point ptr... 0x5b972b90, 0x28970ac0
Caught exception in looking up space point ptr... 0x28970ac0, 0x58489590
Caught exception in looking up space point ptr... 0x58489590, 0x58489430
Caught exception in looking up space point ptr... 0x2edc2db0, 0x5b972b90
Caught exception in looking up space point ptr... 0x2b2cb900, 0x2b2cae20
Caught exception in looking up space point ptr... 0x2b2cae20, 0x14dc9b70
Caught exception in looking up space point ptr... 0x14dc9b70, 0x14dc71e0
Caught exception in looking up space point ptr... 0x14dc71e0, 0x14dc9850
Caught exception in looking up space point ptr... 0x14dc9850, 0x14dc99e0
Caught exception in looking up space point ptr... 0x14dc99e0, 0x2b2cb570
Caught exception in looking up space point ptr... 0x2b2cb570, 0x2b2cb900
Caught exception in looking up space point ptr... 0x10792330, 0x1078fa30
Caught exception in looking up space point ptr... 0x1078fa30, 0x1078f710
Caught exception in looking up space point ptr... 0x1078f710, 0x1078f580
Caught exception in looking up space point ptr... 0x1078f580, 0x107884c0
Caught exception in looking up space point ptr... 0x10792480, 0x10792330
Caught exception in looking up space point ptr... 0x4305bf30, 0x4305cbf0
Caught exception in looking up space point ptr... 0x4305cbf0, 0x4305bf30
Caught exception in looking up space point ptr... 0x1347b8d0, 0x13477080
Caught exception in looking up space point ptr... 0x13477080, 0x197fc4f0
Caught exception in looking up space point ptr... 0x197fc4f0, 0x13484940
Caught exception in looking up space point ptr... 0x13484940, 0x13484a40
Caught exception in looking up space point ptr... 0x13484b40, 0x1347b8d0
Caught exception in looking up space point ptr... 0x441755f0, 0x567ccc20
Caught exception in looking up space point ptr... 0xf4bb120, 0x441755f0
Caught exception in looking up space point ptr... 0x153bb590, 0x2a99cfd0
Caught exception in looking up space point ptr... 0x14accfd0, 0x153bb590
Caught exception in looking up space point ptr... 0x14ecf610, 0x15100990
Caught exception in looking up space point ptr... 0x14ecd470, 0x14ecf610
Caught exception in looking up space point ptr... 0x28f5de90, 0x153f1ef0
Caught exception in looking up space point ptr... 0x4e4a5820, 0x28f5de90
Caught exception in looking up space point ptr... 0x9664000, 0x9669600
Caught exception in looking up space point ptr... 0x9669600, 0x5acad580
Caught exception in looking up space point ptr... 0x4aad1620, 0xd6c7740
Caught exception in looking up space point ptr... 0xd6c7740, 0xd6c78d0
Caught exception in looking up space point ptr... 0xd6c78d0, 0x4751d7e0
Caught exception in looking up space point ptr... 0x58475fd0, 0x4aad1620
Caught exception in looking up space point ptr... 0x2adbb040, 0x2adbb1d0
Caught exception in looking up space point ptr... 0x2adbb1d0, 0x2adbb040
Caught exception in looking up space point ptr... 0xf4c9430, 0xf4c9270
Caught exception in looking up space point ptr... 0xf4c9270, 0x567c7340
Caught exception in looking up space point ptr... 0x3350eb10, 0x2c7b8910
Caught exception in looking up space point ptr... 0x2c7b8910, 0x313f7110
Caught exception in looking up space point ptr... 0x4659e670, 0x584723e0
Caught exception in looking up space point ptr... 0x584723e0, 0x3d70d1f0
Caught exception in looking up space point ptr... 0x3d703f30, 0x4737fba0
Caught exception in looking up space point ptr... 0x2f2994a0, 0x3d703f30
Caught exception in looking up space point ptr... 0x1347dd00, 0x2cb14480
Caught exception in looking up space point ptr... 0x1347dab0, 0x1347dd00
Caught exception in looking up space point ptr... 0x34c9f6a0, 0x2cb19120
Caught exception in looking up space point ptr... 0x2cb19120, 0x34c9f6a0
Caught exception in looking up space point ptr... 0x11635280, 0x116350f0
Caught exception in looking up space point ptr... 0x116350f0, 0x4400c400
Caught exception in looking up space point ptr... 0x4754f490, 0x5938b920
Caught exception in looking up space point ptr... 0x5938b920, 0x4754f7a0
Caught exception in looking up space point ptr... 0x2cb10df0, 0x5010a3c0
Caught exception in looking up space point ptr... 0x5010b870, 0x2cb10df0
++++>>>> total num hits: 45547, num free: 44031
04-Feb-2026 01:20:13 CET  Closed output file "np02vd_raw_run042421_0087_df-s02-d1_dw_0_20260203T141802_reco_stage0_20260204T002013_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         85.9345       96.4219       110.212       95.9793       6.21806        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.9961e-05    0.000103206   0.000188782   8.7301e-05    3.42289e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                4.30593       4.67674       5.13131       4.73734      0.238259        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0104029     0.013751      0.0324395     0.012438     0.00437532       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000403675   0.000575969   0.00159036    0.000546896   0.000213078      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00159539    0.00197269    0.00282892    0.00190239    0.000298592      27     
produce:beamevent:BeamEvent                      6.5921e-05    0.000135332   0.00082013    0.000105972   0.000137828      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.4372        3.01633       3.42271       3.04483      0.252872        27     
produce:ophit:OpHitFinder                         0.0292092     0.0394807     0.0452421     0.0403637      0.00402        27     
produce:wclsdatavd:WireCellToolkit                 67.1519       71.5791       77.0795       70.9348       3.09978        27     
produce:gaushit:GausHitFinder                     0.673159       1.03248       1.68835      0.926745      0.253679        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000176663   0.00027717    0.000549366   0.000265143   7.97132e-05      27     
produce:reco3d:SpacePointSolver                    5.13967       9.77036       18.8803       8.77858       3.46973        27     
produce:hitpdune:DisambigFromSpacePoints          0.0515495     0.107205      0.178506      0.100442      0.0378626       27     
produce:cluster3d:Cluster3D                       0.774676       1.92465       3.30864       1.72691      0.698236        27     
[art]:TriggerResults:TriggerResultInserter       2.2721e-05    3.52859e-05   9.2021e-05    3.3491e-05    1.32404e-05      27     
end_path:out1:RootOutput                          3.55e-06     7.28259e-06    2.429e-05     7.22e-06     3.71201e-06      27     
end_path:out1:RootOutput(write)                    3.22228       4.23592       5.13159       4.28423       0.4976         27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4561.67 MB
  Peak resident set size usage (VmHWM): 2591.23 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0087_df-s02-d1_dw_0_20260203T141802_reco_stage0_20260204T002013_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0087_df-s02-d1_dw_0_20260203T141802_reco_stage0_20260204T002013_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0087_df-s02-d1_dw_0_20260203T141802_reco_stage0_20260204T002013_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161761.0,
    "core.end_time": 1770164415.0,
    "core.events": [
      14149,
      14155,
      14161,
      14167,
      14173,
      14179,
      14185,
      14191,
      14197,
      14203,
      14209,
      14215,
      14221,
      14227,
      14233,
      14239,
      14245,
      14251,
      14257,
      14263,
      14269,
      14275,
      14281,
      14287,
      14293,
      14299,
      14305
    ],
    "core.event_count": 27,
    "core.first_event_number": 14149,
    "core.last_event_number": 14305,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0087_df-s02-d1_dw_0_20260203T141802.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:45:02 UTC       justIN version: 01.06.00