justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.132@dunegpschedd02.fnal.gov

Jobsub ID297923.132@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce504
Last heartbeat2026-02-04 00:48:54
From worker nodeHostnameb9p12p0289.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:04:39
Input filesvd-protodune:np02vd_raw_run042433_0304_df-s04-d0_dw_0_20260203T203739.hdf5
JobscriptExit code0
Real time41m (2515s)
CPU time39m (2399s = 95%)
Max RSS bytes3412566016 (3254 MiB)
Outputting started2026-02-04 00:46:35
Output files
Finished2026-02-04 00:48:54
Saved logsjustin-logs:297923.132-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

.. 0x5f63cab0, 0x5f63cc40
Caught exception in looking up space point ptr... 0x6a0264f0, 0x6a026790
Caught exception in looking up space point ptr... 0x6a026790, 0x6a026ab0
Caught exception in looking up space point ptr... 0x3c5308e0, 0x8c86760
Caught exception in looking up space point ptr... 0x8c86760, 0x8c8c290
Caught exception in looking up space point ptr... 0x15e6fc10, 0x15e6f8f0
Caught exception in looking up space point ptr... 0x15e6f8f0, 0x15e6fc10
Caught exception in looking up space point ptr... 0x4614ad60, 0x46149fa0
Caught exception in looking up space point ptr... 0x46149fa0, 0x4614a160
Caught exception in looking up space point ptr... 0x3b920130, 0x3b9207d0
Caught exception in looking up space point ptr... 0x3b9207d0, 0x3b920130
Caught exception in looking up space point ptr... 0x710110d0, 0x71010520
Caught exception in looking up space point ptr... 0x71010520, 0x71010020
Caught exception in looking up space point ptr... 0x16b141e0, 0x16b0a450
Caught exception in looking up space point ptr... 0x16b0a450, 0x16b140b0
Caught exception in looking up space point ptr... 0x83204d60, 0x83205060
Caught exception in looking up space point ptr... 0x83205060, 0x83204ed0
Caught exception in looking up space point ptr... 0x3b7dfb10, 0x3b7de000
Caught exception in looking up space point ptr... 0x3b7de000, 0x3b7dfb10
Caught exception in looking up space point ptr... 0x3b7dfb10, 0x3b7de140
Caught exception in looking up space point ptr... 0x3b7de140, 0x3b7dfb10
Caught exception in looking up space point ptr... 0x493fbb20, 0x493fad00
Caught exception in looking up space point ptr... 0x493fad00, 0x493fbb20
Caught exception in looking up space point ptr... 0x493fe5b0, 0x493fd000
Caught exception in looking up space point ptr... 0x493fd000, 0x493fe5b0
Caught exception in looking up space point ptr... 0x5c2b51d0, 0x304236b0
Caught exception in looking up space point ptr... 0x304236b0, 0x5c2b51d0
Caught exception in looking up space point ptr... 0x5c5d2900, 0x5c5c2fa0
Caught exception in looking up space point ptr... 0x5c5c2fa0, 0x5c5d4b70
Caught exception in looking up space point ptr... 0x5c2a2d90, 0x5c2a2a70
Caught exception in looking up space point ptr... 0x5c2a2a70, 0x5c2a2d90
Caught exception in looking up space point ptr... 0x2bc2d340, 0x2bc2d490
Caught exception in looking up space point ptr... 0x2bc2d490, 0x2bc2d340
Caught exception in looking up space point ptr... 0x41449e00, 0x41459820
Caught exception in looking up space point ptr... 0x41459820, 0x4145c170
Caught exception in looking up space point ptr... 0x1a916150, 0x1a913360
Caught exception in looking up space point ptr... 0x1a913360, 0x1a916150
Caught exception in looking up space point ptr... 0x5c5b5560, 0x5c5b52c0
Caught exception in looking up space point ptr... 0x5c5b52c0, 0x5c5b5560
Caught exception in looking up space point ptr... 0x5c5b5560, 0x5c5abed0
Caught exception in looking up space point ptr... 0x5c5abed0, 0x5c5b5560
Caught exception in looking up space point ptr... 0x3fc9e7c0, 0x3fc9ec70
Caught exception in looking up space point ptr... 0x3fc9ec70, 0x3fc9e7c0
Caught exception in looking up space point ptr... 0x3fc9e7c0, 0x3fc9eae0
Caught exception in looking up space point ptr... 0x3fc9eae0, 0x3fc9e7c0
Caught exception in looking up space point ptr... 0x5c2a2750, 0x5c2a25c0
Caught exception in looking up space point ptr... 0x5c2a25c0, 0x5c2a2750
Caught exception in looking up space point ptr... 0x628100f0, 0x62810410
Caught exception in looking up space point ptr... 0x62810410, 0x628100f0
Caught exception in looking up space point ptr... 0x1a3cb3e0, 0x1a3cfc80
Caught exception in looking up space point ptr... 0x1a3cfc80, 0x1a3cb3e0
Caught exception in looking up space point ptr... 0x5fec5930, 0x7620e2f0
Caught exception in looking up space point ptr... 0x7620e2f0, 0x5fec5930
Caught exception in looking up space point ptr... 0x6e05e190, 0x5b12f060
Caught exception in looking up space point ptr... 0x5b12f060, 0x6e05e190
Caught exception in looking up space point ptr... 0x1884a9d0, 0x1884a860
Caught exception in looking up space point ptr... 0x1884a860, 0x1884a9d0
Caught exception in looking up space point ptr... 0x3c2bb450, 0x3fc9dd60
Caught exception in looking up space point ptr... 0x3fc9dd60, 0x3c2bb450
Caught exception in looking up space point ptr... 0x3c2bb450, 0x3c2bb5a0
Caught exception in looking up space point ptr... 0x3c2bb5a0, 0x3c2bb450
Caught exception in looking up space point ptr... 0x5fefb650, 0x5fefb7e0
Caught exception in looking up space point ptr... 0x5fefb7e0, 0x5fefb650
Caught exception in looking up space point ptr... 0x3c2b35b0, 0x3c2b9770
Caught exception in looking up space point ptr... 0x3c2b9770, 0x3c2b35b0
Caught exception in looking up space point ptr... 0x3c2b35b0, 0x666460b0
Caught exception in looking up space point ptr... 0x666460b0, 0x3c2b35b0
++++>>>> total num hits: 118332, num free: 113125
04-Feb-2026 01:46:28 CET  Closed output file "np02vd_raw_run042433_0304_df-s04-d0_dw_0_20260203T203739_reco_stage0_20260204T004628_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         114.191       129.533       144.001       132.646       9.19044        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.5111e-05    7.90966e-05   0.000170521   6.8691e-05    3.01582e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                6.62702       7.6255        8.41995       7.70304      0.447041        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0116753     0.0505844     0.193524      0.0168113     0.0584012       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000384231   0.00052454    0.000810083   0.000516141   9.46992e-05      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000357101   0.000491318   0.000735663   0.000468581   8.61999e-05      19     
produce:beamevent:BeamEvent                       5.941e-05    0.00014052    0.000860052   9.5701e-05    0.000171071      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.03328        2.862        3.58299       2.9126       0.446506        19     
produce:ophit:OpHitFinder                         0.0292503     0.0369455     0.0441761     0.036637     0.00463683       19     
produce:wclsdatavd:WireCellToolkit                 84.5634       95.0271       110.087       94.6127       6.35107        19     
produce:gaushit:GausHitFinder                     0.698329       1.18752       1.49391       1.22875      0.230316        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000223971   0.000316476   0.000507373   0.00030417    7.33827e-05      19     
produce:reco3d:SpacePointSolver                    8.41275       15.3461       23.5155       14.9869       4.20102        19     
produce:hitpdune:DisambigFromSpacePoints          0.0846222     0.150026      0.214743      0.143594      0.036341        19     
produce:cluster3d:Cluster3D                       0.968093       2.30307       4.27666       2.10032      0.897548        19     
[art]:TriggerResults:TriggerResultInserter        2.171e-05    2.92032e-05    8.378e-05     2.527e-05    1.3513e-05       19     
end_path:out1:RootOutput                          3.26e-06     6.80953e-06    2.417e-05     7.31e-06     4.66053e-06      19     
end_path:out1:RootOutput(write)                    4.14577       4.91267       6.21585       4.78949      0.614857        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5396.68 MB
  Peak resident set size usage (VmHWM): 3412.57 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0304_df-s04-d0_dw_0_20260203T203739_reco_stage0_20260204T004628_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0304_df-s04-d0_dw_0_20260203T203739_reco_stage0_20260204T004628_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0304_df-s04-d0_dw_0_20260203T203739_reco_stage0_20260204T004628_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163489.0,
    "core.end_time": 1770165989.0,
    "core.events": [
      34666,
      34672,
      34678,
      34684,
      34690,
      34696,
      34702,
      34708,
      34714,
      34720,
      34726,
      34732,
      34738,
      34744,
      34750,
      34756,
      34762,
      34768,
      34774
    ],
    "core.event_count": 19,
    "core.first_event_number": 34666,
    "core.last_event_number": 34774,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0304_df-s04-d0_dw_0_20260203T203739.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:45:30 UTC       justIN version: 01.06.00