justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.32@dunegpschedd02.fnal.gov

Jobsub ID297923.32@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce509
Last heartbeat2026-02-04 00:36:21
From worker nodeHostnameb9p34p1022.cern.ch
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:58:33
Input filesvd-protodune:np02vd_raw_run042433_0099_df-s03-d0_dw_0_20260203T191945.hdf5
JobscriptExit code0
Real time35m (2141s)
CPU time34m (2064s = 96%)
Max RSS bytes3346153472 (3191 MiB)
Outputting started2026-02-04 00:34:16
Output files
Finished2026-02-04 00:36:21
Saved logsjustin-logs:297923.32-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ace point ptr... 0x3d7474d0, 0x1a035540
Caught exception in looking up space point ptr... 0x11cf1c00, 0x11cf1d30
Caught exception in looking up space point ptr... 0x11cf1d30, 0x4f274d30
Caught exception in looking up space point ptr... 0x9266e10, 0x17ccd2e0
Caught exception in looking up space point ptr... 0x17ccd2e0, 0x926e900
Caught exception in looking up space point ptr... 0x3a7770e0, 0x3a7776d0
Caught exception in looking up space point ptr... 0x3a7776d0, 0x3a7770e0
Caught exception in looking up space point ptr... 0xec2dff0, 0xec2de60
Caught exception in looking up space point ptr... 0xec2de60, 0xec2daf0
Caught exception in looking up space point ptr... 0x75956ad0, 0x61d950c0
Caught exception in looking up space point ptr... 0x61d950c0, 0x75956ad0
Caught exception in looking up space point ptr... 0x13deb180, 0x13df1d10
Caught exception in looking up space point ptr... 0x13df1d10, 0x13df1c10
Caught exception in looking up space point ptr... 0x17d0cdc0, 0x17d0cc70
Caught exception in looking up space point ptr... 0x17d0cc70, 0x17d0cdc0
Caught exception in looking up space point ptr... 0x3527bfe0, 0x3527a8e0
Caught exception in looking up space point ptr... 0x3527a8e0, 0x3527b990
Caught exception in looking up space point ptr... 0x38c4fa10, 0x48c5e740
Caught exception in looking up space point ptr... 0x48c5e740, 0x3b0685a0
Caught exception in looking up space point ptr... 0x733dd540, 0x733dd6a0
Caught exception in looking up space point ptr... 0x733dd6a0, 0x733dbeb0
Caught exception in looking up space point ptr... 0x732a46e0, 0x183c0de0
Caught exception in looking up space point ptr... 0x7329e710, 0x732a46e0
Caught exception in looking up space point ptr... 0x5ee68990, 0x13e3e910
Caught exception in looking up space point ptr... 0x13e3e910, 0x38c4fce0
Caught exception in looking up space point ptr... 0x32a46940, 0x50759930
Caught exception in looking up space point ptr... 0x50759930, 0x19f42120
Caught exception in looking up space point ptr... 0x4aa5b300, 0xbc3e310
Caught exception in looking up space point ptr... 0xbc3e310, 0x4aa5b430
Caught exception in looking up space point ptr... 0x3ba2a2e0, 0x3ba2a010
Caught exception in looking up space point ptr... 0x3ba2a010, 0x3ba2a150
Caught exception in looking up space point ptr... 0x7331c7d0, 0x7331c680
Caught exception in looking up space point ptr... 0x7331c680, 0x7331c7d0
Caught exception in looking up space point ptr... 0x7328d240, 0x73291930
Caught exception in looking up space point ptr... 0x73291930, 0x73280930
Caught exception in looking up space point ptr... 0x4bd35580, 0x4bd35750
Caught exception in looking up space point ptr... 0x4bd35750, 0x4bd35580
Caught exception in looking up space point ptr... 0x18ce1310, 0x92f2c70
Caught exception in looking up space point ptr... 0x50146dc0, 0x18ce1310
Caught exception in looking up space point ptr... 0x422f8d50, 0x422f91d0
Caught exception in looking up space point ptr... 0x422f91d0, 0x422f8d50
Caught exception in looking up space point ptr... 0x350f6d10, 0x47ceda20
Caught exception in looking up space point ptr... 0x47ceda20, 0x3c356d90
Caught exception in looking up space point ptr... 0x73327240, 0x73326fc0
Caught exception in looking up space point ptr... 0x73326fc0, 0x73320ff0
Caught exception in looking up space point ptr... 0x385400c0, 0x38541200
Caught exception in looking up space point ptr... 0x38541200, 0x385400c0
Caught exception in looking up space point ptr... 0xb79f000, 0x4808bd10
Caught exception in looking up space point ptr... 0x4808bd10, 0xb79f000
Caught exception in looking up space point ptr... 0x362e98a0, 0x31ba8a90
Caught exception in looking up space point ptr... 0x31ba8a90, 0x362e98a0
Caught exception in looking up space point ptr... 0x362e98a0, 0x42afb5b0
Caught exception in looking up space point ptr... 0x42afb5b0, 0x362e98a0
Caught exception in looking up space point ptr... 0x422f57a0, 0x422f49d0
Caught exception in looking up space point ptr... 0x422f9a00, 0x422f57a0
Caught exception in looking up space point ptr... 0x64f7cf90, 0x60ca3ef0
Caught exception in looking up space point ptr... 0x60ca3ef0, 0x64f7cf90
Caught exception in looking up space point ptr... 0x64f7cf90, 0x1b4f6920
Caught exception in looking up space point ptr... 0x1b4f6920, 0x64f7cf90
Caught exception in looking up space point ptr... 0x7eaec710, 0x17ccb4a0
Caught exception in looking up space point ptr... 0x17ccb4a0, 0x7eaec710
Caught exception in looking up space point ptr... 0x10d080d0, 0x10d0bf80
Caught exception in looking up space point ptr... 0x10d0bf80, 0x10d080d0
Caught exception in looking up space point ptr... 0x9512010, 0x837c8850
Caught exception in looking up space point ptr... 0x837c8850, 0x9512010
Caught exception in looking up space point ptr... 0x9512010, 0x837c8670
Caught exception in looking up space point ptr... 0x837c8670, 0x9512010
++++>>>> total num hits: 70604, num free: 67712
04-Feb-2026 01:34:10 CET  Closed output file "np02vd_raw_run042433_0099_df-s03-d0_dw_0_20260203T191945_reco_stage0_20260204T003410_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         97.0346       109.716       121.734       107.332       6.27046        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.0981e-05    9.29738e-05   0.000252698   8.1291e-05    4.1004e-05       19     
produce:tpcrawdecoder:PDVDTPCReader                5.03819       6.21248       12.7723       5.79623       1.62273        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00791272     0.0188815     0.0501736     0.0162128     0.0110323       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000398056   0.000718487   0.00106151    0.000742771   0.000190197      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000343294   0.000724648   0.00166242    0.000742031   0.000262935      19     
produce:beamevent:BeamEvent                      7.9789e-05    0.000173465   0.00100181    0.000123034   0.000201673      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.86611       2.11414       2.43834       2.1457       0.154826        19     
produce:ophit:OpHitFinder                         0.0283235     0.034611      0.042576      0.0343179    0.00369913       19     
produce:wclsdatavd:WireCellToolkit                 74.9669       82.1691       88.1016       81.9572       3.98541        19     
produce:gaushit:GausHitFinder                     0.545565       1.03731       1.50741       1.09788      0.232486        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000158977   0.000241347   0.000425367   0.000232518   5.71763e-05      19     
produce:reco3d:SpacePointSolver                     6.281        11.7088       19.7264        11.26        2.96976        19     
produce:hitpdune:DisambigFromSpacePoints          0.0541284     0.137244       0.2366       0.114619      0.0472948       19     
produce:cluster3d:Cluster3D                       0.715574       1.81514       4.55315       1.65606      0.821203        19     
[art]:TriggerResults:TriggerResultInserter       2.9754e-05    4.33395e-05   0.000126519   3.8188e-05    2.05471e-05      19     
end_path:out1:RootOutput                          4.948e-06    7.89621e-06    3.348e-05     6.229e-06    6.16525e-06      19     
end_path:out1:RootOutput(write)                    3.72604       4.42735       5.72705       4.31671      0.521032        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5396.37 MB
  Peak resident set size usage (VmHWM): 3346.15 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0099_df-s03-d0_dw_0_20260203T191945_reco_stage0_20260204T003410_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0099_df-s03-d0_dw_0_20260203T191945_reco_stage0_20260204T003410_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0099_df-s03-d0_dw_0_20260203T191945_reco_stage0_20260204T003410_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163123.0,
    "core.end_time": 1770165251.0,
    "core.events": [
      11294,
      11300,
      11306,
      11312,
      11318,
      11324,
      11330,
      11336,
      11342,
      11348,
      11354,
      11360,
      11366,
      11372,
      11378,
      11384,
      11390,
      11396,
      11402
    ],
    "core.event_count": 19,
    "core.first_event_number": 11294,
    "core.last_event_number": 11402,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0099_df-s03-d0_dw_0_20260203T191945.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:19 UTC       justIN version: 01.06.00