justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303650.179@dunegpschedd01.fnal.gov

Jobsub ID303650.179@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:48:28
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_dissel_multicore
Last heartbeat2026-02-04 01:13:30
From worker nodeHostnamewn-snel-012.farm.nikhef.nl
cpuinfoAMD EPYC 7H12 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:16:28
Input filesvd-protodune:np02vd_raw_run042433_0274_df-s04-d1_dw_0_20260203T202617.hdf5
JobscriptExit code0
Real time1h (6887s)
CPU time59m (3580s = 51%)
Max RSS bytes3422617600 (3264 MiB)
Outputting started2026-02-04 01:11:16
Output files
Finished2026-02-04 01:13:30
Saved logsjustin-logs:303650.179-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

. 0x703eef70, 0x703ef8b0
Caught exception in looking up space point ptr... 0x703ef410, 0x703eef70
Caught exception in looking up space point ptr... 0x54c60b60, 0x54c60df0
Caught exception in looking up space point ptr... 0x54c60df0, 0x54c610c0
Caught exception in looking up space point ptr... 0x65dbb720, 0x65dbbd10
Caught exception in looking up space point ptr... 0x65dbbd10, 0x71d319c0
Caught exception in looking up space point ptr... 0x588c41a0, 0x588c3dc0
Caught exception in looking up space point ptr... 0x588c3dc0, 0x588c41a0
Caught exception in looking up space point ptr... 0x4be539f0, 0x4be53c50
Caught exception in looking up space point ptr... 0x4be53c50, 0x4be53b20
Caught exception in looking up space point ptr... 0x4be53b20, 0x4be53790
Caught exception in looking up space point ptr... 0x5f2894d0, 0x6d218890
Caught exception in looking up space point ptr... 0x6d218890, 0x457d9440
Caught exception in looking up space point ptr... 0x3cc671a0, 0x615316c0
Caught exception in looking up space point ptr... 0x615316c0, 0x615315c0
Caught exception in looking up space point ptr... 0x615315c0, 0x615314c0
Caught exception in looking up space point ptr... 0x252c3070, 0x6a044a00
Caught exception in looking up space point ptr... 0x6a044a00, 0x252bfef0
Caught exception in looking up space point ptr... 0x78affc00, 0x847c8a30
Caught exception in looking up space point ptr... 0x847c8a30, 0x22c31b80
Caught exception in looking up space point ptr... 0x7437b630, 0x743793c0
Caught exception in looking up space point ptr... 0x71f06ee0, 0x71f070c0
Caught exception in looking up space point ptr... 0x71f070c0, 0x71f072a0
Caught exception in looking up space point ptr... 0x71f072a0, 0x7437b630
Caught exception in looking up space point ptr... 0x5ceb5f40, 0x5ceb59b0
Caught exception in looking up space point ptr... 0x5ceb59b0, 0x5ceb5ca0
Caught exception in looking up space point ptr... 0x1f395f40, 0x1a2b18a0
Caught exception in looking up space point ptr... 0x1a2b18a0, 0x661915d0
Caught exception in looking up space point ptr... 0x52408c00, 0x2490b520
Caught exception in looking up space point ptr... 0x2490b520, 0x1c2a0ff0
Caught exception in looking up space point ptr... 0x4c5a0830, 0x4c5a09c0
Caught exception in looking up space point ptr... 0x45388860, 0x4c5a0830
Caught exception in looking up space point ptr... 0x54b6fdb0, 0x1b4968e0
Caught exception in looking up space point ptr... 0x1b4968e0, 0x1b496640
Caught exception in looking up space point ptr... 0x6cb4a720, 0x6d5ad820
Caught exception in looking up space point ptr... 0x6d5ad820, 0x6d5ad5c0
Caught exception in looking up space point ptr... 0x66300b10, 0x4812ee10
Caught exception in looking up space point ptr... 0x4812ee10, 0x4cb01120
Caught exception in looking up space point ptr... 0x4cb01120, 0x847bbe10
Caught exception in looking up space point ptr... 0x5f2f2bb0, 0x49dcd710
Caught exception in looking up space point ptr... 0x49dcd710, 0x789bc260
Caught exception in looking up space point ptr... 0x78aedcb0, 0x78698bb0
Caught exception in looking up space point ptr... 0x78698bb0, 0x54f43e60
Caught exception in looking up space point ptr... 0x4be53d80, 0x43964f20
Caught exception in looking up space point ptr... 0x43964f20, 0x4be53d80
Caught exception in looking up space point ptr... 0x5ceb98f0, 0x5ceb9440
Caught exception in looking up space point ptr... 0x5ceb9440, 0x5ceb98f0
Caught exception in looking up space point ptr... 0x5ceb98f0, 0x5ceb9760
Caught exception in looking up space point ptr... 0x5ceb9760, 0x5ceb98f0
Caught exception in looking up space point ptr... 0x69e99c50, 0x1a110e80
Caught exception in looking up space point ptr... 0x1a110e80, 0x1de3cf80
Caught exception in looking up space point ptr... 0x45817cd0, 0x6d4c6010
Caught exception in looking up space point ptr... 0x6d4c6010, 0x45817cd0
Caught exception in looking up space point ptr... 0x45817cd0, 0x5cc70350
Caught exception in looking up space point ptr... 0x5cc70350, 0x45817cd0
Caught exception in looking up space point ptr... 0x46cb6cd0, 0x4c36f6e0
Caught exception in looking up space point ptr... 0x4c36f6e0, 0x46cb6cd0
Caught exception in looking up space point ptr... 0x6cd10360, 0x6cd0ff80
Caught exception in looking up space point ptr... 0x6cd0ff80, 0x6cd10360
Caught exception in looking up space point ptr... 0x25af32a0, 0x20504a00
Caught exception in looking up space point ptr... 0x20504a00, 0x25af32a0
Caught exception in looking up space point ptr... 0x45db0e40, 0x55d051d0
Caught exception in looking up space point ptr... 0x55d051d0, 0x45db0e40
Caught exception in looking up space point ptr... 0x1b13c470, 0x2578f890
Caught exception in looking up space point ptr... 0x2578f890, 0x1b13c470
Caught exception in looking up space point ptr... 0x1d41f9d0, 0x4210f0f0
Caught exception in looking up space point ptr... 0x4210f0f0, 0x26d427b0
++++>>>> total num hits: 47680, num free: 45684
04-Feb-2026 02:11:06 CET  Closed output file "np02vd_raw_run042433_0274_df-s04-d1_dw_0_20260203T202617_reco_stage0_20260204T011106_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         295.281       357.466       433.933       355.397       40.4449        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.0029e-05    0.000121311   0.000322596   0.000111139   4.86709e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                115.421       168.449       238.284       166.69        37.5907        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.600682      0.913058       1.72533      0.786363      0.290183        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0174382     0.0237051     0.0666245     0.0195562     0.0109836       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder            0.01733      0.0177732     0.0182924     0.0177742    0.000213553      19     
produce:beamevent:BeamEvent                      8.3466e-05    0.000210201    0.0016446    0.000129884   0.00033911       19     
produce:pdvddaphne:DAPHNEReaderPDVD                8.64806       15.1482       21.5922       14.2916       3.21496        19     
produce:ophit:OpHitFinder                         0.0356164     0.0449894     0.0526116     0.0454123    0.00470221       19     
produce:wclsdatavd:WireCellToolkit                 123.16        141.021       153.917       142.012       9.15281        19     
produce:gaushit:GausHitFinder                     0.894025       1.68362        2.656        1.66697      0.494851        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000618731   0.000982936   0.00138574    0.00094824    0.000187138      19     
produce:reco3d:SpacePointSolver                    10.5376       19.4038       32.8942       18.3726       6.26128        19     
produce:hitpdune:DisambigFromSpacePoints          0.108531       0.24472      0.382046      0.241538      0.075635        19     
produce:cluster3d:Cluster3D                        1.67114       3.83156       6.87007       3.92915       1.41451        19     
[art]:TriggerResults:TriggerResultInserter       2.5047e-05    4.17467e-05   0.000120957   3.4625e-05    2.19472e-05      19     
end_path:out1:RootOutput                          4.709e-06    7.59537e-06   3.2932e-05     5.36e-06     6.47205e-06      19     
end_path:out1:RootOutput(write)                    5.84513       6.62651       7.85242       6.51512      0.618242        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5366.82 MB
  Peak resident set size usage (VmHWM): 3422.62 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0274_df-s04-d1_dw_0_20260203T202617_reco_stage0_20260204T011106_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0274_df-s04-d1_dw_0_20260203T202617_reco_stage0_20260204T011106_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0274_df-s04-d1_dw_0_20260203T202617_reco_stage0_20260204T011106_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770160600.0,
    "core.end_time": 1770167469.0,
    "core.events": [
      31247,
      31253,
      31259,
      31265,
      31271,
      31277,
      31283,
      31289,
      31295,
      31301,
      31307,
      31313,
      31319,
      31325,
      31331,
      31337,
      31343,
      31349,
      31355
    ],
    "core.event_count": 19,
    "core.first_event_number": 31247,
    "core.last_event_number": 31355,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0274_df-s04-d1_dw_0_20260203T202617.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:58:42 UTC       justIN version: 01.06.00