justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.78@dunegpschedd01.fnal.gov

Jobsub ID303652.78@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_dissel_multicore
Last heartbeat2026-02-04 01:31:01
From worker nodeHostnamewn-pijl-021.farm.nikhef.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:28:47
Input filesvd-protodune:np02vd_raw_run042433_0402_df-s02-d1_dw_0_20260203T211453.hdf5
JobscriptExit code0
Real time1h (7171s)
CPU time1h (3853s = 53%)
Max RSS bytes3411283968 (3253 MiB)
Outputting started2026-02-04 01:28:20
Output files
Finished2026-02-04 01:31:01
Saved logsjustin-logs:303652.78-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

. 0x3e0d9950, 0x5ee11570
Caught exception in looking up space point ptr... 0x5ee11570, 0x42b8d8e0
Caught exception in looking up space point ptr... 0x3be9a330, 0x3be9a640
Caught exception in looking up space point ptr... 0x3be94d00, 0x3be9a330
Caught exception in looking up space point ptr... 0x8e368a50, 0x8e367d60
Caught exception in looking up space point ptr... 0x8e367d60, 0x8e368a50
Caught exception in looking up space point ptr... 0x92344630, 0x4a15bf80
Caught exception in looking up space point ptr... 0x4a15bf80, 0x92344630
Caught exception in looking up space point ptr... 0x405e0e40, 0x405e0f90
Caught exception in looking up space point ptr... 0x405e0f90, 0x405e10e0
Caught exception in looking up space point ptr... 0x4a277250, 0x66f2f740
Caught exception in looking up space point ptr... 0x66f2f740, 0x4a277250
Caught exception in looking up space point ptr... 0x7a7a17e0, 0x7a7a16e0
Caught exception in looking up space point ptr... 0x78ee3990, 0x78ee40d0
Caught exception in looking up space point ptr... 0x78ee40d0, 0x7a7a17e0
Caught exception in looking up space point ptr... 0x82c5e6d0, 0x82c5e7d0
Caught exception in looking up space point ptr... 0x82c5e7d0, 0x82c5d530
Caught exception in looking up space point ptr... 0x82c5fab0, 0x82c63c30
Caught exception in looking up space point ptr... 0x82c63c30, 0x82c5fab0
Caught exception in looking up space point ptr... 0x49085b70, 0x49085850
Caught exception in looking up space point ptr... 0x49085850, 0x490859e0
Caught exception in looking up space point ptr... 0x8f138190, 0x647d36a0
Caught exception in looking up space point ptr... 0x647d36a0, 0x70d59270
Caught exception in looking up space point ptr... 0x405e7d90, 0x405e7af0
Caught exception in looking up space point ptr... 0x405e7af0, 0x405e79a0
Caught exception in looking up space point ptr... 0x8e3766a0, 0x8e376590
Caught exception in looking up space point ptr... 0x8e376590, 0x8e3766a0
Caught exception in looking up space point ptr... 0x924ba9f0, 0x93226610
Caught exception in looking up space point ptr... 0x93226610, 0x69c6ac30
Caught exception in looking up space point ptr... 0xa4f8c5a0, 0xa4f8c1c0
Caught exception in looking up space point ptr... 0xa4f8c1c0, 0xa4f8c5a0
Caught exception in looking up space point ptr... 0x7efe5cf0, 0x7efe4eb0
Caught exception in looking up space point ptr... 0x7efe4eb0, 0x7efe5cf0
Caught exception in looking up space point ptr... 0x3fb42e20, 0x3fb42cd0
Caught exception in looking up space point ptr... 0x3fb42cd0, 0x3fb42990
Caught exception in looking up space point ptr... 0x4884fb60, 0x4884fc90
Caught exception in looking up space point ptr... 0x4884fc90, 0x4884fb60
Caught exception in looking up space point ptr... 0x8e373bd0, 0x8e373e30
Caught exception in looking up space point ptr... 0x8e373e30, 0x8e373f60
Caught exception in looking up space point ptr... 0x79d4fcb0, 0x7433bf10
Caught exception in looking up space point ptr... 0x7433bf10, 0x411ee290
Caught exception in looking up space point ptr... 0x92035130, 0x5ee1f2d0
Caught exception in looking up space point ptr... 0x5ee1f2d0, 0x5ee189c0
Caught exception in looking up space point ptr... 0x8f9a26d0, 0x8f9a2540
Caught exception in looking up space point ptr... 0x8f9a2540, 0x8f9a26d0
Caught exception in looking up space point ptr... 0x71a75dc0, 0x71a75cc0
Caught exception in looking up space point ptr... 0x71a75cc0, 0x8581bf90
Caught exception in looking up space point ptr... 0x40618c80, 0x646aeef0
Caught exception in looking up space point ptr... 0x646a6300, 0x40618c80
Caught exception in looking up space point ptr... 0x7e8294f0, 0x7e8291d0
Caught exception in looking up space point ptr... 0x7e8291d0, 0x7e8294f0
Caught exception in looking up space point ptr... 0x4afbabf0, 0x880cd550
Caught exception in looking up space point ptr... 0x880cd550, 0x4afbabf0
Caught exception in looking up space point ptr... 0x79bda640, 0x4124fc80
Caught exception in looking up space point ptr... 0x4124fc80, 0x79bda640
Caught exception in looking up space point ptr... 0x79bda640, 0x87fccc30
Caught exception in looking up space point ptr... 0x87fccc30, 0x79bda640
Caught exception in looking up space point ptr... 0x83dc12a0, 0xaaff2380
Caught exception in looking up space point ptr... 0xaaff2380, 0x83dc12a0
Caught exception in looking up space point ptr... 0x39811070, 0x7d8a8f40
Caught exception in looking up space point ptr... 0x7d8a8f40, 0x3996d940
Caught exception in looking up space point ptr... 0x43d7d790, 0xa543e930
Caught exception in looking up space point ptr... 0xa543e930, 0x43d7d790
Caught exception in looking up space point ptr... 0x412b2f70, 0x69b6bac0
Caught exception in looking up space point ptr... 0x69b6bac0, 0x412b2f70
Caught exception in looking up space point ptr... 0x7ebeda40, 0x7ac61890
Caught exception in looking up space point ptr... 0x7ac61890, 0x7ebeda40
++++>>>> total num hits: 80922, num free: 77708
04-Feb-2026 02:28:05 CET  Closed output file "np02vd_raw_run042433_0402_df-s02-d1_dw_0_20260203T211453_reco_stage0_20260204T012805_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         315.157       371.231       454.729       370.636       40.3542        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.0016e-05    0.000133061   0.000309478   0.000113832   5.00913e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                116.765       167.568       250.422       153.256       37.3043        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.587419      0.892479       1.6216       0.854184      0.218442        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0176555     0.0208845     0.0264823     0.0207208    0.00303292       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0175834     0.0178966     0.0184411     0.0178892    0.00021072       19     
produce:beamevent:BeamEvent                      0.000136146   0.000242315   0.00142636    0.000171409   0.000279918      19     
produce:pdvddaphne:DAPHNEReaderPDVD                11.2926       15.4831       20.1537       14.7464       2.98927        19     
produce:ophit:OpHitFinder                         0.0395923     0.0497262     0.0559753     0.0503189    0.00382055       19     
produce:wclsdatavd:WireCellToolkit                 137.117       148.41        163.157       147.618       6.02315        19     
produce:gaushit:GausHitFinder                      1.16591       1.95212       2.89508       1.97653      0.411973        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000517292   0.00164962    0.00418381    0.00165175    0.000878721      19     
produce:reco3d:SpacePointSolver                    16.8362       24.7014       37.9675       24.8522       4.78016        19     
produce:hitpdune:DisambigFromSpacePoints          0.231763      0.303915       0.43776      0.295713      0.0552653       19     
produce:cluster3d:Cluster3D                         2.473        4.19351       7.23871       4.04566       1.03328        19     
[art]:TriggerResults:TriggerResultInserter       3.0375e-05    5.44822e-05   0.000126861   4.8323e-05    2.32653e-05      19     
end_path:out1:RootOutput                          5.068e-06    9.71147e-06   3.1698e-05     6.67e-06     6.97385e-06      19     
end_path:out1:RootOutput(write)                    6.36982       7.53498       8.2125        7.65814      0.490612        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5368.12 MB
  Peak resident set size usage (VmHWM): 3411.28 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0402_df-s02-d1_dw_0_20260203T211453_reco_stage0_20260204T012805_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0402_df-s02-d1_dw_0_20260203T211453_reco_stage0_20260204T012805_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0402_df-s02-d1_dw_0_20260203T211453_reco_stage0_20260204T012805_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161341.0,
    "core.end_time": 1770168489.0,
    "core.events": [
      45835,
      45841,
      45847,
      45853,
      45859,
      45865,
      45871,
      45877,
      45883,
      45889,
      45895,
      45901,
      45907,
      45913,
      45919,
      45925,
      45931,
      45937,
      45943
    ],
    "core.event_count": 19,
    "core.first_event_number": 45835,
    "core.last_event_number": 45943,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0402_df-s02-d1_dw_0_20260203T211453.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:56:33 UTC       justIN version: 01.06.00