justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303779.2@dunegpschedd01.fnal.gov

Jobsub ID303779.2@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 02:26:33
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2026-02-04 04:27:15
From worker nodeHostnamewn-snel-010.farm.nikhef.nl
cpuinfoAMD EPYC 7H12 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 02:26:49
Input filesvd-protodune:np02vd_raw_run042433_0060_df-s03-d1_dw_0_20260203T190455.hdf5
JobscriptExit code0
Real time1h (7089s)
CPU time1h (3876s = 54%)
Max RSS bytes3437649920 (3278 MiB)
Outputting started2026-02-04 04:25:00
Output files
Finished2026-02-04 04:27:15
Saved logsjustin-logs:303779.2-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

g up space point ptr... 0x3baba500, 0x3baba930
Caught exception in looking up space point ptr... 0x835b46a0, 0x835b4510
Caught exception in looking up space point ptr... 0x8061aea0, 0x835b46a0
Caught exception in looking up space point ptr... 0x3abf1f90, 0x3abf2380
Caught exception in looking up space point ptr... 0x3abf2380, 0x3abf2230
Caught exception in looking up space point ptr... 0x832a9e90, 0x8328f010
Caught exception in looking up space point ptr... 0x8328f010, 0x832a7280
Caught exception in looking up space point ptr... 0x5ce0e4f0, 0x5ce0e290
Caught exception in looking up space point ptr... 0x5ce0e290, 0x5ce0e4f0
Caught exception in looking up space point ptr... 0x6b74e8a0, 0x44486b90
Caught exception in looking up space point ptr... 0x4ccd81e0, 0x919c89c0
Caught exception in looking up space point ptr... 0x919c89c0, 0x959b3530
Caught exception in looking up space point ptr... 0x4cfc5a70, 0x6b74e8a0
Caught exception in looking up space point ptr... 0x83709a50, 0x83709900
Caught exception in looking up space point ptr... 0x83709900, 0x83709a50
Caught exception in looking up space point ptr... 0x75337120, 0x75338950
Caught exception in looking up space point ptr... 0x75336d00, 0x75337120
Caught exception in looking up space point ptr... 0x99f48ed0, 0x3cf4cb40
Caught exception in looking up space point ptr... 0x3cf4cb40, 0x99f48ed0
Caught exception in looking up space point ptr... 0x4c2dfa40, 0x4c2be3a0
Caught exception in looking up space point ptr... 0x4c2be3a0, 0x4c2dfa40
Caught exception in looking up space point ptr... 0x93ce06e0, 0x835a46f0
Caught exception in looking up space point ptr... 0x835a46f0, 0x93ce06e0
Caught exception in looking up space point ptr... 0x4bd153d0, 0x80188180
Caught exception in looking up space point ptr... 0x80188180, 0x4bd153d0
Caught exception in looking up space point ptr... 0x9991de80, 0x9991f6b0
Caught exception in looking up space point ptr... 0x9991f6b0, 0x9991de80
Caught exception in looking up space point ptr... 0x82e14510, 0x82e146a0
Caught exception in looking up space point ptr... 0x8b944500, 0x82e14510
Caught exception in looking up space point ptr... 0x5cbab6c0, 0x4a2cc840
Caught exception in looking up space point ptr... 0x4a2cc840, 0x5cbab6c0
Caught exception in looking up space point ptr... 0x3abfc390, 0x8cb33410
Caught exception in looking up space point ptr... 0x8cb33410, 0x3abfc390
Caught exception in looking up space point ptr... 0x92c4b080, 0x92c4b210
Caught exception in looking up space point ptr... 0x6c5a39b0, 0x92c4b080
Caught exception in looking up space point ptr... 0x4b9eb0b0, 0x4b9eb5d0
Caught exception in looking up space point ptr... 0x4b9eb5d0, 0x4b9eaf70
Caught exception in looking up space point ptr... 0x4ada4ce0, 0x95b97f20
Caught exception in looking up space point ptr... 0x95b97f20, 0x6ac09310
Caught exception in looking up space point ptr... 0x8328e580, 0x8328e3f0
Caught exception in looking up space point ptr... 0x8328e3f0, 0x8328e580
Caught exception in looking up space point ptr... 0x9389ed80, 0x87f33c60
Caught exception in looking up space point ptr... 0x87f33c60, 0x938a0280
Caught exception in looking up space point ptr... 0x91641420, 0x8aae6750
Caught exception in looking up space point ptr... 0x8aae6750, 0x91641420
Caught exception in looking up space point ptr... 0x801968d0, 0x80196a90
Caught exception in looking up space point ptr... 0x80196a90, 0x80196c70
Caught exception in looking up space point ptr... 0x8e45be50, 0x4a414a60
Caught exception in looking up space point ptr... 0x4a414a60, 0x8e45be50
Caught exception in looking up space point ptr... 0xa0254a20, 0xa02546b0
Caught exception in looking up space point ptr... 0xa02546b0, 0xa0254a20
Caught exception in looking up space point ptr... 0xa0254a20, 0xa0240b60
Caught exception in looking up space point ptr... 0xa0240b60, 0xa0254a20
Caught exception in looking up space point ptr... 0x4cacbfb0, 0x4cac9980
Caught exception in looking up space point ptr... 0x4cac9980, 0x4caca3c0
Caught exception in looking up space point ptr... 0xb8b6eff0, 0xb8b6c120
Caught exception in looking up space point ptr... 0xb8b6c120, 0xb8b6e850
Caught exception in looking up space point ptr... 0x80196340, 0x801960a0
Caught exception in looking up space point ptr... 0x801960a0, 0x801961f0
Caught exception in looking up space point ptr... 0x7fcb05b0, 0x6f44ce60
Caught exception in looking up space point ptr... 0x6f44ce60, 0x7fcb05b0
Caught exception in looking up space point ptr... 0x82e36b10, 0x82e3c340
Caught exception in looking up space point ptr... 0x82e3c340, 0x82e36b10
Caught exception in looking up space point ptr... 0x82e36b10, 0x82e394a0
Caught exception in looking up space point ptr... 0x82e394a0, 0x82e36b10
Caught exception in looking up space point ptr... 0x470e96c0, 0x470e8b70
Caught exception in looking up space point ptr... 0x470e8b70, 0x470e96c0
++++>>>> total num hits: 97812, num free: 94276
04-Feb-2026 05:24:50 CET  Closed output file "np02vd_raw_run042433_0060_df-s03-d1_dw_0_20260203T190455_reco_stage0_20260204T042450_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         311.965       367.772       418.747       364.807       30.8246        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.0771e-05    0.000114076   0.000320071   0.000102923   4.89581e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                101.744       161.624       201.392       166.159       26.9944        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.588155      0.812977       1.37483      0.794718      0.185199        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0174971     0.0224034     0.077968      0.0177671     0.0136567       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0175398     0.0199467     0.0588139     0.0177772    0.00916374       19     
produce:beamevent:BeamEvent                      9.6922e-05    0.000213668   0.00147099    0.00014379    0.000297124      19     
produce:pdvddaphne:DAPHNEReaderPDVD                11.6972       15.2384       20.8612       15.2552       2.33376        19     
produce:ophit:OpHitFinder                         0.0290567     0.0445879     0.0513807     0.0452987    0.00478786       19     
produce:wclsdatavd:WireCellToolkit                 135.64        152.887       166.415       154.795       7.50997        19     
produce:gaushit:GausHitFinder                      1.13949       1.87884       3.27359       1.67448      0.560738        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000346301   0.000650564   0.00097969    0.000635784   0.000171871      19     
produce:reco3d:SpacePointSolver                    12.6462       20.9566       34.6429       19.1971       5.82914        19     
produce:hitpdune:DisambigFromSpacePoints          0.165012      0.298242      0.475733      0.299575      0.0821075       19     
produce:cluster3d:Cluster3D                        1.95683       4.60675       10.0302       4.55392       1.84532        19     
[art]:TriggerResults:TriggerResultInserter       2.3484e-05    3.72004e-05   0.000107822   3.3503e-05    1.92287e-05      19     
end_path:out1:RootOutput                          3.757e-06    6.24384e-06   2.8343e-05     4.759e-06    5.41953e-06      19     
end_path:out1:RootOutput(write)                    7.42678       9.32761       10.2724       9.5089       0.734958        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5373.5 MB
  Peak resident set size usage (VmHWM): 3437.65 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0060_df-s03-d1_dw_0_20260203T190455_reco_stage0_20260204T042450_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0060_df-s03-d1_dw_0_20260203T190455_reco_stage0_20260204T042450_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0060_df-s03-d1_dw_0_20260203T190455_reco_stage0_20260204T042450_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770172023.0,
    "core.end_time": 1770179092.0,
    "core.events": [
      6849,
      6855,
      6861,
      6867,
      6873,
      6879,
      6885,
      6891,
      6897,
      6903,
      6909,
      6915,
      6921,
      6927,
      6933,
      6939,
      6945,
      6951,
      6957
    ],
    "core.event_count": 19,
    "core.first_event_number": 6849,
    "core.last_event_number": 6957,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0060_df-s03-d1_dw_0_20260203T190455.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:32:02 UTC       justIN version: 01.06.00