justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303650.103@dunegpschedd01.fnal.gov

Jobsub ID303650.103@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:48:28
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk_multicore
Last heartbeat2026-02-04 01:13:01
From worker nodeHostnamewn-snel-032.farm.nikhef.nl
cpuinfoAMD EPYC 7H12 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:15:02
Input filesvd-protodune:np02vd_raw_run042433_0179_df-s04-d1_dw_0_20260203T195009.hdf5
JobscriptExit code0
Real time1h (6948s)
CPU time56m (3384s = 48%)
Max RSS bytes3401703424 (3244 MiB)
Outputting started2026-02-04 01:10:51
Output files
Finished2026-02-04 01:13:01
Saved logsjustin-logs:303650.103-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0x2a6a2d50, 0x2a6a1490
Caught exception in looking up space point ptr... 0x2a6a1490, 0x2a6a0b20
Caught exception in looking up space point ptr... 0x2a6a0b20, 0x2a6a1490
Caught exception in looking up space point ptr... 0x28573250, 0x28573150
Caught exception in looking up space point ptr... 0x28573150, 0x28573250
Caught exception in looking up space point ptr... 0x67e81840, 0x7d958160
Caught exception in looking up space point ptr... 0x7d958160, 0x7d95b980
Caught exception in looking up space point ptr... 0x2516c330, 0x2516be80
Caught exception in looking up space point ptr... 0x2516be80, 0x2516c330
Caught exception in looking up space point ptr... 0x2516c330, 0x2516c1a0
Caught exception in looking up space point ptr... 0x2516c1a0, 0x2516c330
Caught exception in looking up space point ptr... 0x2516c330, 0x2516bcf0
Caught exception in looking up space point ptr... 0x2516bcf0, 0x2516c330
Caught exception in looking up space point ptr... 0x7a8d01d0, 0x7a8d0080
Caught exception in looking up space point ptr... 0x7a8d0080, 0x2d755c90
Caught exception in looking up space point ptr... 0x1dfcf9f0, 0x22e9b9b0
Caught exception in looking up space point ptr... 0x22e9b9b0, 0x1dfcf8f0
Caught exception in looking up space point ptr... 0x4e616980, 0x4e616d40
Caught exception in looking up space point ptr... 0x4e616d40, 0x4e616980
Caught exception in looking up space point ptr... 0x4cf6ded0, 0x4cf6e170
Caught exception in looking up space point ptr... 0x4cf6e170, 0x4cf6e020
Caught exception in looking up space point ptr... 0x2dde5670, 0x2dde5d60
Caught exception in looking up space point ptr... 0x2dde5d60, 0x2dde5670
Caught exception in looking up space point ptr... 0x76797320, 0x3de22ce0
Caught exception in looking up space point ptr... 0x3de22ce0, 0x861d48b0
Caught exception in looking up space point ptr... 0x825e35a0, 0x2550e7c0
Caught exception in looking up space point ptr... 0x2550e7c0, 0x825e35a0
Caught exception in looking up space point ptr... 0x20705520, 0x207070c0
Caught exception in looking up space point ptr... 0x207070c0, 0x20705520
Caught exception in looking up space point ptr... 0x59424600, 0x59424ab0
Caught exception in looking up space point ptr... 0x59424ab0, 0x59424790
Caught exception in looking up space point ptr... 0x5d3a9820, 0x9936fad0
Caught exception in looking up space point ptr... 0x9936fad0, 0x9936f9d0
Caught exception in looking up space point ptr... 0x878a4630, 0x878a49f0
Caught exception in looking up space point ptr... 0x878a49f0, 0x878a4630
Caught exception in looking up space point ptr... 0x26a8bd50, 0x26a8bc00
Caught exception in looking up space point ptr... 0x26a8bc00, 0x26a8c050
Caught exception in looking up space point ptr... 0x3dd443b0, 0x3dd446b0
Caught exception in looking up space point ptr... 0x3dd446b0, 0x3dd443b0
Caught exception in looking up space point ptr... 0x485cc6a0, 0x485cc510
Caught exception in looking up space point ptr... 0x485cc510, 0x485cc6a0
Caught exception in looking up space point ptr... 0x506faef0, 0x506fab60
Caught exception in looking up space point ptr... 0x506fab60, 0x506faef0
Caught exception in looking up space point ptr... 0x7f83db10, 0x7f834300
Caught exception in looking up space point ptr... 0x7f834300, 0x7f83db10
Caught exception in looking up space point ptr... 0x78473ab0, 0x78473920
Caught exception in looking up space point ptr... 0x78473920, 0x78473ab0
Caught exception in looking up space point ptr... 0x3dd6fa70, 0x3dd6fc00
Caught exception in looking up space point ptr... 0x3dd6fc00, 0x5687b090
Caught exception in looking up space point ptr... 0x5687b090, 0x3dd6f8e0
Caught exception in looking up space point ptr... 0x3dd6f8e0, 0x5687af90
Caught exception in looking up space point ptr... 0x58fb3780, 0x58fb0db0
Caught exception in looking up space point ptr... 0x58fb0db0, 0x58fb3780
Caught exception in looking up space point ptr... 0x5a38a200, 0x5a382150
Caught exception in looking up space point ptr... 0x5a382150, 0x5a384180
Caught exception in looking up space point ptr... 0x8f570b20, 0x8f5709d0
Caught exception in looking up space point ptr... 0x8f5709d0, 0x8f570b20
Caught exception in looking up space point ptr... 0x5a38a110, 0x5a38d030
Caught exception in looking up space point ptr... 0x5a38d030, 0x5a37f370
Caught exception in looking up space point ptr... 0x7f83fd20, 0x7f833e80
Caught exception in looking up space point ptr... 0x7f833e80, 0x7f83fa00
Caught exception in looking up space point ptr... 0x91008e30, 0x91008b90
Caught exception in looking up space point ptr... 0x91008b90, 0x91008ce0
Caught exception in looking up space point ptr... 0x41fd1560, 0x41fd1740
Caught exception in looking up space point ptr... 0x41fd1740, 0x41fd1560
Caught exception in looking up space point ptr... 0x22047970, 0x2aeae8f0
Caught exception in looking up space point ptr... 0x2aeae8f0, 0x22047970
++++>>>> total num hits: 129456, num free: 122895
04-Feb-2026 02:10:40 CET  Closed output file "np02vd_raw_run042433_0179_df-s04-d1_dw_0_20260203T195009_reco_stage0_20260204T011040_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         311.056       360.529       430.786       347.78        35.7818        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.5849e-05    0.000113467   0.000279994   0.000104566   3.95857e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                112.508       180.949       248.616       169.673       38.137         19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.722442      0.918132       1.14545      0.949189       0.13235        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0175045     0.0219488     0.0343288     0.0214858    0.00447789       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0173186     0.0195242     0.0517188     0.0177091    0.00759153       19     
produce:beamevent:BeamEvent                      8.7925e-05    0.000192175   0.00115101    0.000127358   0.000229463      19     
produce:pdvddaphne:DAPHNEReaderPDVD                9.6273        16.0393       21.7002       16.0384       3.2326         19     
produce:ophit:OpHitFinder                         0.0305883     0.0422845     0.0719651     0.0416121    0.00887335       19     
produce:wclsdatavd:WireCellToolkit                 115.923       131.989       153.936       131.683       11.6195        19     
produce:gaushit:GausHitFinder                     0.993599       1.56899       2.82979       1.49718      0.425271        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000366826   0.000763072   0.00115915    0.000768538   0.000251935      19     
produce:reco3d:SpacePointSolver                    10.8978       18.6023       24.2211       18.8017       3.45746        19     
produce:hitpdune:DisambigFromSpacePoints          0.131532      0.234058      0.368051      0.229875      0.0621718       19     
produce:cluster3d:Cluster3D                        1.71792       3.69481       6.23336       3.55063       1.23426        19     
[art]:TriggerResults:TriggerResultInserter       2.4265e-05    3.5624e-05    0.000130765   2.9084e-05    2.29606e-05      19     
end_path:out1:RootOutput                          3.807e-06     6.83e-06     3.2151e-05     4.608e-06    6.61706e-06      19     
end_path:out1:RootOutput(write)                    5.37258       6.3976        7.72539       6.14164      0.688091        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5358.02 MB
  Peak resident set size usage (VmHWM): 3401.7 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0179_df-s04-d1_dw_0_20260203T195009_reco_stage0_20260204T011040_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0179_df-s04-d1_dw_0_20260203T195009_reco_stage0_20260204T011040_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0179_df-s04-d1_dw_0_20260203T195009_reco_stage0_20260204T011040_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770160514.0,
    "core.end_time": 1770167443.0,
    "core.events": [
      20417,
      20423,
      20429,
      20435,
      20441,
      20447,
      20453,
      20459,
      20465,
      20471,
      20477,
      20483,
      20489,
      20495,
      20501,
      20507,
      20513,
      20519,
      20525
    ],
    "core.event_count": 19,
    "core.first_event_number": 20417,
    "core.last_event_number": 20525,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0179_df-s04-d1_dw_0_20260203T195009.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:00:23 UTC       justIN version: 01.06.00