justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.76@dunegpschedd01.fnal.gov

Jobsub ID303652.76@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_dissel_multicore
Last heartbeat2026-02-04 01:31:55
From worker nodeHostnamewn-pijl-021.farm.nikhef.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:28:48
Input filesvd-protodune:np02vd_raw_run042433_0507_df-s02-d1_dw_0_20260203T215449.hdf5
JobscriptExit code0
Real time2h (7247s)
CPU time1h (3791s = 52%)
Max RSS bytes3397271552 (3239 MiB)
Outputting started2026-02-04 01:29:37
Output files
Finished2026-02-04 01:31:55
Saved logsjustin-logs:303652.76-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

tr... 0x38147530, 0x38147080
Caught exception in looking up space point ptr... 0x38147080, 0x381473a0
Caught exception in looking up space point ptr... 0x381473a0, 0x381585a0
Caught exception in looking up space point ptr... 0x2e621ec0, 0x2e6250a0
Caught exception in looking up space point ptr... 0x2e6250a0, 0x2e621ec0
Caught exception in looking up space point ptr... 0x15f73840, 0x15f71be0
Caught exception in looking up space point ptr... 0x15f71be0, 0x15f71a70
Caught exception in looking up space point ptr... 0x59c4c9a0, 0x6850abb0
Caught exception in looking up space point ptr... 0x6850abb0, 0x6850d350
Caught exception in looking up space point ptr... 0x77587ee0, 0x698f39d0
Caught exception in looking up space point ptr... 0x698f39d0, 0x77587ee0
Caught exception in looking up space point ptr... 0x750b6f50, 0x750b6aa0
Caught exception in looking up space point ptr... 0x750b6aa0, 0x750b6f50
Caught exception in looking up space point ptr... 0x6882acb0, 0x36531ff0
Caught exception in looking up space point ptr... 0x36531ff0, 0x41136ea0
Caught exception in looking up space point ptr... 0x41145b90, 0x411456e0
Caught exception in looking up space point ptr... 0x411456e0, 0x41145870
Caught exception in looking up space point ptr... 0x76dcb580, 0xf4f9550
Caught exception in looking up space point ptr... 0xf4f9550, 0x76dcb710
Caught exception in looking up space point ptr... 0x62f63ae0, 0x62f63930
Caught exception in looking up space point ptr... 0x62f63930, 0x62f63ae0
Caught exception in looking up space point ptr... 0x6d1924d0, 0x6d198420
Caught exception in looking up space point ptr... 0x6d198420, 0x6d1924d0
Caught exception in looking up space point ptr... 0x6d1924d0, 0x76bf5720
Caught exception in looking up space point ptr... 0x76bf5720, 0x6d1924d0
Caught exception in looking up space point ptr... 0x6d1924d0, 0x6d192da0
Caught exception in looking up space point ptr... 0x6d192da0, 0x6d1924d0
Caught exception in looking up space point ptr... 0x480f8350, 0x48100ae0
Caught exception in looking up space point ptr... 0x48100ae0, 0x480f8350
Caught exception in looking up space point ptr... 0x1a1e46c0, 0x1a1f3fe0
Caught exception in looking up space point ptr... 0x1a1f3fe0, 0x1a1e46c0
Caught exception in looking up space point ptr... 0x1a1e46c0, 0x1a1eec00
Caught exception in looking up space point ptr... 0x1a1eec00, 0x1a1e46c0
Caught exception in looking up space point ptr... 0x5dacce80, 0x3d3d3550
Caught exception in looking up space point ptr... 0x3d3d3550, 0x3d3d3300
Caught exception in looking up space point ptr... 0x59c52ab0, 0x59c52520
Caught exception in looking up space point ptr... 0x59c52520, 0x684fc5c0
Caught exception in looking up space point ptr... 0x40ffe8e0, 0x41005820
Caught exception in looking up space point ptr... 0x41005820, 0x40ffe8e0
Caught exception in looking up space point ptr... 0x40ffe8e0, 0x41005220
Caught exception in looking up space point ptr... 0x41005220, 0x40ffe8e0
Caught exception in looking up space point ptr... 0x36833d30, 0x4106b7d0
Caught exception in looking up space point ptr... 0x4106b7d0, 0x36833d30
Caught exception in looking up space point ptr... 0x111686a0, 0x45496010
Caught exception in looking up space point ptr... 0x45496010, 0x79d7b0d0
Caught exception in looking up space point ptr... 0x3885c840, 0x9fd5950
Caught exception in looking up space point ptr... 0x9fd5950, 0x3885c840
Caught exception in looking up space point ptr... 0x6d3fb440, 0x6d405380
Caught exception in looking up space point ptr... 0x79d054a0, 0x6d3fb440
Caught exception in looking up space point ptr... 0x51cc55e0, 0x51cc5490
Caught exception in looking up space point ptr... 0x51cc5490, 0x37a70de0
Caught exception in looking up space point ptr... 0x163a6e70, 0x163a7760
Caught exception in looking up space point ptr... 0x163ac960, 0x163a6e70
Caught exception in looking up space point ptr... 0x672fd2b0, 0x66bfe130
Caught exception in looking up space point ptr... 0x66bfe130, 0x672fd400
Caught exception in looking up space point ptr... 0x684cbdd0, 0x684d38c0
Caught exception in looking up space point ptr... 0x684d38c0, 0x684cbdd0
Caught exception in looking up space point ptr... 0x688048a0, 0x38882820
Caught exception in looking up space point ptr... 0x38882820, 0x688048a0
Caught exception in looking up space point ptr... 0x688048a0, 0x6151ece0
Caught exception in looking up space point ptr... 0x6151ece0, 0x688048a0
Caught exception in looking up space point ptr... 0x688048a0, 0x6857cc70
Caught exception in looking up space point ptr... 0x6857cc70, 0x688048a0
Caught exception in looking up space point ptr... 0x7728a9e0, 0x59cee0b0
Caught exception in looking up space point ptr... 0x59cee0b0, 0x7de4f0f0
Caught exception in looking up space point ptr... 0x54365ac0, 0x62972e50
Caught exception in looking up space point ptr... 0x62972e50, 0x54365ac0
++++>>>> total num hits: 99358, num free: 95578
04-Feb-2026 02:29:27 CET  Closed output file "np02vd_raw_run042433_0507_df-s02-d1_dw_0_20260203T215449_reco_stage0_20260204T012927_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         311.913       375.938       469.218       360.878       45.2503        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.1899e-05    0.000115502   0.000321866   0.000108134   5.03672e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                122.348       175.562       263.485       171.875       37.4895        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.620322      0.914798       1.54708      0.855004      0.233228        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0178847     0.0385102     0.268436      0.0186273     0.0569072       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0177263     0.0210396     0.0586082     0.0181091    0.00941676       19     
produce:beamevent:BeamEvent                      0.000115755   0.000247228   0.00146167    0.000180493   0.000288093      19     
produce:pdvddaphne:DAPHNEReaderPDVD                9.22035       15.6395       24.7648       15.7497       3.81329        19     
produce:ophit:OpHitFinder                         0.0416819     0.0499192     0.0595563     0.0502459    0.00475899       19     
produce:wclsdatavd:WireCellToolkit                 131.216       148.979       166.751        152.3        8.30886        19     
produce:gaushit:GausHitFinder                      1.10495       1.89135       2.62174       1.89502      0.467134        19     
produce:nhitsfilter:NumberOfHitsFilter           0.00038411    0.00157189    0.00368421     0.0013187    0.000931042      19     
produce:reco3d:SpacePointSolver                    13.3554       20.6767       27.9342       21.5835       4.46283        19     
produce:hitpdune:DisambigFromSpacePoints           0.14819      0.323192      0.879131      0.304859      0.161699        19     
produce:cluster3d:Cluster3D                        2.06908       4.50264        10.42        4.22458       1.73398        19     
[art]:TriggerResults:TriggerResultInserter        3.285e-05    5.16117e-05   8.8824e-05    4.9785e-05    1.42681e-05      19     
end_path:out1:RootOutput                          5.348e-06    1.04335e-05   2.2283e-05     8.573e-06    5.21146e-06      19     
end_path:out1:RootOutput(write)                    6.07675       7.23531       7.91928       7.33971      0.617038        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5368.81 MB
  Peak resident set size usage (VmHWM): 3397.27 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0507_df-s02-d1_dw_0_20260203T215449_reco_stage0_20260204T012927_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0507_df-s02-d1_dw_0_20260203T215449_reco_stage0_20260204T012927_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0507_df-s02-d1_dw_0_20260203T215449_reco_stage0_20260204T012927_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161342.0,
    "core.end_time": 1770168569.0,
    "core.events": [
      57805,
      57811,
      57817,
      57823,
      57829,
      57835,
      57841,
      57847,
      57853,
      57859,
      57865,
      57871,
      57877,
      57883,
      57889,
      57895,
      57901,
      57907,
      57913
    ],
    "core.event_count": 19,
    "core.first_event_number": 57805,
    "core.last_event_number": 57913,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0507_df-s02-d1_dw_0_20260203T215449.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:56:20 UTC       justIN version: 01.06.00