justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303651.71@dunegpschedd01.fnal.gov

Jobsub ID303651.71@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:24
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod03
Last heartbeat2026-02-04 00:33:21
From worker nodeHostnamewk03.grid.hep.ph.ic.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:19:14
Input filesvd-protodune:np02vd_raw_run042421_0140_df-s04-d1_dw_0_20260203T144643.hdf5
JobscriptExit code0
Real time1h (4302s)
CPU time1h (3839s = 89%)
Max RSS bytes2591444992 (2471 MiB)
Outputting started2026-02-04 00:30:58
Output files
Finished2026-02-04 00:33:21
Saved logsjustin-logs:303651.71-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

77920
Caught exception in looking up space point ptr... 0x3e0c9d00, 0x56481670
Caught exception in looking up space point ptr... 0x56481670, 0x3e0c9d00
Caught exception in looking up space point ptr... 0x3e0c9d00, 0x3b81df70
Caught exception in looking up space point ptr... 0x3b81df70, 0x3e0c9d00
Caught exception in looking up space point ptr... 0x270633b0, 0x27062100
Caught exception in looking up space point ptr... 0x27062100, 0x270633b0
Caught exception in looking up space point ptr... 0x235610b0, 0x3d6edae0
Caught exception in looking up space point ptr... 0x3d6edae0, 0x3b2530b0
Caught exception in looking up space point ptr... 0x577ebbb0, 0x43d984c0
Caught exception in looking up space point ptr... 0x43d984c0, 0x577ebbb0
Caught exception in looking up space point ptr... 0x2a21ab00, 0x2a21bb00
Caught exception in looking up space point ptr... 0x2a21bb00, 0x2a21bca0
Caught exception in looking up space point ptr... 0x484d90e0, 0x15345480
Caught exception in looking up space point ptr... 0x15345480, 0x484d90e0
Caught exception in looking up space point ptr... 0x46af5600, 0x2b9b0400
Caught exception in looking up space point ptr... 0x2b9b0400, 0x46af5600
Caught exception in looking up space point ptr... 0x15188b20, 0x2bb6e090
Caught exception in looking up space point ptr... 0x2bb6e090, 0x15188b20
Caught exception in looking up space point ptr... 0x4f19ca80, 0x4f19c8f0
Caught exception in looking up space point ptr... 0x4f19c8f0, 0x4f19ca80
Caught exception in looking up space point ptr... 0x46dbd2b0, 0x46db6820
Caught exception in looking up space point ptr... 0x46db6820, 0x46db66f0
Caught exception in looking up space point ptr... 0x425b6950, 0x425b6800
Caught exception in looking up space point ptr... 0x425b6800, 0x425b6510
Caught exception in looking up space point ptr... 0x2a4e6640, 0x4c8743b0
Caught exception in looking up space point ptr... 0x4c8743b0, 0x2a4e6640
Caught exception in looking up space point ptr... 0x3b91aaa0, 0x3b91a800
Caught exception in looking up space point ptr... 0x3b91a800, 0x3b91aaa0
Caught exception in looking up space point ptr... 0xbca8dc0, 0x15130140
Caught exception in looking up space point ptr... 0x15130140, 0x555d5520
Caught exception in looking up space point ptr... 0x46d25bc0, 0x46d25960
Caught exception in looking up space point ptr... 0x46d25960, 0x46d25a90
Caught exception in looking up space point ptr... 0x14fab880, 0x14fa9c00
Caught exception in looking up space point ptr... 0x4f1b91c0, 0x14facaf0
Caught exception in looking up space point ptr... 0x14facaf0, 0x14fab880
Caught exception in looking up space point ptr... 0x242c1710, 0x3b9108f0
Caught exception in looking up space point ptr... 0x3b9108f0, 0x28bec990
Caught exception in looking up space point ptr... 0x28bec990, 0x3556f2b0
Caught exception in looking up space point ptr... 0x4f1ab6a0, 0x242c1710
Caught exception in looking up space point ptr... 0x2c496440, 0x2c494760
Caught exception in looking up space point ptr... 0x2c494760, 0x2c492a50
Caught exception in looking up space point ptr... 0x2b3f67f0, 0x2b3f6700
Caught exception in looking up space point ptr... 0x2b3f6700, 0x2b3f65c0
Caught exception in looking up space point ptr... 0x1556c470, 0x4318f900
Caught exception in looking up space point ptr... 0x4318f900, 0xe502190
Caught exception in looking up space point ptr... 0x2aef04b0, 0xbca1a80
Caught exception in looking up space point ptr... 0xbca1a80, 0x86f3860
Caught exception in looking up space point ptr... 0x86f3860, 0xbd73c40
Caught exception in looking up space point ptr... 0xbd73c40, 0x2aef04b0
Caught exception in looking up space point ptr... 0x46d31ce0, 0x43eba570
Caught exception in looking up space point ptr... 0x43eba570, 0x46d31ce0
Caught exception in looking up space point ptr... 0x42596500, 0x42596320
Caught exception in looking up space point ptr... 0x42596320, 0x42596500
Caught exception in looking up space point ptr... 0x48038840, 0x87ae000
Caught exception in looking up space point ptr... 0x87ae000, 0x48038840
Caught exception in looking up space point ptr... 0x509e6fe0, 0x4f30d1f0
Caught exception in looking up space point ptr... 0x4f30d1f0, 0x509e6fe0
Caught exception in looking up space point ptr... 0x109cfa00, 0xba295c0
Caught exception in looking up space point ptr... 0xba295c0, 0x109cfa00
Caught exception in looking up space point ptr... 0x28197f80, 0x28196ee0
Caught exception in looking up space point ptr... 0x28196ee0, 0x281945c0
Caught exception in looking up space point ptr... 0x205a3410, 0x205a35a0
Caught exception in looking up space point ptr... 0x205a35a0, 0x205a3280
Caught exception in looking up space point ptr... 0x425a73a0, 0x425a6ea0
Caught exception in looking up space point ptr... 0x425a6ea0, 0x425a73a0
++++>>>> total num hits: 42410, num free: 40413
04-Feb-2026 00:30:47 GMT  Closed output file "np02vd_raw_run042421_0140_df-s04-d1_dw_0_20260203T144643_reco_stage0_20260204T003047_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         119.265       155.491       225.61        146.622       28.1197        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.2794e-05    9.01093e-05   0.000291767   6.8552e-05    4.3879e-05       27     
produce:tpcrawdecoder:PDVDTPCReader                12.2978       14.3468       43.8063       13.0834       5.80999        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.486483      0.514755      0.611173      0.505084      0.0248537       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0161871     0.0172223     0.0340106     0.016319     0.00346851       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0801803     0.0813184     0.0966125     0.080613     0.00303214       27     
produce:beamevent:BeamEvent                      8.1719e-05    0.000159219   0.00139132    9.8812e-05    0.000243152      27     
produce:pdvddaphne:DAPHNEReaderPDVD                5.36076       6.93974       9.01765       6.94537       1.10978        27     
produce:ophit:OpHitFinder                         0.0351018     0.059731      0.0840529     0.0675418     0.0189351       27     
produce:wclsdatavd:WireCellToolkit                 79.6668       106.463       139.382       98.7075       23.0924        27     
produce:gaushit:GausHitFinder                     0.746737       1.46718       2.59919       1.39159      0.412069        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00027568    0.000435161   0.000722409   0.000415446   0.000114452      27     
produce:reco3d:SpacePointSolver                    5.97041       14.0202       27.8167       13.5114       5.03305        27     
produce:hitpdune:DisambigFromSpacePoints          0.0878193     0.189689      0.399653      0.172521      0.0734197       27     
produce:cluster3d:Cluster3D                        1.25989       3.2099        9.71585       2.77521       1.72485        27     
[art]:TriggerResults:TriggerResultInserter        2.569e-05    4.74053e-05   0.00027626     3.505e-05    4.68188e-05      27     
end_path:out1:RootOutput                          3.964e-06     6.93e-06     2.2644e-05     5.667e-06    3.66755e-06      27     
end_path:out1:RootOutput(write)                    5.78793       8.15502       10.8746       7.77197       1.57069        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4497.94 MB
  Peak resident set size usage (VmHWM): 2591.44 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0140_df-s04-d1_dw_0_20260203T144643_reco_stage0_20260204T003047_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0140_df-s04-d1_dw_0_20260203T144643_reco_stage0_20260204T003047_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0140_df-s04-d1_dw_0_20260203T144643_reco_stage0_20260204T003047_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770160768.0,
    "core.end_time": 1770165050.0,
    "core.events": [
      22751,
      22757,
      22763,
      22769,
      22775,
      22781,
      22787,
      22793,
      22799,
      22805,
      22811,
      22817,
      22823,
      22829,
      22835,
      22841,
      22847,
      22853,
      22859,
      22865,
      22871,
      22877,
      22883,
      22889,
      22895,
      22901,
      22907
    ],
    "core.event_count": 27,
    "core.first_event_number": 22751,
    "core.last_event_number": 22907,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0140_df-s04-d1_dw_0_20260203T144643.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:52 UTC       justIN version: 01.06.00