justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297920.43@dunegpschedd02.fnal.gov

Jobsub ID297920.43@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:27
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_dissel_multicore
Last heartbeat2026-02-04 01:42:06
From worker nodeHostnamewn-pep-003.farm.nikhef.nl
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:38:26
Input filesvd-protodune:np02vd_raw_run042433_0567_df-s03-d1_dw_0_20260203T221736.hdf5
JobscriptExit code0
Real time2h (7285s)
CPU time57m (3479s = 47%)
Max RSS bytes3412299776 (3254 MiB)
Outputting started2026-02-04 01:39:53
Output files
Finished2026-02-04 01:42:06
Saved logsjustin-logs:297920.43-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

.. 0xb99e08a0, 0xb99e0b00
Caught exception in looking up space point ptr... 0xa2d081f0, 0xaf5d4420
Caught exception in looking up space point ptr... 0xaf5d4420, 0xaf5d45b0
Caught exception in looking up space point ptr... 0xaef61520, 0xa00f4620
Caught exception in looking up space point ptr... 0x85933ea0, 0xaef61520
Caught exception in looking up space point ptr... 0x44c77920, 0x99191dc0
Caught exception in looking up space point ptr... 0x99191dc0, 0x44c77920
Caught exception in looking up space point ptr... 0x44c77920, 0x74acb6c0
Caught exception in looking up space point ptr... 0x74acb6c0, 0x44c77920
Caught exception in looking up space point ptr... 0x5cb97440, 0x5cb98820
Caught exception in looking up space point ptr... 0x5cb98820, 0x5cb97440
Caught exception in looking up space point ptr... 0x6b6c6ed0, 0x7574b030
Caught exception in looking up space point ptr... 0x7574b030, 0x6b6c6ed0
Caught exception in looking up space point ptr... 0xaf47dcb0, 0x4531a2d0
Caught exception in looking up space point ptr... 0x4531a2d0, 0xaf47dcb0
Caught exception in looking up space point ptr... 0x62b26680, 0x62b22d60
Caught exception in looking up space point ptr... 0x62b22d60, 0x62b26680
Caught exception in looking up space point ptr... 0x62b26680, 0x62b1f500
Caught exception in looking up space point ptr... 0x62b1f500, 0x62b26680
Caught exception in looking up space point ptr... 0x3bd6e5c0, 0x3bd6e460
Caught exception in looking up space point ptr... 0x3bd6e460, 0x3bd6e5c0
Caught exception in looking up space point ptr... 0x78951ab0, 0xadae4030
Caught exception in looking up space point ptr... 0xadae4030, 0x8b207210
Caught exception in looking up space point ptr... 0x92016da0, 0x40698da0
Caught exception in looking up space point ptr... 0x40698da0, 0x40698ed0
Caught exception in looking up space point ptr... 0x7a97a650, 0x7a97a8a0
Caught exception in looking up space point ptr... 0xad895920, 0x7a97a650
Caught exception in looking up space point ptr... 0x82eb6c10, 0x82eb6a80
Caught exception in looking up space point ptr... 0x82eb6a80, 0x82eb6c10
Caught exception in looking up space point ptr... 0x78555240, 0x44bd36a0
Caught exception in looking up space point ptr... 0x44bd36a0, 0x78555240
Caught exception in looking up space point ptr... 0x7266f720, 0x7266f260
Caught exception in looking up space point ptr... 0x7266f260, 0x44584750
Caught exception in looking up space point ptr... 0x7475a9f0, 0x5e148d30
Caught exception in looking up space point ptr... 0x5e148d30, 0x72309f60
Caught exception in looking up space point ptr... 0x72c5a650, 0x72c5a790
Caught exception in looking up space point ptr... 0x72c5a790, 0x72c5a650
Caught exception in looking up space point ptr... 0x79d5b0a0, 0x72257180
Caught exception in looking up space point ptr... 0x72257180, 0x79d5af40
Caught exception in looking up space point ptr... 0x780203b0, 0x780202b0
Caught exception in looking up space point ptr... 0x780202b0, 0x780203b0
Caught exception in looking up space point ptr... 0x4b429e30, 0x40bf60e0
Caught exception in looking up space point ptr... 0x40bf60e0, 0x4b429e30
Caught exception in looking up space point ptr... 0x4b429e30, 0x6ade4410
Caught exception in looking up space point ptr... 0x6ade4410, 0x4b429e30
Caught exception in looking up space point ptr... 0x787a1520, 0x5e5ef850
Caught exception in looking up space point ptr... 0x5e5ef850, 0x787a1520
Caught exception in looking up space point ptr... 0x82e95670, 0x82e94830
Caught exception in looking up space point ptr... 0x82e94830, 0xb45283d0
Caught exception in looking up space point ptr... 0x6de99000, 0x6de98f00
Caught exception in looking up space point ptr... 0x6de98f00, 0x6de99000
Caught exception in looking up space point ptr... 0x98c3f030, 0x6f92cdf0
Caught exception in looking up space point ptr... 0x6f92cdf0, 0x98c3f030
Caught exception in looking up space point ptr... 0x82f5bef0, 0xa1cf06d0
Caught exception in looking up space point ptr... 0xa1cf06d0, 0x82f5c0d0
Caught exception in looking up space point ptr... 0x9d0299c0, 0x72edda00
Caught exception in looking up space point ptr... 0x72edda00, 0x6a6e5b00
Caught exception in looking up space point ptr... 0x81720140, 0x485ec460
Caught exception in looking up space point ptr... 0x485ec460, 0x6961c070
Caught exception in looking up space point ptr... 0x74922b80, 0x74922a80
Caught exception in looking up space point ptr... 0x40bc6c60, 0x74922b80
Caught exception in looking up space point ptr... 0x9cd8f740, 0x9cd876f0
Caught exception in looking up space point ptr... 0x9cd876f0, 0x9cd7f9d0
Caught exception in looking up space point ptr... 0xb3e64680, 0x99226e40
Caught exception in looking up space point ptr... 0x99226e40, 0xb3e64680
Caught exception in looking up space point ptr... 0x88355960, 0x99e42930
Caught exception in looking up space point ptr... 0x99e42930, 0x88355960
++++>>>> total num hits: 71771, num free: 69123
04-Feb-2026 02:39:46 CET  Closed output file "np02vd_raw_run042433_0567_df-s03-d1_dw_0_20260203T221736_reco_stage0_20260204T013946_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         300.29        379.127       475.261       370.217       43.7828        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.2502e-05    0.000142024   0.000381803   0.000130836   6.41917e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                132.676       194.066       290.86        190.003       39.8697        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.715988       1.03785       1.63932      0.965254      0.226408        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0174452     0.0179163     0.0219714     0.0177263    0.000966259      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0174065     0.0176315     0.0179105     0.0176174    0.000142417      19     
produce:beamevent:BeamEvent                      0.000120442   0.000267783   0.00101461    0.000222643   0.000182373      19     
produce:pdvddaphne:DAPHNEReaderPDVD                13.4005       18.6347       24.2262       17.3836       2.89656        19     
produce:ophit:OpHitFinder                         0.0318573     0.0375816     0.042603      0.0377653    0.00334336       19     
produce:wclsdatavd:WireCellToolkit                 133.551       135.59        137.906       135.125       1.43133        19     
produce:gaushit:GausHitFinder                      1.15007       1.76143       2.4588        1.84929      0.411435        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000487114   0.000673791   0.000954958   0.000672949   0.000124388      19     
produce:reco3d:SpacePointSolver                    10.057        18.1887       29.7716       19.273        5.03963        19     
produce:hitpdune:DisambigFromSpacePoints          0.151016      0.259922      0.402139      0.266449      0.0757586       19     
produce:cluster3d:Cluster3D                        1.42053       2.96133       5.63224       2.86501       1.16853        19     
[art]:TriggerResults:TriggerResultInserter       3.3076e-05    4.05034e-05    9.148e-05    3.6133e-05    1.28343e-05      19     
end_path:out1:RootOutput                          6.786e-06    9.29284e-06   2.4818e-05     8.162e-06    4.03717e-06      19     
end_path:out1:RootOutput(write)                    5.31301       6.50245       7.45878       6.50391      0.550166        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5374.43 MB
  Peak resident set size usage (VmHWM): 3412.3 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0567_df-s03-d1_dw_0_20260203T221736_reco_stage0_20260204T013946_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0567_df-s03-d1_dw_0_20260203T221736_reco_stage0_20260204T013946_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0567_df-s03-d1_dw_0_20260203T221736_reco_stage0_20260204T013946_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161915.0,
    "core.end_time": 1770169187.0,
    "core.events": [
      64647,
      64653,
      64659,
      64665,
      64671,
      64677,
      64683,
      64689,
      64695,
      64701,
      64707,
      64713,
      64719,
      64725,
      64731,
      64737,
      64743,
      64749,
      64755
    ],
    "core.event_count": 19,
    "core.first_event_number": 64647,
    "core.last_event_number": 64755,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0567_df-s03-d1_dw_0_20260203T221736.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:56:24 UTC       justIN version: 01.06.00