justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 20401.62@dunegpschedd01.fnal.gov

Jobsub ID20401.62@dunegpschedd01.fnal.gov
Workflow ID249
Stage ID1
User nameepennacc@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-08-02 09:11:09
SiteUK_Glasgow
EntryCLAS12_T3_UK_ScotGrid_GLA_ce04_scitok
Last heartbeat2025-08-02 10:01:53
From worker nodeHostnamewn-h17-041.beowulf.cluster
cpuinfoIntel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-02 09:51:35
Input filesmonte-carlo-000249-008901
JobscriptExit code0
Real time9m (590s)
CPU time9m (574s = 97%)
Max RSS bytes1044631552 (996 MiB)
Outputting started2025-08-02 10:01:26
Output files
Finished2025-08-02 10:01:53
Saved logsjustin-logs:20401.62-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

races
[11:00:25.278] W [sigproc ] <OmnibusSigProc:apa7sigproc7> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[11:00:25.291] W [sigproc ] <OmnibusSigProc:apa8sigproc8> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[11:00:25.305] W [sigproc ] <OmnibusSigProc:apa9sigproc9> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[11:00:25.318] W [sigproc ] <OmnibusSigProc:apa10sigproc10> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[11:00:25.331] W [sigproc ] <OmnibusSigProc:apa11sigproc11> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
Inputer: "wclsSimDepoSource:"
Outputer: "wclsSimChannelSink:postdrift"
Outputer: "wclsFrameSaver:spsignals"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
wclsFrameSaver: promising to produce recob::Wires named "dnnsp"
ProvidedPedestalAdditionService::dtor: Deleting random engine with seed 669708517
ExponentialChannelNoiseService::dtor: Deleting random engine with seed 1299961164
StuckBitAdcDistortionService:dtor: Deleting random engine with seed 9876
%MSG-s ArtException:  TriggerResultInserter:TriggerResults@Construction  02-Aug-2025 11:00:25 BST ModuleConstruction
cet::exception caught in art
---- CatalogServiceError BEGIN
  Input file not found: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T095140Z_gen_008901_supernova_g4.root.
---- CatalogServiceError END
%MSG
Art has completed and will exit with status 24.
ls: cannot access *_20250802T095140Z_*: No such file or directory
ERROR: lar (detsim) exit code: 41024 
output files size: 
../justin-jobscript: line 155: return: can only `return' from a function or sourced script
============reco =========================
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  anglereconue
  anglereconuepfps
  anglereconumu
  anglereconumupfps
  blurredcluster
  caldata
  cvneva
  cvnmap
  emshower
  emtrkmichelid
  energyrecnc
  energyrecnue
  energyrecnumu
  energyrecnumumcs
  energyrecnumurange
  linecluster
  opdec
  opflash
  ophitspe
  opslicer
  pandora
  pandoraShower
  pandoraTrack
  pandoracalo
  pandorapid
  pmtrack
  pmtrackcalo
  pmtrackpid
  pmtracktc
  pmtracktccalo
  pmtracktcpid
  pmtrajfit
  pmtrajfitcalo
  pmtrajfitpid
  pmtrajfittc
  pmtrajfittccalo
  pmtrajfittcpid
  trajcluster
  trkshowersplit
  wclsmcnfsp
DuneDeconvolutionService::ctor: DuneDeconvolutionService:
DuneDeconvolutionService::ctor:                LogLevel: 1
DuneRoiBuildingService::ctor: DuneRoiBuildingService:
DuneRoiBuildingService::ctor:     LogLevel: 1
DuneRoiBuildingService::ctor:  NSigmaStart: 3
DuneRoiBuildingService::ctor:    NSigmaEnd: 1
DuneRoiBuildingService::ctor:       PadLow: 50
DuneRoiBuildingService::ctor:      PadHigh: 50
StandardAdcWireBuildingService::ctor: StandardAdcWireBuildingService:
StandardAdcWireBuildingService::ctor:     LogLevel: 1
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v09_91_04d00/gdml/dune10kt_v6_refactored_1x2x6.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Error: Unsupported GDML Tag Used :gdml_simple_extension. Please Check Geometry/Schema.
Error: Unsupported GDML Tag Used :extension. Please Check Geometry/Schema.
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1149
Info in <TGeoManager::CloseGeometry>: 67545 nodes/ 5439 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
GeoApaChannelGroupService::ctor: Group 0 (apa00) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 1 (apa01) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 2 (apa02) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 3 (apa03) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 4 (apa04) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 5 (apa05) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 6 (apa06) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 7 (apa07) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 8 (apa08) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 9 (apa09) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 10 (apa10) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 11 (apa11) has 2560 channels from 4/4 readout planes.
DuneToolManager::fclFilename: Taking fcl name from command line: lar -c reco1_supernova_dune10kt_1x2x6.fcl prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T095140Z_gen_008901_supernova_g4_detsim.root -o prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T095140Z_gen_008901_supernova_g4_detsim_reco.root -n -1
AcdDigitReader::ctor:     LogLevel: 1
DuneToolManager::getPrivate: ERROR: Tool name is blank
StandardRawDigitExtractService::ctor: Retrieved digit read tool digitReader
StandardRawDigitExtractService::ctor: StandardRawDigitExtractService:
StandardRawDigitExtractService::ctor:         LogLevel: 1
StandardRawDigitExtractService::ctor:    DigitReadTool: digitReader
StandardRawDigitExtractService::ctor:   PedestalOption: 1
StandardRawDigitExtractService::ctor:     FlagStuckOff: 0
StandardRawDigitExtractService::ctor:      FlagStuckOn: 0
StandardRawDigitPrepService::ctor: Fetching extract service.
StandardRawDigitPrepService::ctor: Fetching channel status provider.
StandardRawDigitPrepService::ctor:   Channel status provider: @0x25e6290
StandardRawDigitPrepService::ctor:   Extract service: @0x58b7330
StandardRawDigitPrepService::ctor: Fetching deconvolution service.
StandardRawDigitPrepService::ctor:   Deconvolution service: @0x25e67c0
StandardRawDigitPrepService::ctor: Fetching ROI building service.
StandardRawDigitPrepService::ctor:   ROI building service: @0x51722b0
StandardRawDigitPrepService::ctor: Fetching wire building service.
StandardRawDigitPrepService::ctor:   Wire building service: @0x5172280
StandardRawDigitPrepService::ctor: StandardRawDigitPrepService:
StandardRawDigitPrepService::ctor:              LogLevel: 1
StandardRawDigitPrepService::ctor:               SkipBad: 1
StandardRawDigitPrepService::ctor:             SkipNoisy: 0
StandardRawDigitPrepService::ctor:   ChannelStatusOnline: 0
StandardRawDigitPrepService::ctor:          DoMitigation: 0
StandardRawDigitPrepService::ctor:  DoEarlySignalFinding: 0
StandardRawDigitPrepService::ctor:        DoNoiseRemoval: 0
StandardRawDigitPrepService::ctor:       DoDeconvolution: 1
StandardRawDigitPrepService::ctor:  DoPedestalAdjustment: 0
StandardRawDigitPrepService::ctor:                 DoROI: 1
StandardRawDigitPrepService::ctor:               DoWires: 1
StandardRawDigitPrepService::ctor:                DoDump: 0
StandardRawDigitPrepService::ctor:  DoIntermediateStates: 0
StandardRawDigitPrepService::ctor: No display tools.
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
%MSG-s ArtException:  TriggerResultInserter:TriggerResults@Construction  02-Aug-2025 11:01:23 BST ModuleConstruction
cet::exception caught in art
---- CatalogServiceError BEGIN
  Input file not found: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T095140Z_gen_008901_supernova_g4_detsim.root.
---- CatalogServiceError END
%MSG
Art has completed and will exit with status 24.
ls: cannot access *_20250802T095140Z_*: No such file or directory
ERROR: lar (reco) exit code: 51024 
output files size: 
../justin-jobscript: line 181: return: can only `return' from a function or sourced script
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 373, in main
    mddict = expSpecificMetadata.getmetadata()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 344, in getmetadata
    jobt = self.get_job(proc)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 69, in get_job
    raise RuntimeError('sam_metadata_dumper returned nonzero exit status {}.'.format(rc))
RuntimeError: sam_metadata_dumper returned nonzero exit status 1.
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T095140Z_gen_008901_supernova_g4_detsim_reco.root.ext.json 0
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T095140Z_gen_008901_supernova_g4_detsim_reco.root.json 0
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T095140Z_gen_008901_supernova_g4_detsim_reco.root.temp.json 0
justIN time: 2025-08-04 14:16:53 UTC       justIN version: 01.04.00