GitHub

📅 Weather Data

Several (and more coming) weather station format can be read and transformed to DataFrame.

ECA dataset

From the European Climate Assessment & Dataset project at this link for zip of all stations per variables and at this link for custom manual query. I asked them about an API to extract directly a specific file automatically, but they answer it is not currently available. I tried unzip-http but could not get it working with ECA website[1]

using StochasticWeatherGenerators, DataFrames, Dates
collect_data_ECA(33, Date(1956), Date(2019, 12, 31), "https://raw.githubusercontent.com/dmetivie/StochasticWeatherGenerators.jl/master/weather_files/ECA_blend_rr/RR_", portion_valid_data=1, skipto=22, header=21, url=true)[1:10,:]
10×5 DataFrame
RowSTAIDSOUIDDATERRQ_RR
Int64Int64DateInt64Int64
1331051956-01-01230
2331051956-01-0210
3331051956-01-0300
4331051956-01-0400
5331051956-01-0500
6331051956-01-06210
7331051956-01-0730
8331051956-01-08250
9331051956-01-09200
10331051956-01-1000
StochasticWeatherGenerators.collect_data_ECA — Function
collect_data_ECA(STAID::Integer, path::String, var::String="RR"; skipto=19, header = 18)

path gives the path where all data files are stored in a vector

source
collect_data_ECA(STAID, date_start::Date, date_end::Date, path::String, var::String="RR"; portion_valid_data=1, skipto=19, header = 18, return_nothing = true)
  • path gives the path where all data files are stored in a vector
  • Filter the DataFrame s.t. date_start ≤ :DATE ≤ date_end
  • var = "RR", "TX" etc.
  • portion_valid_data is the portion of valid data we are ok with. If we don't want any missing, fix it to 1.
  • skipto and header for csv files with meta informations/comments at the beginning of files. See CSV.jl.
  • return_nothing if true it will return nothing is the file does not exists or does not have enough valid data.
source

Météo France

Météo France do have a version of this data and it is accessible through an API on the website Data.Gouv.fr. This package provides a simple command to extract the data of one station (given its STAtionID) from the API.

collect_data_MeteoFrance(49215002)[1:10,:]
10×141 DataFrame
Row__idSTAIDSTANAMELATLONALTIDATERRQRRTNQTNHTNQHTNTXQTXHTXQHTXTMQTMTNTXMQTNTXMTAMPLIQTAMPLITNSOLQTNSOLTN50QTN50DGQDGFFMQFFMFF2MQFF2MFXYQFXYDXYQDXYHXYQHXYFXIQFXIDXIQDXIHXIQHXIFXI2QFXI2DXI2QDXI2HXI2QHXI2FXI3SQFXI3SDXI3SQDXI3SHXI3SQHXI3SDRRQDRRDHUMECQDHUMECPMERMQPMERMPMERMINQPMERMININSTQINSTGLOTQGLOTDIFTQDIFTDIRTQDIRTINFRARTQINFRARTUVQUVUV_INDICEXQUV_INDICEXSIGMAQSIGMAUNQUNHUNQHUNUXQUXHUXQHUXUMQUMDHUMI40QDHUMI40DHUMI80QDHUMI80TSVMQTSVMETPMONQETPMONETPGRILLEQETPGRILLEECOULEMENTMQECOULEMENTMHNEIGEFQHNEIGEFNEIGETOTXQNEIGETOTXNEIGETOT06QNEIGETOT06NEIGQNEIGBROUQBROUORAGQORAGGRESILQGRESILGRELEQGRELEROSEEQROSEEVERGLASQVERGLASSOLNEIGEQSOLNEIGEGELEEQGELEEFUMEEQFUMEEBRUMEQBRUMEECLAIRQECLAIRNB300QNB300BA300QBA300TMERMINQTMERMINTMERMAXQTMERMAX
Int64Int64String31Float64Float64Int64DateFloat64?Int64?Float64Int64Int64?Int64?Float64Int64Int64?Int64?Float64?Int64?Float64Int64Float64Int64Float64?Int64?Float64?Int64?Int64?Int64?MissingMissingFloat64?Int64?Int64?Int64?Int64?Int64?Int64?Int64?Int64?Int64?Int64?Int64?Int64?Int64?Float64?Int64?Int64?Int64?Int64?Int64?MissingMissingMissingMissingMissingMissingMissingInt64?Int64?Int64?MissingMissingMissingMissingMissingMissingInt64?Int64?MissingMissingMissingMissingMissingMissingMissingMissingMissingMissingMissingMissingInt64?Int64?Int64?Int64?Int64?Int64?Int64?Int64?Int64?Int64?Int64?Int64?Int64?Int64?Float64?Int64?Float64?Int64?Float64?Int64?MissingMissingMissingMissingMissingMissingMissingMissingString1?Int64?String1?Int64?String1?Int64?MissingMissingString1?Int64?MissingMissingMissingMissingMissingMissingMissingMissingMissingMissingMissingMissingMissingMissingMissingMissingMissingMissingMissingMissingMissingMissing
156638749215002MONT-BELLAY-INRAE47.13-0.1415581986-01-01missingmissing-4.31missingmissing7.01missingmissingmissingmissing1.4111.31missingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissing
256638849215002MONT-BELLAY-INRAE47.13-0.1415581986-01-02missingmissing5.51missingmissing8.01missingmissingmissingmissing6.812.51missingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissing
356638949215002MONT-BELLAY-INRAE47.13-0.1415581986-01-03missingmissing5.71missingmissing8.01missingmissingmissingmissing6.912.31missingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissing
456639049215002MONT-BELLAY-INRAE47.13-0.1415581986-01-04missingmissing-1.31missingmissing5.01missingmissingmissingmissing1.916.31missingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissing
556639149215002MONT-BELLAY-INRAE47.13-0.1415581986-01-05missingmissing1.41missingmissing7.31missingmissingmissingmissing4.415.91missingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissing
656639249215002MONT-BELLAY-INRAE47.13-0.1415581986-01-06missingmissing2.41missingmissing8.01missingmissingmissingmissing5.215.61missingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissing
756639349215002MONT-BELLAY-INRAE47.13-0.1415581986-01-07missingmissing2.11missingmissing8.01missingmissingmissingmissing5.115.91missingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissing
856639449215002MONT-BELLAY-INRAE47.13-0.1415581986-01-08missingmissing4.41missingmissing11.01missingmissingmissingmissing7.716.61missingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissing
956639549215002MONT-BELLAY-INRAE47.13-0.1415581986-01-09missingmissing5.91missingmissing8.01missingmissingmissingmissing7.012.11missingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissing
1056639649215002MONT-BELLAY-INRAE47.13-0.1415581986-01-10missingmissing2.71missingmissing11.41missingmissingmissingmissing7.118.71missingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissingmissing
StochasticWeatherGenerators.collect_data_MeteoFrance — Function
collect_data_MeteoFrance(STAID; show_warning=false, impute_missing=[], period="1950-2021", variables = "all")

Given a STAID (station ID given by Météo France), it returns a DataFrame with data in period and for the variables.

  • STAID can be an integer or string.
  • Option for period are "1846-1949", "1950-2021", "2022-2023"
  • Option for variables are all, "RR-T-Wind", "others"
  • impute_missing expects a vector of column name(s) where to impute missing with Impute.Interpolate e.g. impute_missing=[:TX].
  • show_warning in case of missing data. false for no column, true for all variables columns and for selected columns e.g. show_warning = [:TX].

The data is available through the French Data.gouv.fr website api. Data may be updated without notice. See the following two links to get informations on the "RR-T-Wind" and "others" variables (in French)

  • https://object.files.data.gouv.fr/meteofrance/data/synchroftp/BASE/QUOT/QdescriptifchampsRR-T-Vent.csv
  • https://object.files.data.gouv.fr/meteofrance/data/synchroftp/BASE/QUOT/Qdescriptifchampsautres-parametres.csv

Or the the SICLIMA website with information (in French) about computation and conversion for some weather variables/index.

source

INRAE

The INRAE CLIMATIK platform (Delannoy et al., 2022) (https://agroclim.inrae.fr/climatik/, in French) managed by the AgroClim laboratory of Avignon, France has weather stations. However, their API is not open access.

StochasticWeatherGenerators.collect_data_INRAE — Function
collect_data_INRAE(station_path::String; show_warning=false, impute_missing=[])

Read from a file an INRAE formatted weather station data and transform it to match ECA standard naming conventions.

  • impute_missing expects a vector of column name(s) where to impute missing with Impute.Interpolate e.g. impute_missing=[:TX].
  • show_warning in case of missing data. false for no column, true for all variables columns and for selected columns e.g. show_warning = [:TX].
source

Others

Data manipulation

StochasticWeatherGenerators.clean_data — Function
clean_data(df::DataFrame; show_warning=false, impute_missing=[])

Impute missing and show warning for missings. It assumes that the first two columns are not numeric.

  • impute_missing expects a vector of column name(s) where to impute missing with Impute.Interpolate e.g. impute_missing=[:TX].
  • show_warning in case of missing data. false for no column, true for all variables columns and for selected columns e.g. show_warning = [:TX].
source
StochasticWeatherGenerators.select_in_range_df — Function
select_in_range_df(datas, start_Date, interval_Date, [portion])

Select station with some data availability in dates and quality (portion of valid data). Input is a vector (array) of DataFrame (one for each station for example) or a Dict of DataFrame. If 0 < portion ≤ 1 is specified, it will authorize some portion of data to be missing.

source
StochasticWeatherGenerators.shortname — Function
shortname(name::String)

Experimental function that returns only the most relevant part of a station name.

long_name = "TOULOUSE-BLAGNAC"
shortname(long_name) # "TOULOUSE"
source

References

  • Delannoy, D.; Maury, O. and Décome, J. (2022). CLIMATIK: système d’information pour les données du réseau agroclimatique INRAE.
  • 1I don't remember exactly in fact.