12 May

Download daymet data with R

Hey folks ! It’s been a while !

Interested in meteorological data over US ? Here is a way to easily download Daymet data inspired from DaymetR package.

Daymet is a collection of algorithms and computer software designed to interpolate and extrapolate from daily meteorological observations to produce gridded estimates of daily weather parameters. Weather parameters generated include daily surfaces of minimum and maximum temperature, precipitation, humidity, and radiation produced on a 1 km x 1 km gridded surface over the conterminous United States, Mexico, and Southern Canada.

Download the “tiles” shapefile here.

setwd("/guru-blard/DAYMET")
 
param = c("vp", "tmin", "tmax", "swe", "srad", "prcp", "dayl") #see here https://daymet.ornl.gov/overview.html for the available variable
tiles=c(11922:11925, 11742:11745) #id of the tiles of interest (cfr shapefile)
year_range=1985:2015
 
for (i in year_range) {
  for (j in tiles) {
    for (k in param) {
      download_string = sprintf("https://daymet.ornl.gov/thredds/fileServer/ornldaac/1219/tiles/%s/%s_%s/%s.nc", 
                                i, j, i, k)
      daymet_file = paste(k, "_", i, "_", j, ".nc", 
                          sep = "")
      cat(paste("Downloading DAYMET data for tile: ", 
                j, "; year: ", i, "; product: ", k, "\n", 
                sep = ""))
      try(download(download_string, daymet_file, quiet = TRUE, 
                   mode = "wb"), silent = FALSE)
    }
  }
}
01 Nov

Batch download files using R

Hello GIS world !

Here are the steps to batch download.files function in R. The example below shows how to download NDVI gimms data from ecocast.arc.nasa.gov website.

setwd("/homes/blackguru/gimms/")
 
library(XML)
url<-c("http://ecocast.arc.nasa.gov/data/pub/gimms/3g/2010s_new/")
doc<-htmlParse(url)
#get <a> nodes.
Anodes<-getNodeSet(doc,"//a")
#Select the files of interest 
files<-grep("*VI3g",sapply(Anodes, function(Anode) xmlGetAttr(Anode,"href")),value=TRUE)
#make the full url
urls<-paste(url,files,sep="")
 
#Download each file.
mapply(function(x,y) download.file(x,y),urls,files)
 
# or faster but ressource 'intensive', download each file in parallel
require(parallel)
cl <- makeCluster(detectCores())
clusterMap(cl, download.file, url = urls, destfile = files, 
           .scheduling = 'dynamic')
# 'dynamic' scheduling means that each core won't have to wait for others to finish before starting 
#  its next download

Another example with MODIS data and files in subfolders.

setwd("homes/blackguru/MODIS/")
 
library(XML)
url<-c("http://e4ftl01.cr.usgs.gov/MOLT/MOD11C3.006/")
doc<-htmlParse(url)
#get <a> nodes.
Anodes<-getNodeSet(doc,"//a")
 
files<-grep("2",sapply(Anodes, function(Anode) xmlGetAttr(Anode,"href")),value=TRUE)
urls<-paste(url,files,sep="")
 
doc<-lapply(urls, function(x){htmlParse(x)})
Anodes<-lapply(doc, function(x){getNodeSet(x,"//a")})
files<-lapply(Anodes, function(x){grep(".hdf$",sapply(x, function(Anode) xmlGetAttr(Anode,"href")),value=TRUE)})
urls<-paste(urls,files,sep="")
 
require(parallel)
cl <- makeCluster(detectCores())
clusterMap(cl, download.file, url = urls, destfile = files, 
           .scheduling = 'dynamic')