Data quality analysis

This Forum cover all data issues, from sources to quality, ground or remote sensing etc.

Moderator: michele49

Data quality analysis

Postby joabwamari » Thu Dec 04, 2014 1:27 pm

I've got plenty a series of daily climatic data between the year 1976 to 2007 and would like to be assisted on quality analysis and also institute quality control so as to be confident in its utility in crop modelling.
joab :?:
joabwamari
 
Posts: 1
Joined: Sat Nov 01, 2014 6:29 pm

Re: Data quality analysis

Postby rene » Mon Jan 05, 2015 2:42 am

Hello Joabwamari

there are basically three methods:

1. "Horizontal comparison" which compares data with those of neighbouring stations. Double mass plotting is such a method which is very common among hydrologists for rainfall;

2. "Vertical methods" which analyse time series and identify "outliers" using various statistical approaches. This is usually tricky, except for very unusual values, for instance 865 mm of rain per day in Khartoum, a semi-arid place;

3. Consistency checks that are based on the correlations between different variables. For instance, you do not expect any rain when you have no clouds, or if moisture is high, minimum temperature will not be able to drop below the dew point.

Most methods (and certainly 2.) assume that data are stationary, i.e. that they are not affected by a time trend. The same applies to 1 which assumes no spatial gradient (i.e. the climatic variable is not correlated with latitude and longitude). This is to say that the data have somehow to be "normalised" before they can be inter-compared, for instance to correct for the above mentioned time and space gradient, but also for elevation, sometimes slope and aspect.

There exists software to quality check climate data. I shall now drop a line to some experts to ask them to have a look at this question of yours and to provide some more technical answers. If they cannot write the answers directly, I'll ask them to give me some references which I ll share with you.

Regards

R.
Last edited by rene on Thu Jan 08, 2015 2:45 am, edited 1 time in total.
User avatar
rene
 
Posts: 36
Joined: Sun Aug 17, 2014 9:57 am
Location: Beijing

Re: Data quality analysis

Postby Crusoi » Wed Jan 07, 2015 12:39 pm

Hi Joabwamari,

As Rene said, there are different approaches (spatially, temporally, physical consistency). And some scientists spend their whole career on just one of these aspects. If you are a bit more specific on what issues you face with what kind of data I might point you in a more specific direction.

Cheers,

Juergen.
Crusoi
 
Posts: 1
Joined: Wed Jan 07, 2015 10:02 am

Re: Data quality analysis

Postby rene » Thu Jan 08, 2015 2:43 am

Here is a couple of references that I got from Dominique Fasbender at EC/JRC but, as Jürgen noted, you should be a bit more specific and explain your needs.

Claudie Beaulieu, Taha B.M.J. Ouarda, Ousmane Seidou. (2010) A Bayesian normal homogeneity test for the detection of artificial discontinuities in climatic series. International Journal of Climatology 30:15, 2342-2357.
Online publication date: 1-Dec-2010.

Claudie Beaulieu, Ousmane Seidou, Taha B. M. J. Ouarda, Xuebin Zhang. (2009) Intercomparison of homogenization techniques for precipitation data continued: Comparison of two recent Bayesian change point models. Water Resources Research 45:8.
Online publication date: 1-Aug-2009.

Vincent, L., X. Zhang, B. Bonsal, and W. Hogg, 2002: Homogenization
of daily temperatures over Canada. J. Climate, 15, 1322–1334.

Regards

R
User avatar
rene
 
Posts: 36
Joined: Sun Aug 17, 2014 9:57 am
Location: Beijing


Return to Data

cron